In what limit does the estimator sample variance converge to the Cramer-Rao bound?

Question

In the context of a single phase estimation problem of a quantum photonics experiment (related post). For example consider a 3-photon quantum circuit (such as the Mach-Zehnder which depends on some phase shift operator which encodes a parameter $\theta$) with a photon counting measurement (two detectors) at the end of the circuit with measurement probabilities:

• P(0,2): 0 photons detected in Detector 1, 2 photons detected in Detector 2.
• P(1,1): 1 photon detected in each detector.
• P(2,0): 2 photons detected in Detector 1, and none in Detector 2.

Consider that at a given time we carry out $\nu$ total measurements. We will get some set of measurement outcomes {$m_{02},~m_{11},~m_{20}$}, where $\nu = m_{02}+m_{11}+m_{20}$. We can define the corresponding likelihood function $L(\theta|\text{data})$ by: $$L(\theta|\text{data}):= P(0,2)^{m_{02}}P(1,1)^{m_{11}}P(2,0)^{m_{20}}.$$

Using Baye's rule, given some assumed prior, $P(\theta)$ and the likelihood function $L(\theta | \text{data})$ above, we can update the posterior (our knowledge about the distribution of the phase $\theta$) iteratively, from the data of the measurement outcomes, by $$P(\theta| \text{data}) = \frac{L(\theta| \text{data})P(\theta)}{P(\text{data})} ,$$ where $P(\text{data})$ is the normalization constant. We can instead consider many (of the order of 100) unnormalized Log-Posterior updates of the form $$\log(P(\theta| \text{data})) = \log(L(\theta| \text{data}) + \log(P(\theta)).$$ Where after each update round $\log(P(\theta| \text{data}))$ replaces $\log(P(\theta))$ for the next update round. This is commonly done for more efficient numerical simulation. We can then consider the MAP estimator (to estimate the encoded phase $\theta$) which is defined as $$\hat{\theta}:= \text{max arg}_{\theta}\log(\theta|\text{data}).$$

If we consider the sample variance $(\Delta \hat{\theta})^2$ of the estimation of the true $\theta$ by carrying out this process $\mu$ independent times (with $\nu$ measurements for every 100 update rounds), yielding a set of $\mu$ different estimation values, am I correct that based on the Cramer-Rao bound we expect that, for $\mu \to \infty$, the sample variance of this set of estimations for an efficient estimator should converge to $\frac{1}{\nu \cdot \text{FI}}$, where FI is the Fisher information? An inefficient estimator should yield some sample variance $\geq \frac{1}{\nu \cdot \text{FI}}$. Are these expectations valid? Thanks for your time and assistance.

Answer 1

The answer is the same as for the other question of yours: no, the efficiency is asymptotic in $\nu$ not in $\mu$. The MLE is not efficient (in general) for any finite finite number of observations.

To see it clearly, consider a toy example where you can work out things analytically.

Say $X_i\sim\operatorname{Bern}(p)$ are IID random variables, and you collect a statistic of $N$ such samples. To be clear, I mean that $X_i\in\{0,1\}$ for each $i$, and $p$ is the probability of getting $1$ at each coin toss, so that $\mathbb{E}[X_i^k]=p$ for all integers $k\ge1$. Say you want to estimate the value of the parameter $p^2$. The corresponding MLE is $\hat p_N^2$ with $\hat p_N$ the MLE for $p$, which is the standard one given by the fraction of times you observed the outcome $1$: $$\hat p_N^2 = \left(\frac{\hat N_1}{N}\right)^2, \qquad \hat N_1 \equiv \sum_{i=1}^N X_i.$$ Let's compute the variance of $\hat p_N^2$. This requires computing the expectation values $\mathbb{E}[(\sum_i X_i)^2]$ and $\mathbb{E}[(\sum_i X_i)^4]$. Using standard multinomial expansions considerations you get $$\mathbb{E}\left[\left(\sum_{i=1}^N X_i\right)^4\right] = N p + 7N(N-1)p^2 \\+ 6 N(N-1)(N-2) p^3 + N(N-1)(N-2)(N-3)p^4, \\ \mathbb{E}\left[\left(\sum_{i=1}^N X_i\right)^2\right] = Np + N(N-1)p^2.$$ It follows that $$\mathbb{E}[\hat p_N] = p^2 + \frac{p(1-p)}{N},$$ $$\operatorname{Var}[\hat p_N^2] = \frac{\mathbb{E}[(\sum_i X_i)^4] - \mathbb{E}[(\sum_i X_i)^2]^2}{N^4} \\= \frac{4p^3(1-p)}{N} + \frac{2p^2(1-p)(3-5p)}{N^2} + O(1/N^3).$$ In the large $N$ limit, you thus get an unbiased estimator with variance $$\operatorname{Var}[\hat p_N^2] \sim \frac{4p^3(1-p)}{N},$$ which is exactly what you should expect, being the Fisher information associated with estimating the parameter $p^2$ equal to $\frac{1}{4p^2 p(1-p)}$ (see e.g. the wiki page and this post on math.SE).

In conclusion, this shows you clearly that while in the $N\to\infty$ limit the MLE indeed becomes efficient, it is never actually efficient for finite $N$. This is probably what you're observing: unless you compute the MLE for $\nu\to\infty$ (in your notation), you'll always have a variance larger than the minimum imposed by the CR bound. Sending $\mu\to\infty$ only serves to accurately estimate the variance at fixed $N$ or $\nu$.