How is estimation bias quantified in reinforcement learning?

Question

In various estimation problems, especially in RL domains, where we currently looking into Q-learning and its variants, we often encounter the term estimation bias, which refers to the systematic deviation of an estimator’s expected value from the true parameter.

For instance, Thrun (1993) [1] mentioned that an estimator has an estimation bias, but I am looking for a standard way to quantify it. I know that bias is generally defined as:

Bias(θ̂) = E[θ̂] - θ

where θ̂ is the estimator and θ is the true parameter.

However, in practical applications, what are the standard techniques used to measure or quantify this bias when we only have a sample of data? Are there specific numerical or computational methods commonly used in machine learning, statistics, or econometrics to estimate it in real-world scenarios?

My Perspective:

I am currently conducting research on quantifying the bias in Q-learning and Double Q-learning algorithms. One of the key challenges in reinforcement learning is understanding how estimation bias propagates in value function updates. Double Q-learning was introduced to mitigate the overestimation bias present in standard Q-learning, but accurately measuring this bias remains an open problem.

From what I have observed, most studies analyze bias through empirical performance evaluations rather than through direct quantification. Some techniques, such as using bootstrapped confidence intervals or Monte Carlo rollouts, attempt to estimate bias by comparing learned value functions with ground truth returns. However, is there a more standardized way to quantify and compare bias across different learning algorithms?

Additionally, I came across the concept of AMSE (Asymptotic Mean Squared Error) in a recent NeurIPS paper [2]. The AMSE is given by:

AMSE(θ̂_n) = E[(θ̂_n - θ)²]

where n is the sample size. However, this paper takes a zero-reference approach, meaning that instead of assuming a known true value θ, the reference is set to zero, and all error measurements are taken relative to that. This effectively means that:

AMSE(θ̂_n) = E[θ̂_n²]

where all bias estimates are relative rather than absolute.

How does this zero-reference approach impact the interpretation of estimation bias in reinforcement learning? Could AMSE be a suitable metric for quantifying bias in Q-learning estimators, or are there alternative approaches that would be more appropriate for reinforcement learning applications?

Any references or examples of bias estimation in RL, particularly in Q-learning and Double Q-learning, would be appreciated.

---

References:

[1] Thrun, S. (1993). Bias and the quantification of stability in learning algorithms. Carnegie Mellon University. Available here.

[2] Bartlett, P. L., Long, P. M., Lugosi, G., & Tsigler, A. (2020). Benign Overfitting in Linear Regression. NeurIPS. Available here.

Answer 1

Indeed bootstrapped confidence intervals and Monte Carlo rollouts are standard empirical statistical methods to quantify and compare value estimator's bias across different model-free RL algorithms which don't have mechanism to infer true values.

The zero-reference AMSE simplifies such analysis but risks conflating bias with the combination of bias and variance, so the usual Q-learning overestimation bias ($\mathbb{E}[\max_a Q(s,a)]-\max_a \mathbb{E}[Q(s,a)]$) cannot be isolated using zero-reference AMSE alone. By the way, zero-reference applies to unknown true parameter cases via statistical standardization for the observed empirical data.