This is not the case. The Aharanov-Bohm effect yields an observable of the form
$$\oint_{\mathcal C} A,$$
where $\mathcal{C}$ is some circuit. This is however gauge invariant. A way to see this is by noting that it can be written in terms of the field strength
$$\int_\Sigma F,$$
via Stokes' theorem. In here we chose a surface $\Sigma$ whose boundary is $\mathcal{C}$. The point however is that in the Aharanov-Bohm effect, the observable corresponds to an experiment that happens in $\mathcal{C}$, not $\Sigma$. In particular, the unintuitive part of it is that there is no apparent reason why the observable should contain information of $F$ everywhere on $\Sigma$ if the particles never went there. Therefore, if you want to describe physics locally, you must use the potential $A$ instead of the field strength $F$. Gauge invariance is however not lost.