I'll allow for a degenerate ground state in the following. It seems like there's two questions (including the comments thread). Paraphrased,
What is meant by "not observable" in the context of parallel transport along a path $\gamma$ with respect to Berry connection $\mathbf{A}$?
If the Berry phase is only physically meaningful for loops (closed paths), how can it be observed in real-life systems where it might be impossible to span a perfect loop in the parameter space?
Answers
Let $\Gamma_{\mathbf{A}}(\gamma)$ denote the parallel transport along path $\gamma:[0,T]\to\mathcal{M}$ (from point $a$ to $b$ in the parameter space $\mathcal{M}$) with respect to Berry connection $\mathbf{A}$. Let $g:[0,T]\to\mathcal{M}$ be a smooth family of gauge transformations along $\gamma$. Gauge transformations only act on the parallel transport at the endpoints of $\gamma$. That is, if $\mathbf{A}\mapsto\mathbf{A}'$ under the gauge transformation, then $\Gamma_{\mathbf{A}'}(\gamma)=g^{\dagger}(b)\circ\Gamma_{\mathbf{A}}(\gamma)\circ g(a)$. For more details / background, see my answer to this question. If the ground state is $m$-fold degenerate, then $\Gamma_{\mathbf{A}}(\gamma), g(t)\in U(m)$. Since $g$ amounts to a mere change of orthonormal basis on the ground state space for each point along $\gamma$, it should not have any physically observable consequences. However, we can see that when $\gamma$ is not closed, the parallel transport can be arbitrarily transformed. On the other hand, if $\gamma$ is a closed loop $C$ (i.e. $a=b$), we see that $\Gamma_{\mathbf{A}'}(C)=G^{\dagger}\circ\Gamma_{\mathbf{A}}(C)\circ G$ where $G:=g(a)$. That is, the "holonomy" (i.e. parallel transport around a loop) has gauge-invariant meaning, since its spectral decomposition is invariant. Specifically, if $\mathcal{H}(t)$ is the ground state over $\gamma(t)$, then the holonomy $\Gamma_{\mathbf{A}}(C)$ acts on $\mathcal{H}(0)$ and has spectral decomposition $\sum_{k} u_{k}\Pi_{k}$ where $u_k \in U(1)$ are eigenvalues of the unitary operator $\Gamma_{\mathbf{A}}(C)$, and $\Pi_{k}$ are orthogonal projectors onto its eigenspaces. Clearly, said eigenspaces are always isomorphic under conjugation by $G$. This is why it is said that the Berry phase is undefined for open paths (-- with a bit of wiggle room I suppose! See answer 2), and has physical significance for closed paths.
I'll sketch how I think the argument would go. Suppose $\gamma: [0 , T] \to \mathcal{M}$ parametrises a smooth loop $C$ in $\mathcal{M}$ with $\gamma(0)=\gamma(T)$. In practice, such a loop might be adiabatically, and imperfectly spanned according to a path of form $\gamma : [0,T'] \to \mathcal{M}$ where $T'=T-\epsilon$. The parallel transport is realised by an operator $\mathcal{P}\left(\exp (h(T-\epsilon))\right)$ where we define $h(\tau):=\int_{0}^{\tau} A_{\mu}\dot{\gamma}^{\mu} dt\in\mathfrak{u}(m)$. Let $\tilde{h}:=h(T-\epsilon)$. First, we Taylor expand $\exp(\tilde{h})$ to get $\Gamma:=\sum_{n=0}^{\infty}\frac{1}{n!}(h(T-\epsilon))^{n}=\mathbf{1}+\tilde{h}+\frac{1}{2!}\tilde{h}^{2}+\frac{1}{3!}\tilde{h}^{3}+\cdots$. Next, we Taylor expand $h(T-\epsilon)=\sum_{n=0}^{\infty}\frac{1}{n!}(-\epsilon)^{n}h^{(n)}(T)=h-\epsilon\dot{h}+\frac{1}{2}\epsilon^{2}\ddot{h}+\cdots=h(T)+O(\epsilon)$, which we then plug back into $\Gamma$ to get $\Gamma=\exp (h(T))+O(\epsilon)$. Note that all terms in $O(\epsilon)$ are composed from time-derivatives of $h$; all such terms will be gauge-variant. For instance, $\dot{h}(T)=A_{\mu}(T)\dot{\lambda}^{\mu}(T)$, $\ddot{h}(T)=\dot{A}_{\mu}(T)\dot{\lambda}^{\mu}(T)+A_{\mu}(T)\ddot{\lambda}^{\mu}(T)$ etc. We know that the Berry connection (and consequently its derivatives with respect to $\tau$) are gauge-variant. Consequently, the parallel transport operator along $\gamma$ (from $\tau=0$ to $\tau=T-\epsilon$) is given by $\mathcal{P}(\exp\oint_{\gamma}\mathbf{A}\cdot d\lambda)+O(\epsilon)$. Since the $O(\epsilon)$ correction-terms are gauge-variant, they do not contribute to any physically observable effect of the parallel transport; however, the holonomy $\mathcal{P}(\exp\oint_{\gamma}\mathbf{A}\cdot d\lambda)$ with respect to the closed loop $\gamma$ (i.e. from $\tau=0$ to $\tau=T$) is a gauge-invariant contribution, and hence in principle, is observable. This argument should hold up as long as $\epsilon$ (the accuracy to which the loop is spanned) is sufficiently small that the Taylor expansion of $h(T-\epsilon)$ converges.
tl;dr: 1. Gauge-variant of quantity implies that it is merely a mathematical artefact of formulation, thus not physically observable; gauge-invariance of a quantity is necessary for it to be physically observable. Parallel transport along an open path w.r.t Berry connection is gauge-variant, and is gauge-invariant for a closed path. 2. Taylor expand transport along a path which is almost a loop (out by $\epsilon$); correction terms from expansion are of order $\epsilon$ and are all gauge-variant, leaving a gauge-invariant contribution coming from the holonomy.
