You can disappear into this problem to any bottomless depth you like, or at least to the depth that you need as the crucial step to solve one of the great problems of 20th Century Mathematics, namely Hilbert's Fifth Problem.
But your insight of there being an affine parameterization is pretty much a good answer to your own question. If we write down what this means, intuitively and colloquially, it's a "constant amount of group oomph per unit time", or whatever you like to call your affine parameter.
The existence of an affine parameterization, i believe, still requires continuity assumptions about the manifold: that the unit interval's image, and, through the group product and inversion, the real line's image is a continuous, connected subset of the parameterization map, and then you can use continuity to "squidge"and "stretch" the parameterization.
You can then define (in the absence of a metric) informally an affine parameterization in the Lie Group as a parameterization that gives you constant "group action oomph per unit time". To formalize this: Let $\gamma:\mathbb{R}\to\mathfrak{H}\subseteq\mathfrak{G}$ be a parameterization of a one parameter subgroup $\mathfrak{H}\subseteq\mathfrak{G}$ of a Lie Group $\mathfrak{G}$.
If the oomph per unit time equates to the group element $\gamma(1)\in\mathfrak{G}$ (or, in an application, the group action of this element on whatever is the transformed space or set in question), then the oomph in half the time, or, in general $1/N$ of the time has to fulfill $\gamma\left(\frac{1}{2}\right)^2 = \gamma(1)$ (or in general $\gamma\left(\frac{1}{N}\right)^N = \gamma(1)$). Affine here just means that the same group element is imparted for a given fixed time interval whenever that interval happens, so for $N$ times that interval is that group element iterated $N$ times.
To ensure that the square root exists in the one parameter subgroup, one has to use at least continuity AND the famous "No Small Subgroups" assumption. I sketch a proof for the Lorentz group here that is readily generalized to the one parameter subgroups of any Lie Group. Note that, in my proof there, i assumed matrix groups and therefore existence of the matrix logarithm instead of the no small subgroups (NSS) assumption. This guarantees existence and uniqueness, but NSS also does so and relieves us of the constraint of working in matrix groups. Basically NSS says that there must be a neighborhood $\mathcal{N}\subseteq\mathfrak{G}$ of the identity such that every element $\zeta\mathcal{N}$ arbitrarily near to the identity has an "escape sequence", i.e. the discrete subgroup $\left\{\mathrm{id},\, \zeta^{\pm 1},\,\zeta^{\pm 2},\,...\right\}$ eventually breaks out of $\mathcal{N}$, i.e. $\exists k\in\mathbb{Z}\ni \zeta^k\notin\mathcal{N}$.
So, at long winded last, a corollary of making the famous "No Small Subgroups" assumption together with group continuity is that square roots exist and are unique in a small enough neighborhood of the identity, and, since we found one by this alone is enough to show the stated identity is true on a dense subset of the real line (the subset in fact being all real numbers whose fractional parts have a finite binary expansion, i.e. fractions like $0.11011_2$). An assumption of group continuity then means that our affine parameterization defines a path $\gamma(\tau)\in\mathfrak{G}$ that passes through the identity when $\tau=0$ and which fulfills $\gamma(x)=\gamma(1)^x$ for all $x\in\mathbb{R}$, as well as $\gamma(x)\gamma(y)=\gamma(x+y)$.
At last we are done. I feel like a lie down after all that, don't you?!☺️
If you assume a $C^1$ manifold (which , then the description of unit oomph per unit affine time is then the statement:
$$\mathrm{d}_\tau \gamma(\tau)=\gamma(\tau) X$$
where $X$ is some differential operator at the identity for a left invariant vector field on the tangent bundle and $\gamma(\tau) X$ its left translation. Then $\gamma(\tau)=\exp(\tau X)$ in the sense of the manifold's exponential map as a Riemannian manifold $C^1$ (or at least as manifold with affine connexion). The stated identity then trivially follows.
So, in summary, there are all kinds of ways wherein you can see that your identity must be true that are much more basic than just assuming it.
But when you dig deep, you DO have to assume something - it's not obvious or automatically true. As Dean Montgomery, Lev Zippin and Andrew Gleason showed in their solution of Hilbert's Fifth Problem, one of the most basic sets of assumptions needed is continuity and No Small Subgroups.
And your thinking in terms of affine parameterizations is a way that i find to be most elegant!
