Given two coprime integers $a$ and $N$, you can compute $a^{-1} \text{ mod } N$ efficiently classically by using the extended Euclidean algorithm even if the factorization of $N$ is unknown. This is what saves you.
To see this easily, note that said algorithm gives you integers $u, v$ and $d$ such that $ua + vN = d = \gcd(a, N) = 1$. It follows that $ua \equiv 1 \:\: (\text{mod } N)$ and hence that $u \equiv a^{-1} \:\: (\text{mod } N)$.
(Note also that had we know the order $r$ of $a$ perceived as an element of a cyclic subgroup to $\mathbb Z_N^*$, then we could have used that $a^{-1} \equiv a^{r-1} \:\: (\text{mod } N)$ to find the inverse $a^{-1} \text{ mod } N$. This technique generalizes to any cyclic group, and so may be useful for taking inverses efficiently classically when implementing Shor's algorithm for computing discrete logarithms — if needed to implement the arithmetic, and if the group in question does not admit a more efficient algorithm for taking inverses. But in Shor's order-finding algorithm, the order $r$ of $a$ can of course not be assumed to be classically known.)