You almost give the answer in your question:
With increasing flow rates, the inertial forces become larger than the viscous forces
We can state it more precisely: The inertial forces scale quadratically with the flow speed $U$: $\rho \mathbf u\nabla \mathbf u\sim \rho U^2/L$, while the viscous forces scale linearly: $\mu \nabla^2\mathbf u \sim \mu U/L^2$. Here $\rho$ is fluid density, $\mu$ dynamic viscosity and $L$ is a characteristic length scale.
The ratio between the two is the Reynolds number $\mathrm{Re} = \rho UL/\mu$. Therefore, when $\mathrm{Re}$ is large, the viscous term may be treated as small.
Now, I cannot downvote, but I'd like to amend a misconception present in the question and the two existing answers.
The viscous term $\mu \nabla^2\mathbf u$ is not to be interpreted as "diffusive flow", nor as molecular diffusion (as described by the Péclet number.) Instead it describes the effects of internal friction in the fluid. Friction arises when neighboring fluid parcels have different velocities. The effect of friction is to even out the velocities, thus reducing flow velocity gradients. In fact, the very form of the viscous term comes from the assumption of a Newtonian fluid: the friction force is proportional to the local flow velocity gradient$^1$, and the constant of proportionality is the dynamic viscosity $\mu$.
It is common practice to call the viscous term "diffusion", presumably because of the second derivative. The quantity "diffusing" in Navier-Stokes is the fluid velocity, in the sense that high fluid velocities diffuse toward regions of lower fluid velocities, in the negative gradient direction. But in contrast to molecular diffusion there is no randomness involved in the viscous term$^2$. The irreversibility is a result of frictional dissipation.
I'll re-use an important paragraph from tpg2114's answer:
The important thing to remember is that there are limits to where the
approximations can be applied. At high Reynolds number then one can
use the Euler equations ignoring viscosity outside of the thin region
around bodies where no matter how large the convective velocity is
there will always be viscous effects there.
We now understand why this is: the viscous term actually scales with fluid velocity gradients, while the convective term scales with the fluid velocity itself. Therefore, near a stationary boundary where the velocity is zero, it follows that the velocity is small and the gradients are large, hence the value of $\mathrm{Re}$ is locally small, and there's a boundary layer.
Finally, I'd like to comment, also with respect to tpg2114's answer, that units do indeed matter. The important thing for the inviscid approximation to hold is that the dimensionless Reynolds number is large. Even if we among friends may say "for large velocities..", we must understand that we mean "for large values of the Reynolds number" in this case. It is similar to quantum mechanics where we may say "because $\hbar$ is so small..", but in fact we usually employ units where $\hbar=1$.
$^1$ The particular form for a Newtonian fluid arises from the statement that the stress $\tau_{ij}$ (force per area due to pressure and friction) is given by $\tau_{ij} = -p\delta_{ij} + \mu (\partial_ju_i+\partial_iu_j)$. The Navier-Stokes equations are in fact momentum conservation equations stating that the divergence of the stress tensor equals the time rate of change of momentum:
$$
\frac{D\rho u_i}{Dt}=\partial_j\tau_{ij}
$$
$^2$ However, it must be said that the friction (viscosity) is of course a consequence of random collisions on the molecular level of modeling. The viscosity often has a strong dependence upon temperature, for example. But on the continuum level, this randomness is averaged into macroscopic quantities and modeled by constitutive equations like the Newtonian fluid above.
