A proper answer to your question is more complicated than you might have thought; you are not missing something obvious!
Take the case of light travelling from air to glass. It's easy to show from a simple diagram of wavefronts that, if $\theta_a$ and $\theta_g$ are angles to the normal of a light ray in air and glass,
$$n=\frac{\sin {\theta_a}}{\sin{\theta_g}}=\frac{\text{speed of light in air}}{\text{effective speed of light in glass}}=\frac cv\ \ \ \text {(say)}$$
in which we're taking the speed of light in air to be $c$, its speed in a vacuum.
The critical angle, $\theta_{crit}$, is the value of $\theta_g$ when $\theta_a =90°$, so
$$n=\frac {\sin 90°}{\sin \theta_{crit}}=\frac 1{\sin \theta_{crit}}=\frac cv,$$
As you say, experiments show that, in the visible region, $\theta_{crit}$ for glass decreases with frequency. According to the equation above, this implies that the refractive index, $n$, of glass increases with frequency – as famously confirmed by the splitting of light into colours by a glass prism. More fundamentally, the equation shows that the effective speed, $v$, of light in glass decreases with frequency. Indeed we can say that the decrease in critical angle with frequency is caused by the decrease in $v$ with frequency.
But, you should now ask, why does $v$ decrease with frequency in the visible region? This is where things get complicated, and I'll simply outline what happens... Inside the glass the electric field of the light waves exerts forces on the (bound) electrons in the atoms, causing them to oscillate at the frequency of the light. Their amplitude and phase of oscillation can be determined by modelling them as mass-spring systems. But oscillating electrons emit electromagnetic waves. The light wave that advances through the glass is the resultant of (what's left of) the original wave, and the waves emitted by the oscillating electrons. The effective speed of this wave is less than $c$ because of phase lags between the emitted waves compared with the waves incident on the atoms holding back the progress of the resultant wave.
Paradoxically the wave continues to travel at speed $c$, but the phase of the wave keeps being changed, reducing its effective speed. A loose analogy (that captures none of the subtleties) is with a relay race with clumsy passings-on of the baton: the mean speed of the baton as it goes from start to finish is less than the mean speed of the runners, because of the passings-on times.]
We have attempted to explain why the effective speed of light in glass is less than that in air. But why should its speed decrease with increasing frequency? We need to pay attention to the forced oscillations that the electrons make when exposed to the electric field of an incident light wave. Treated as mass-spring systems, these electrons will have a natural frequency, $f_0$, of oscillation. The frequency, $f$, of light is below this natural frequency, but gets nearer $f_0$ as $f$ increases. This increases the amplitude of forced oscillations and therefore of the e-m waves emitted by the electrons, so the phase lag introduced by the emitted waves increases, decreasing the effective speed of light through the glass even more than for lower frequencies of light.