As physicists we usually don't get lost into existential questions about it. If some case would arise where this thing is not differentiable, it would likely be so bizarre that it would be obvious that the classical approach has some problems. But I can guess you are more a mathematician at heart, and I will try to bring some elements along those lines
In classical mechanics:
Well, the realm of classical mechanics is pretty much defined as those systems that can be described by Newton’s equations (I think). So, well, by definition, the acceleration has to be well defined in any classical mechanics situation.
But we can look at what it means. I think we can agree on the fact that the speed of a particle (in the generic sense) is a continuous function of time, as it could not make sense on an energetic point of view to jump instantly from a speed to another. That does not make it differentiable just yet; a continuous function could still have some nasty behavior at specific points.
Let’s assume that the speed is differentiable on at least some interval $[t_1,t_2[$, but some nasty thing makes it non-differentiable at $t_2$ (for a speed to never be differentiable on any interval at all, it would have to be so insanely discontinuous that I do not take that case into account in a reasonable physical framework). We can then define the acceleration $a(t)$ for times t in $[t_1,t_2[$. The non-differentiability in $t_2$ implies that the following condition is not met:
$$\lim_{\epsilon \to 0^+} \frac{v(t_2+\epsilon)-v(t_2)}{\epsilon}=\lim_{\epsilon \to 0^-} \frac{v(t_2+\epsilon)-v(t_2)}{\epsilon}=a(t_2)$$
Well, let’s consider the two reasons why the above condition could not be met. Let’s first imagine that at this specific time $t_2$, this limit takes an infinite value. That means I can pick an arbitrarily small $\epsilon>0$ such that $a(t_2-\epsilon)$ is arbitrarily large. That would require through newton’s law (which is still defined at $t_2-\epsilon$), an arbitrarily large force. I’m sure you can see how that would be unphysical to have an infinitely large force.
In the second case, where $a(t_2^-)$ and $a(t_2^+)$ are finite but are not equal (think about the absolute value function in 0), that means that acceleration is defined on both sides, but is basically discontinuous in $t_2$ (and so not defined at that point). That implies that your force on your system has been been changing infinitely fast at $t_2$. Which means that some information has travelled faster than light. And that would piss off Einstein. And we kinda like the guy so we avoid doing that.
So it would be quite unphysical to have a time where you can not define the acceleration.
In other formalisms:
Well, the law you are referring to simply does not hold in many contexts. In quantum mechanics for example, position itself is ill-defined, so that’s an issue right from the start.
In fact, the notion of momentum is more general than the idea of a second derivative of position. In different formalisms, momentum may take different forms. In quantum mechanics, you can define the momentum of a monochromatic plane wave as $p=h/\lambda$. In a relativistic framework, it looks more like
$$p=\frac{m_0v}{\sqrt{1-v^2/c^2}}$$
So that already makes quite a difference from the quantity in Newton’s law
The equations of motions are then generalized using a Lagrangian formalism, rather than Newtonian, so the equation you mention becomes quite irrelevant in those fields.
