Training an MNIST classifier with a regular ANN will make the model recognize its unrotated version.
But is there such a model where I train the unrotated version as usual, but it also recognizes its rotated version, e.g., the 90-degree version, a.k.a. the transposed version that I never fed the rotated version during training before?
If you only care about whether an image contains a a certain digit after arbitrary rotation, not where the digit is located, rotation invariance suffices and the latest Rotation-Invariant Coordinate-CNNs as recommended in the comment could be a simple straightforward approach without rotated data augmentations during training in theory, simply based on the input space's polar coordinate system, where after filter convolutions in polar space aggregating over the angular axis makes the representation invariant to the specific angle of rotation.
However, in deep learning equivariance is a more important and useful property than invariance as Cohen & Welling emphasized in their 2016 paper on the G-CNNs to address similar rotation issue albeit in a group-theoretic way.
the network or layer $Φ$ that maps one representation to another should be structure preserving. For G-spaces this means that $Φ$ has to be equivariant: $Φ(T_g x)=T′_gΦ(x)$ (1). That is, transforming an input $x$ by a transformation $g$ (forming $T_g x$)... Equivariance can be realized in many ways, and in particular the operators $T$ and $T′$ need not be the same... From equation (1) we see that the familiar concept of invariance is a special kind of equivariance where $T′_g$ is the identity transformation for all $g$. In deep learning, general equivariance is more useful than invariance because it is impossible to determine if features are in the right spatial configuration if they are invariant.
Therefore in many real world CNN deep learning facial-recognition and object-detection tasks rotation-equivariance is required instead of mere invariance for classification tasks. For example, for self-driving cars knowing the relative location of the pedestrian is more important than knowing a pedestrian exists. If you rotate and shift the captured image, the feature maps in a CNN should also rotate and shift correspondingly preserving the structure of features. In such cases Cohen & Welling's 2016 steerable CNNs could be kept in mind for many types of equivariance preservation including rotation, translation and reflection. Here's another blog dedicated to such topic.
