What specifically about the torque vector is perpendicular? Is the torque vector like this only so that it works smoothly with linear algebra?
The only explanation I get usually is "because it's just like that" or "because cross product". But why is it a cross product? Just for convenience?
I am a first year physics student. Do I have to be an expert in calculus and linear algebra before I can understand this first year physics concept?
I just want to understand physics ... :(
It's neither calculus nor linear algebra; rather, it's geometry, or perhaps representation theory, or Clifford algebra... each of which can lead to a long an in-depth answer. That will not be presented here.
Clifford algebra is the most complete. In that, torque ($\tau$) is not a vector, it's a bi-vector, as is angular acceleration ($\alpha$). The bi-vector in 3 dimensions can be characterized by the cross product.
From a geometric perspective, it is au currant to express physical laws in a coordinate free manner that relates similar geometric objects on both sides of the equal sign. Hence:
$$ \tau = I\alpha$$
relates two bi-vectors.
If you'd rather stick with plain old Euclidean objects, then torque is really the 3 independent components of an antisymmetric rank 2 tensor:
$$ T_{ij} = r_iF_j - F_jr_i $$
This can be converted into an axial-vector via:
$$ \tau_i = \frac 1 2 \epsilon_{ijk}T_{jk}$$
which rotates like a vector, but it's different (e.g., it's parity even).
