I've learned some basic ideas of group theory like the basis for a group representation.
In 2 spin-1/2 electrons coupling case, it is said $$2\otimes2=1\oplus3.$$
The direct product can be reduced to some direct sum. The basis belongs to one representation of a "group".
My question is, what is the concrete "group" here? Is it "$\{S^2_{total }\}$" alone?
My question is, what is the concrete "group" here? Is it "$\{S^2_{total }\}$" alone?
No, it is not $\{S^2_{total }\}$ alone, nor any combination of the angular momenta alone that form the group of interest. The group of interest is the rotation group.
The angular momenta (e.g., $S_x$, $S_y$, $S_z$) are the "generators" of the group.
We exponentiate the generators to get the group elements: $$ U(\hat n, \theta) = e^{-i\theta \hat n\cdot \vec S}\;, $$
For example, for rotations about the $z$ axis where $\hat n = \hat z$ we have: $$ U(\hat z, \theta) = e^{-i\theta S_z}\;, $$ which, for the 2x2 representation just looks like: $$ e^{-i\theta S_z}= \left(\begin{matrix} e^{-i\theta/2} & 0 \\ 0 & e^{+i\theta/2} \end{matrix}\right) $$
As a further example, consider the addition of two 2x2 $S_z$ angular momenta, which results in a 4x4 matrix representation: $$ S_z^{(\text{total})} = S_z\otimes 1_2 + 1_2\otimes S_z $$ $$ = \left(\begin{matrix} 1 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & -1 \end{matrix}\right)\;, $$ which can be transformed via a unitary transformation into: $$ \left(\begin{matrix} 1 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & -1 & 0 \\ 0 & 0 & 0 & 0 \end{matrix}\right)\;, $$ where now it is more clear that this matrix is the direct sum of a "spin 1" $S_z$ matrix: $$ S_z^{\text{spin-$1$}}=\left(\begin{matrix} 1 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & -1 \end{matrix}\right)\;, $$ which is non-trivial, and a "spin 0" S_z matrix: $$ S_z^{\text{spin-$0$}} = \left(\begin{matrix}0\end{matrix}\right)\;, $$ where you can see why we call this latter "spin 0" representation "trivial." (It generates the trivial representation of the group where all members are represented by the number $1 = e^0$).
Here, we have only provided an example for z-angular momenta, which is fairly simple. However, the block diagonalization "works" for all the momentum components.
In general when we add two angular momenta, we can perform matrix manipulations to put the resulting direct product matrix into a direct sum form. This is what is meant by the notation: $$ 2\otimes 2 = 3\otimes 1\;, $$ or equivalently, if stated in terms of spin values rather than matrix sizes: $$ \frac{1}{2}\otimes \frac{1}{2} = 1\oplus 0\;. $$
As another example, in terms of spin not matrix size: $$ \frac{1}{2}\otimes 1 = \frac{3}{2}\oplus\frac{1}{2}\;, $$ or, in terms of matrix size: $$ 2*3 = 4 + 2\;. $$
For the example of $\frac{1}{2}\otimes 1$, after block diagonalizing the spin matrices, we find that the rotation matrices can be written in block diagonal form as well like: $$ R^{\frac{3}{2}\oplus\frac{1}{2}} = \left( \begin{matrix} \left( \begin{matrix}.&.&.&.\\ .&.&.&.\\ .&.&.&.\\ .&.&.&. \end{matrix} \right) & \vec 0 \\ \vec 0^T & \left( \begin{matrix} .&. \\ .&. \end{matrix} \right) \end{matrix} \right)\;. $$
