You appear to know a bit more about the rotation group than the authors assume: They start from basic infinitesimal rotations in three dimensions, standard orthogonal matrices, and steer you in unravelling their Lie group structure in elementary terms.
- Remember, the group commutator involves only multiplications/inversions, but the algebra commutator involves those and also additions/subtractions.
A Lie group sophisticate understands, instead,
$$
R_y(\epsilon)= e^{\epsilon L_y}= {\mathbb I} + \epsilon L_y + O(\epsilon^2), \\ L_y\equiv
\begin{pmatrix} 0&0& 1 \\
0&0&0\\ -1&0&0 \\ \end{pmatrix}= -L_y^T,
$$
etc, for the other two directions.
So a rotation, a group element, is the identity plus a generator, plus smaller higher order terms.
The group commutator of two group elements, $R_x(\epsilon), R_y(\epsilon)$, then, is defined as the succession of rotations, so a group element,
$$
R^{-1}_x(\epsilon)R^{-1}_y(\epsilon)R_x(\epsilon)R_y(\epsilon)\\ \approx ({\mathbb I} - \epsilon L_x +...)({\mathbb I} - \epsilon L_y +...)({\mathbb I} + \epsilon L_x +...)({\mathbb I} + \epsilon L_y +...)\\
\approx ({\mathbb I} - \epsilon L_x - \epsilon L_y +\epsilon^2[L_x,L_y]/2+...)({\mathbb I} + \epsilon L_x +\epsilon L_y +\epsilon^2[L_x,L_y]/2+...)\\
={\mathbb I}+\epsilon^2 [L_x,L_y] + O(\epsilon^3) \approx
R_z(\epsilon^2) \\ \approx {\mathbb I}+ R_x(\epsilon)R_y(\epsilon)-R_y(\epsilon)R_x(\epsilon) .
$$
N.B. Recall the CBH expansion,
$e^{\epsilon L_x}e^{\epsilon L_y}= e^{\epsilon L_x+\epsilon L_y+\epsilon^2 [L_x,L_y]/2+...}$. So, to $O(\epsilon^2)$, the group commutator reduces to the identity, corrected to a smaller rotation at $\epsilon^2$.
- At order $\epsilon^2$, infinitesimal rotations reconstruct the Lie algebra of generators, (3.9),
$$[R_x(\epsilon)-{\mathbb I}, R_y(\epsilon)-{\mathbb I}]= R_z(\epsilon^2)-{\mathbb I}
~\leadsto ~ [L_x,L_y]=L_z.$$
That is, the authors introduce the Lie algebra commutator without explicitly introducing the Lie algebra, or its exponential, merely through infinitesimal rotation sequences, and addition of group elements (!) approximating the well-defined group commutator. The addition is a feature of the algebra, not the group, but is well defined for matrices, as utilized here. Neat, huh?
Reading on will connect you to this underlying Lie algebra.
If you already appreciate the magic of the Lie structure, the book's illustration is reassuring or distracting overkill; but if you don't, it is as good as a bona-fide seat-of-the pants introduction, quite sly for an introductory physics text―the hallmark of JunJohn S.
Once you have appreciated this underlying group theory, the text eases you into the most general representations suitable to your Hilbert space, all subject to the same Lie algebraic, and thus Lie group symmetry constraints! The combinatorics involved here in the CBH expansion are identical to those in every and any representation.
