Consider a particle $\,\rm P\,$ of rest mass $\,m_0\,$ moving in an inertial system $\,\mathrm S\boldsymbol{\equiv}\mathrm Ox_1x_2x_3t\,$ with velocity
\begin{equation}
\mathbf{u}\boldsymbol{=}\dfrac{\mathrm d\mathbf{x}}{\mathrm dt}\boldsymbol{=}
\begin{bmatrix}
u_1\\
u_2\\
u_3
\end{bmatrix}
\tag{01}\label{01}
\end{equation}
For the relativistic linear and angular momentum 3-vectors $\,\mathbf{p}\,$ and $\,\mathbf{h}\,$ respectively we have
\begin{equation}
\mathbf{p}\boldsymbol{=}\gamma_{\rm u}m_0\mathbf{u}\boldsymbol{=}
\begin{bmatrix}
\gamma_{\rm u}m_0u_1\\
\gamma_{\rm u}m_0u_2\\
\gamma_{\rm u}m_0u_3
\end{bmatrix}
\boldsymbol{=}
\begin{bmatrix}
p_1\\
p_2\\
p_3
\end{bmatrix}
\quad \text{where} \quad \gamma_{\rm u} \boldsymbol{=} \left(1\boldsymbol{-}\dfrac{u^2}{c^2}\right)^{\boldsymbol{-}\frac12}
\tag{02}\label{02}
\end{equation}
and
\begin{equation}
\mathbf{h}\boldsymbol{=}\mathbf{x}\boldsymbol{\times}\mathbf{p}\boldsymbol{=}
\begin{bmatrix}
x_2p_3\boldsymbol{-}x_3p_2\\
x_3p_1\boldsymbol{-}x_1p_3\\
x_1p_2\boldsymbol{-}x_2p_1
\end{bmatrix}
\boldsymbol{=}
\begin{bmatrix}
\gamma_{\rm u}m_0\left(x_2u_3\boldsymbol{-}x_3u_2\right)\\
\gamma_{\rm u}m_0\left(x_3u_1\boldsymbol{-}x_1u_3\right)\\
\gamma_{\rm u}m_0\left(x_1u_2\boldsymbol{-}x_2u_1\right)
\end{bmatrix}
\boldsymbol{=}
\begin{bmatrix}
h_1\\
h_2\\
h_3
\end{bmatrix}
\tag{03}\label{03}
\end{equation}
In equations \eqref{02} and \eqref{03} the only mass term is the rest one $\,m_0$. There is no such a quantity like 'relativistic mass' and it would be a good practice not to use this term for $\,\gamma_{\rm u}m_0\,$ as suggested by the experts in the field (see the comments under the question).
Now, to see how the angular momentum is transformed under a Lorentz transformation let an inertial system $\,\mathrm S'\boldsymbol{\equiv}\mathrm O'x'_1x'_2x'_3t'\,$ translating with respect to the inertial system $\:\mathrm S\:$ with constant velocity
\begin{align}
\boldsymbol{\upsilon} & \boldsymbol{=}\left(\upsilon_{1},\upsilon_{2},\upsilon_{3}\right)
\tag{04a}\label{04a}\\
\upsilon & \boldsymbol{=}\Vert \boldsymbol{\upsilon} \Vert \boldsymbol{=} \sqrt{ \upsilon^2_{1}\boldsymbol{+}\upsilon^2_{2}\boldsymbol{+}\upsilon^2_{3}}\:\in \left(0,c\right)
\tag{04b}\label{04b}
\end{align}
as in Figure-01.
The Lorentz transformation is
\begin{align}
\mathbf{x}^{\boldsymbol{\prime}} & \boldsymbol{=} \mathbf{x}\boldsymbol{+} \dfrac{\gamma_\upsilon^2}{c^2 \left(\gamma_\upsilon\boldsymbol{+}1\right)}\left(\boldsymbol{\upsilon}\boldsymbol{\cdot} \mathbf{x}\right)\boldsymbol{\upsilon}\boldsymbol{-}\gamma_\upsilon\boldsymbol{\upsilon}\,t
\tag{05a}\label{05a}\\
t^{\boldsymbol{\prime}} & \boldsymbol{=} \gamma_\upsilon\left(t\boldsymbol{-} \dfrac{\boldsymbol{\upsilon}\boldsymbol{\cdot} \mathbf{x}}{c^2}\right)
\tag{05b}\label{05b}\\
\gamma_\upsilon & \boldsymbol{=} \left(1\boldsymbol{-}\dfrac{\upsilon^2}{c^2}\right)^{\boldsymbol{-}\frac12}
\tag{05c}\label{05c}
\end{align}
For the Lorentz transformation \eqref{05a}-\eqref{05c} it could be proved that the pair of the angular momentum 3-vector $\,\mathbf{h}\,$ and the following defined 3-vector
\begin{equation}
\boldsymbol{\varrho}\boldsymbol{=}\gamma_{\rm u}m_0\mathbf{x}\boldsymbol{-}\mathbf{p}\,t\boldsymbol{=}\gamma_{\rm u}m_0\left(\mathbf{x}\boldsymbol{-}\mathbf{u}\,t\right)
\boldsymbol{=}
\begin{bmatrix}
\gamma_{\rm u}m_0\left(x_1\boldsymbol{-}u_1\,t\right)\\
\gamma_{\rm u}m_0\left(x_2\boldsymbol{-}u_2\,t\right)\\
\gamma_{\rm u}m_0\left(x_3\boldsymbol{-}u_3\,t\right)
\end{bmatrix}
\boldsymbol{=}
\begin{bmatrix}
\varrho_1\\
\varrho_2\\
\varrho_3
\end{bmatrix}
\tag{06}\label{06}
\end{equation}
is transformed as follows
\begin{align}
\mathbf{h}' & \boldsymbol{=}\gamma_\upsilon \mathbf{h}\,\boldsymbol{-}\,\dfrac{\gamma_\upsilon^2}{c^2 \left(\gamma_\upsilon\boldsymbol{+}1\right)}\left(\mathbf{h}\boldsymbol{\cdot} \boldsymbol{\upsilon}\right)\boldsymbol{\upsilon}\,\boldsymbol{+}\,\gamma_\upsilon\left(\boldsymbol{\upsilon}\boldsymbol{\times}\boldsymbol{\varrho}\right)
\tag{07a}\label{07a}\\
\boldsymbol{\varrho}' & \boldsymbol{=} \gamma_\upsilon \boldsymbol{\varrho}\,\boldsymbol{-}\,\dfrac{\gamma_\upsilon^2}{c^2 \left(\gamma_\upsilon\boldsymbol{+}1\right)}\left(\boldsymbol{\varrho}\boldsymbol{\cdot} \boldsymbol{\upsilon}\right)\boldsymbol{\upsilon}\,\boldsymbol{-}\,\dfrac{\gamma_\upsilon}{c^2}\left(\boldsymbol{\upsilon}\boldsymbol{\times}\mathbf{h}\right)
\tag{07b}\label{07b}
\end{align}
Now, for the component of the relativistic angular momentum along the direction of motion
\begin{equation}
\mathbf{n} \boldsymbol{=}\dfrac{\boldsymbol{\upsilon}}{\Vert \boldsymbol{\upsilon} \Vert}\boldsymbol{=}\dfrac{\boldsymbol{\upsilon}}{ \upsilon}
\tag{08}\label{08}
\end{equation}
we have from equation \eqref{07a}
\begin{align}
\left(\mathbf{h}'\boldsymbol{\cdot}\mathbf{n}\right)\mathbf{n} & \boldsymbol{=}\dfrac{\left(\mathbf{h}'\boldsymbol{\cdot}\boldsymbol{\upsilon}\right)\boldsymbol{\upsilon}}{\upsilon^2}\boldsymbol{=}\gamma_\upsilon\dfrac{\left(\mathbf{h}\boldsymbol{\cdot}\boldsymbol{\upsilon}\right)\boldsymbol{\upsilon}}{\upsilon^2}\,\boldsymbol{-}\,\dfrac{\upsilon^2\gamma_\upsilon^2}{c^2 \left(\gamma_\upsilon\boldsymbol{+}1\right)}\dfrac{\left(\mathbf{h}\boldsymbol{\cdot}\boldsymbol{\upsilon}\right)\boldsymbol{\upsilon}}{\upsilon^2}
\nonumber\\
&\boldsymbol{=}\underbrace{\left[\gamma_\upsilon\,\boldsymbol{-}\,\dfrac{\upsilon^2\gamma_\upsilon^2}{c^2 \left(\gamma_\upsilon\boldsymbol{+}1\right)}\right]}_{\boldsymbol{=}1}\dfrac{\left(\mathbf{h}\boldsymbol{\cdot}\boldsymbol{\upsilon}\right)\boldsymbol{\upsilon}}{\upsilon^2}\boldsymbol{=}\left(\mathbf{h}\boldsymbol{\cdot}\mathbf{n}\right)\mathbf{n}
\tag{09}\label{09}
\end{align}
that is
\begin{equation}
\left(\mathbf{h}'\boldsymbol{\cdot}\mathbf{n}\right)\mathbf{n} \boldsymbol{=}\left(\mathbf{h}\boldsymbol{\cdot}\mathbf{n}\right)\mathbf{n}
\tag{10}\label{10}
\end{equation}
So the component of the relativistic angular momentum along the direction of motion between these two inertial frames is the same in both frames. This result has nothing to do with any concept of 'relativistic mass'.
$\boldsymbol{=\!=\!=\!==\!=\!=\!==\!=\!=\!==\!=\!=\!==\!=\!=\!==\!=\!=\!==\!=\!=\!==\!=\!=\!==\!=\!=\!==\!=\!=\!==\!=\!=\!==\!=\!=\!=}$
$\textbf{ADDENDUM 01}$
Note that for the Lorentz transformation \eqref{05a}-\eqref{05c} the vectors $\:\mathbf{E}\:$ and $\:\mathbf{B}\:$ of the electromagnetic field are transformed as follows
\begin{align}
\mathbf{E}' & \boldsymbol{=}\gamma_\upsilon \mathbf{E}\,\boldsymbol{-}\,\dfrac{\gamma_\upsilon^2}{c^2 \left(\gamma_\upsilon\boldsymbol{+}1\right)}\left(\mathbf{E}\boldsymbol{\cdot} \boldsymbol{\upsilon}\right)\boldsymbol{\upsilon}\,\boldsymbol{+}\,\gamma_\upsilon\left(\boldsymbol{\upsilon}\boldsymbol{\times}\mathbf{B}\right)
\tag{11a}\label{11a}\\
\mathbf{B}' & \boldsymbol{=} \gamma_\upsilon \mathbf{B}\,\boldsymbol{-}\,\dfrac{\gamma_\upsilon^2}{c^2 \left(\gamma_\upsilon\boldsymbol{+}1\right)}\left(\mathbf{B}\boldsymbol{\cdot} \boldsymbol{\upsilon}\right)\boldsymbol{\upsilon}\,\boldsymbol{-}\,\dfrac{\gamma_\upsilon}{c^2}\left(\boldsymbol{\upsilon}\boldsymbol{\times}\mathbf{E}\right)
\tag{11b}\label{11b}
\end{align}
Comparing equations \eqref{07a},\eqref{07b} with \eqref{11a},\eqref{11b}
we conclude that the pair of 3-vectors $\:\left(\mathbf{h},\boldsymbol{\varrho}\right)\:$ is transformed as the pair of 3-vectors $\:\left(\mathbf{E},\mathbf{B}\right)$. But as for the electromagnetic field we construct from $\,\left(\mathbf{E},\mathbf{B}\right)\,$ the anti-symmetric four-tensor
\begin{equation}
\mathrm F^{\mu\nu}\boldsymbol{=}
\begin{bmatrix}
0 & \boldsymbol{-}E_{1} & \boldsymbol{-}E_{2} & \boldsymbol{-}E_{3} \\
E_{1} & \hphantom{\boldsymbol{-}} 0 \hphantom{_{1}} & \boldsymbol{-}cB_{3} & \hphantom{\boldsymbol{-}}cB_{2} \\
E_{2} & \hphantom{\boldsymbol{-}}cB_{3} & \hphantom{\boldsymbol{-}} 0 & \boldsymbol{-}cB_{1} \\
E_{3} & \boldsymbol{-}cB_{2} & \hphantom{\boldsymbol{-}}cB_{1} & \hphantom{\boldsymbol{-}} 0
\end{bmatrix}
\tag{12}\label{12}
\end{equation}
so from the pair of 3-vectors $\:\left(\mathbf{h},\boldsymbol{\varrho}\right)\:$ we construct the anti-symmetric four-tensor
\begin{equation}
\mathrm H^{\mu\nu}\boldsymbol{=}
\begin{bmatrix}
0 & \boldsymbol{-}h_{1} & \boldsymbol{-}h_{2} & \boldsymbol{-}h_{3} \\
h_{1} & \hphantom{\boldsymbol{-}} 0 \hphantom{_{1}} & \boldsymbol{-}c\varrho_{3} & \hphantom{\boldsymbol{-}}c\varrho_{2} \\
h_{2} & \hphantom{\boldsymbol{-}}c\varrho_{3} & \hphantom{\boldsymbol{-}} 0 & \boldsymbol{-}c\varrho_{1} \\
h_{3} & \boldsymbol{-}c\varrho_{2} & \hphantom{\boldsymbol{-}}c\varrho_{1} & \hphantom{\boldsymbol{-}} 0
\end{bmatrix}
\tag{13}\label{13}
\end{equation}
which represents the relativistic angular momentum of a particle.
$\boldsymbol{=\!=\!=\!==\!=\!=\!==\!=\!=\!==\!=\!=\!==\!=\!=\!==\!=\!=\!==\!=\!=\!==\!=\!=\!==\!=\!=\!==\!=\!=\!==\!=\!=\!==\!=\!=\!=}$
$\textbf{ADDENDUM 02 : The Lorentz transformation of velocity 3-vectors}$
This $\textbf{ADDENDUM 02}$ is motivated by the following OP's comment with incorrect arguments :
Suppose frame $\,\mathrm S'\,$ moves relative to frame $\,\mathrm S\,$ in the $\,x_3\,$ direction with speed $\,u_3\,$ so that the particle's velocity relative to $\,\mathrm S'\,$ in the $\,x_3\,$ direction, $\,u'_3\,$, is 0; and $u'_1=u_1,u'_2=u_2$. The Lorentz factor for the particle's momentum relative to frame $\,\mathrm S\,$ will be $\gamma_{u'} = \frac {1}{\sqrt {1- \frac {u_1^2+u_2^2}{c^2} } } \neq \gamma_{u}$. However, $h_3=\gamma_{u} m_0\left(x_1u_2\boldsymbol{-}x_2u_1\right)$ but $h'_3=\gamma_{u'} m_0\left(x_1u_2\boldsymbol{-}x_2u_1\right) \neq h_3$. But we had concluded that $h'_3= h_3$. Hence, we have a contradiction. Something is wrong.
The differential version of the Lorentz transformation \eqref{05a}-\eqref{05c} is
\begin{align}
\mathrm d\mathbf{x}^{\boldsymbol{\prime}} & \boldsymbol{=} \mathrm d\mathbf{x}\boldsymbol{+} \dfrac{\gamma_\upsilon^2}{c^2 \left(\gamma_\upsilon\boldsymbol{+}1\right)}\left(\boldsymbol{\upsilon}\boldsymbol{\cdot} \mathrm d\mathbf{x}\right)\boldsymbol{\upsilon}\boldsymbol{-}\gamma_\upsilon\boldsymbol{\upsilon}\,\mathrm dt
\tag{14a}\label{14a}\\
\mathrm dt^{\boldsymbol{\prime}} & \boldsymbol{=} \gamma_\upsilon\left(\mathrm dt\boldsymbol{-} \dfrac{\boldsymbol{\upsilon}\boldsymbol{\cdot} \mathrm d\mathbf{x}}{c^2}\right)
\tag{14b}\label{14b}\\
\gamma_\upsilon & \boldsymbol{=} \left(1\boldsymbol{-}\dfrac{\upsilon^2}{c^2}\right)^{\boldsymbol{-}\frac12}
\tag{14c}\label{14c}
\end{align}
Now, suppose that a particle is moving relative to frame $\,\mathrm S\,$ with velocity
\begin{equation}
\mathbf{u}\boldsymbol{=}\dfrac{\mathrm d\mathbf{x}}{\mathrm dt}
\tag{15}\label{15}
\end{equation}
To find the velocity of the particle relative to the frame $\,\mathrm S'\,$
\begin{equation}
\mathbf{u'}\boldsymbol{=}\dfrac{\mathrm d\mathbf{x'}}{\mathrm dt'}
\tag{16}\label{16}
\end{equation}
we divide equations \eqref{14a} and \eqref{14b} side by side and we have
\begin{equation}
\left(\dfrac{\mathrm d\mathbf{x'}}{\mathrm dt'}\right)\boldsymbol{=}\dfrac{\left(\dfrac{\mathrm d\mathbf{x}}{\mathrm dt}\right)\boldsymbol{+} \dfrac{\gamma_\upsilon^2}{c^2 \left(\gamma_\upsilon\boldsymbol{+}1\right)}\left[\boldsymbol{\upsilon}\boldsymbol{\cdot} \left(\dfrac{\mathrm d\mathbf{x}}{\mathrm dt}\right)\right]\boldsymbol{\upsilon}\boldsymbol{-}\gamma_\upsilon\boldsymbol{\upsilon}}{\gamma_\upsilon\left[1\boldsymbol{-} \dfrac{\boldsymbol{\upsilon}\boldsymbol{\cdot} \left(\dfrac{\mathrm d\mathbf{x}}{\mathrm dt}\right)}{c^2}\right]}
\tag{17}\label{17}
\end{equation}
that is
\begin{equation}
\mathbf{u'}\boldsymbol{=}\dfrac{\mathbf{u}\boldsymbol{+} \dfrac{\gamma_\upsilon^2}{c^2 \left(\gamma_\upsilon\boldsymbol{+}1\right)}\left(\boldsymbol{\upsilon}\boldsymbol{\cdot} \mathbf{u}\right)\boldsymbol{\upsilon}\boldsymbol{-}\gamma_\upsilon\boldsymbol{\upsilon}}{\gamma_\upsilon\left(1\boldsymbol{-} \dfrac{\boldsymbol{\upsilon}\boldsymbol{\cdot}\mathbf{u}}{c^2}\right)}
\tag{18}\label{18}
\end{equation}
So, for the special case of motions of systems $\,\mathrm S,\mathrm S'\,$ and particle, as in OP's comment, we have
\begin{equation}
\mathbf{u}\boldsymbol{=}
\begin{bmatrix}
u_1\\
u_2\\
u_3
\end{bmatrix}\,,
\quad
\boldsymbol{\upsilon}\boldsymbol{=}
\begin{bmatrix}
0\\
0\\
u_3
\end{bmatrix}
\,,
\quad u_3\boldsymbol{\ne}0
\tag{19}\label{19}
\end{equation}
From equation \eqref{18}
\begin{equation}
\mathbf{u'}\boldsymbol{=}
\begin{bmatrix}
u'_1\\
u'_2\\
u'_3
\end{bmatrix}\boldsymbol{=}\gamma_\upsilon
\begin{bmatrix}
u_1\\
u_2\\
0
\end{bmatrix}
\quad =\!=\!=\!\Longrightarrow \quad \
\begin{bmatrix}
u'_1\boldsymbol{=}\gamma_\upsilon u_1\boldsymbol{=}\left(1\boldsymbol{-}\dfrac{u^2_3}{c^2}\right)^{\boldsymbol{-}\frac12}u_1\\
u'_2\boldsymbol{=}\gamma_\upsilon u_2\boldsymbol{=}\left(1\boldsymbol{-}\dfrac{u^2_3}{c^2}\right)^{\boldsymbol{-}\frac12}u_2\\
u'_3\boldsymbol{=}0\hphantom{_\upsilon u_2\boldsymbol{=}\left(1\boldsymbol{-}\dfrac{u^2_3}{c^2}\right)^{\boldsymbol{-}\frac12}u_2}
\end{bmatrix}
\tag{20}\label{20}
\end{equation}
From \eqref{20} $\,u'_1\boldsymbol{=}u_1\,$ and/or $\,u'_2\boldsymbol{=}u_2\,$ if and only if $\,u'_1\boldsymbol{=}0\boldsymbol{=}u_1\,$ and/or $\,u'_2\boldsymbol{=}0\boldsymbol{=}u_2\,$ respectively.
