We might need some specifics of what's given in the problem.
We start with conservation of total 4-momentum for a single particle after the collision:
$$ \tilde m_1 + \tilde m_2 = \tilde M,$$
where $\tilde m_1$ is the 4-momentum of a particle with invariant mass $m_1$, etc.
The above 4-momentum equation can be visualized in energy-momentum space as a "triangle with three timelike legs" (the sum of two 4-momentum vectors tip-to-tail equals a single 4-momentum vector).
So (since the sum of two vectors lies on the plane spanned by those vectors), this problem reduces to a problem in (1+1)-dimensions.
- The relative-rapidity is ${\rm arccosh}(\hat m_1 \cdot \hat m_2)$ [involving the Minkowski-dot product of their 4-velocities] assuming the $(+,-,-,-)$ signature convention.
[Think "angle between two unit-vectors": $\gamma_{rel}=\cosh\phi_{rel}=\hat m_1 \cdot \hat m_2$.]
- (Note: $v_{rel}=\tanh(\phi_{rel})= \tanh(\phi_2-\phi_1)=\frac{\tanh\phi_2 - \tanh\phi_1}{1-\tanh\phi_2\tanh\phi_1}$.
That is, the relative-rapidity $\phi_{rel}={\rm arctanh}(v_{rel})$, the inverse-hyperbolic-tangent of the relative-velocity [the velocity of $m_2$ in the frame of $m_1$].
The mass $M$ of the output particle (the magnitude of $\tilde M$) can be gotten by the Minkowski version of the law of cosines. (I'll leave the details to you.) Do you know the input masses?
(Possibly useful: my answers to Momentum diagram for two colliding Particles and
How to determine particle energies in center of momentum frame? )
