If for example, 2 masses (m1 and m2) are travelling towards each other at speeds close to c and then fuse on collision, is there a simple way to calculate the total mass (M) afterwards? I'm assuming it's not as simple as M = m1 + m2
NB: when I say masses, I refer to rest mass, not relativistic mass
Rest masses do not add linearly - that's one of the awkward things about mass, and they are not conserved in processes and reactions.
You can easily understand this by thinking of the simple case where two masses approach one another at the same speed (relative to the measurement frame), so that the total momentum is nought. The rest mass of the system equals the total energy, which is the sum $2\,m$ of the rest masses together with the kinetic energy (multiplied by $c^2$, if you're not using natural units). So the rest mass of the whole is greater than the sum of the rest masses of the bits. Another simple example: two photons approaching one-another: sum of the rest masses is zero, but the rest mass of the whole system is the total energy of the two photons combined.
To your specific problem: IF you can be sure that the only "reaction" product is the fused masses smacked together (in particular, that there is no radiation - electromagnetic, acoustic or otherwise), then you can find the rest mass of the fused system simply by equating the components of the four-momentum before and after the collision. If the approach speeds of masses $m_1,\,m_2$ are $v_1,\,v_2$, and if the sought final velocity (a signed real in this one dimensional case) and rest mass are written $v$ and $m$ respectively, then this four-momentum equation is going to give you (in SI or other non-natural units):
$$m_1^2 c^2 = \frac{E_1^2}{c^2} -p_1^2\tag{1}$$ $$m_2^2 c^2 = \frac{E_2^2}{c^2} -p_2^2\tag{2}$$ $$m^2 c^2 = \frac{E^2}{c^2} -p^2\tag{3}$$ $$p_1 = m_1\,\gamma_1\,v_1;\quad p_2 = -m_2\,\gamma_2\,v_2;\quad p = m\,\gamma\,v\tag{4}$$ $$E=E_1+E_2\tag{5}$$ $$p = p_1+p_2\text{ (watch the signs!)}\tag{6}$$
and you can use this set to eliminate $\gamma\,v$ and derive the new rest mass $m$ in terms of $m_1,\,m_2,\,\gamma_1\,v_1$ and $\gamma_2\,v_2$
There sure is. You are correct in assuming that in relativistic collisions the total mass of the constituents is not additive. However, the linear momentum and energy are.
Consider the fact that the momentum and energy are conserved for the system so we can say the energy and momentum before and after collision are equivalent.
If we define total momentum $p_t$ and energy $E_t$ as
$$ p_t = \sum_i m_iv_i [1-(v_i/c)^2]^{-1/2}, \quad E_t=\sum_i m_i c^2 [1-(v_i/c)^2]^{-1/2}. $$
Then for particles of mass $m_1$ and $m_2$ with associated velocities $v_1$ and $v_2$ respectively we have
$$p_t = m_1 v_1 [1-(v_1/c)^2]^{-1/2} + m_2 v_2 [1-(v_2/c)^2]^{-1/2}, \\ E_t = m_1 c^2 [1-(v_1/c)^2]^{-1/2} + m_2 c^2 [1-(v_2/c)^2]^{-1/2}. $$
Now, let's consider a solid example. A ball of mass $16kg$ with velocity $3/5c$ inelasticity collides with a ball of mass $9kg$ with velocity $-4/5c$ (i.e. moves in the opposite direction to the first ball with respect to some inertial observer). What is the combined mass after the collision?
If you use the formulae that I've given above you will find, post-collision the "merged" ball will have zero momentum i.e. is at rest. BUT will have a new mass $M=35kg$.
The example I've mentioned can be found here. Hope this helps.
Here are two geometrical ways to calculate the invariant mass in this totally-inelastic collision.
I will use the example cited by Rumplestillskin:
A ball of mass 16kg with velocity 3/5c inelasticity collides with a ball of mass 9kg with velocity −4/5c (i.e. moves in the opposite direction to the first ball with respect to some inertial observer).
[Analogous to a problem in Euclidean geometry (that one might do in a free-body diagram),] note that this triangle can be decomposed into two Minkowski-right triangles by drawing a segment Minkowski-perpendicular to $\vec P_f$ through the tip of $\vec P_1$. Then the magnitude of $\vec P_f$ is $$M=(m_1\cosh\theta_1+m_2\cosh\theta_2)=(\gamma_1 m_1 + \gamma_2 m_2).$$
With the numbers given in the example, $$35=16 \cosh(\rm arctanh(3/5)) +9\cosh(\rm arctanh(-4/5))$$
Note that this result is analogous to the situation in the clock paradox--the elapsed time of the inertial observer in terms of the elapsed times of the legs of the trip of the traveling piecewise-inertial observer.
Admittedly, the problem is relatively simple since we work in the center of momentum frame. In general, one would have three rapidities with respect to the given frame of reference. The rapidities in the triangle are simply differences-in-rapidities. Complications arise if one has to express the result in terms of velocities in the given frame.
