Is negative mass for a bound system of two particles forbidden?

Question

Is there any theorem that forbids the bound system of two massive particles to have negative mass?

Answer 1

A negative binding energy would make the vaccum unstable.

For example suppose a virtual electron and positron pop out of the vacuum. This costs energy to create the particles, but if their binding energy could be greater in magnitude than their rest masses then they could bind to form an energy state lower than the vacuum from which they were created. The result is that the vaccum would spontaneously decay into a lower energy state, which would then be the new vacuum state.

So the vacuum is by definition the lowest energy state that can exist, and no bound state can have a total energy lower than this.

Answer 2

Within special relativity definitions mass is the positive root of the square root of the dot product in four vector space.

With this definition there is no way a mass can be negative by construction. Two particles at rest will have zero momenta and their masses will add linearly for the minimum invariant mass of their system. Once they get momentum the invariant mass goes up. Bound particles have a momentum .