Suppose object A orbits object B, and that there are no outside gravitational influences non-Sun gravity influences, and that object A is completely spherical and its mass is evenly distributed. Now, suppose half of object A's matter is annihilated evenly from the surface of object A.
Given the above scenario, would the orbit of object A remain exactly the same? If not, would the variation be due only to the decrease in gravitational influence of object A on object B (e.g., less wobble)?
If the two objects are equal in mass (or close to it), both orbit their barycenter, which would be a point outside either body. If one object suddenly loses half its mass, the COM of the binary system moves with respect to the current locations of both objects, resulting in changes to acceleration for both ($a=\frac{GM}{ r^2}$, where r is distance to ). i.e, orbits are affected.
If there is a great difference in mass, then the smaller mass experiencing a mass loss for the negligible object has negligible effect on the orbits of either object. In such a case the COM of the binary system is negligibly changed ($r^2$ constant), orbits change negligibly.
Another way to look at it is to see a binary system as containing both kinetic and gravitational potential energy. If the sudden loss of mass through annihilation is to be taken as a removal of energy from the system, then it must be gravitational potential energy that is removed. The objects must end up farther apart (change to orbits) as a result. So it is just a matter of how big a fraction of energy you want to remove from the system to make a change visible--annihilating half of the moon wont appear to affect anything because the earth-moon barycenter would still be inside the earth (the moon would still orbit us as if little changed) while annihilating half the earth would noticeably cause the moon to upgrade to a higher orbit.
