Suppose we have a mechanical balance, with two identical particles kept in the two sides. Now the balance does not show any deflection. Now, one of the particles is given some constant horizontal velocity. Will the balance show the moving particle to be heavier (that side will move downward )or not?
(There is no friction between the balance and the moving particle)
Yes, it would -- the elementary pre-general relativity answer is "because gravity (which is what is measured by your balance) depends on the energy (mass plus kinetic energy) of an object, not the rest mass". So although the mass remains the same, a special relativistic correction to Newtonian gravity would be to consider the total energy instead of invariant mass.
This answer is unsatisfactory, however, because it doesn't seem to make sense to consider energy individually when calculating gravity, when it's just the time-like component of the four-momentum $(E, p_x,p_y,p_z)$ -- it would actually seem more natural to use mass (the norm of this vector) in your calculation than to use energy.
This is actually the theoretical motivation for general relativity, which explains that this force which depends on energy, $\Gamma_{00}^i$, is just one of the components (the "time-time component") of the sixteen components of the gravitational field tensor, albeit the most significant component we see at weak gravitational fields and low speeds. This is seen, e.g. in the Einstein field equation, where each of the sixteen components of the Einstein tensor depends on a corresponding component of the energy-momentum tensor, such as energy, momentum, pressure and shear stress.
Your experiment doesn't actually distinguish between the two definitions, because the two definitions give equivalent dynamics.
In the "rest-mass" framework, the gravitational force between two objects, where one is stationary and the observer's frame and one is moving, is
$$F=\gamma \frac{GMm_0}{r^2}$$
where $m_0$ is the rest mass, because forces perpendicular to the velocity (as is the case here) transform as $F\to \gamma F$ under Lorentz boosts in this framework.
In the "relativistic-mass" framework, we get to keep $F\to F$ under Lorentz transforms, because $m=\gamma m_0$ is no longer Lorentz-invariant. So the force is still
$$F=\frac{GMm}{r^2}=\gamma \frac{GMm_0}{r^2}$$
The difference is merely convention. In one case, you associate the required $\gamma$ with the way that force (or more fundamentally, momentum) transforms under Lorentz boosts. In the other case, you associate $\gamma$ with the way that mass transforms under Lorentz boosts. The two formulations give consistent results.
