Assuming that R and U have signed the Conventions (some protections apply to countries that haven't signed the Conventions, but a treaty isn't binding on countries that haven't signed it, although of course the principle that countries can't be held to treaties they haven't signed is itself not binding, so there's nothing practically stopping a country from prosecuting for war crimes by a country that didn't sign, it's just that the country doesn't have the moral standing of "you violated your agreements):
The first question: can captured soldiers from R's army claim protection under Geneva Convention?
Generally speaking, yes. The Conventions (not that it's plural) apply to armed conflict, regardless of what the parties label it as. However, claiming full POW status has several requirements, such as openly displaying marking of the country. So, for instance, if, in this hypothetical of yours, soldiers from R were to enter U while not in R uniforms, and pretending to be local "freedom fighters" native to U, then they would be unlawful combatants, and would have their protections under the Conventions significantly reduced (but not eliminated).
The second question: is R required to respect Geneva Convention rights of the captured U soldiers?
In the sense of the Conventions saying they are required to, yes. In the sense of being practically held to that requirement, not necessarily. You don't get immunity for war crimes by simply claiming it's not a "war".
Furthermore, R hired mercenaries from yet another country, S. Those mercenaries, even though acting at R's behest and under R's instructions, are not a part of R's or S's armies. The third question: does the Geneva Convention apply to them too?
The Conventions give lesser protections to mercenaries. However, there are particular requirements for someone to be considered a "mercenary" for this purpose. Not everyone who might be called a "mercenary" in common parlance falls under this category:
Additional Protocol I defines a mercenary as a person who:
a) is specially recruited locally or abroad in order to fight in an armed conflict;
b) does, in fact, take a direct part in the hostilities;
c) is motivated to take part in the hostilities essentially by the desire for private gain and, in fact, is promised, by or on behalf of a Party to the conflict, material compensation substantially in excess of that promised or paid to combatants of similar ranks and functions in the armed forces of that Party;
d) is neither a national of a Party to the conflict nor a resident of territory controlled by a Party to the conflict;
e) is not a member of the armed forces of a Party to the conflict; and
f) has not been sent by a State which is not a Party to the conflict on official duty as a member of its armed forces.
https://ihl-databases.icrc.org/customary-ihl/eng/docs/v1_rul_rule108
Finally, volunteers from other countries are volunteering to help U's resistance, despite not being citizens of U and despite the volunteers' nations refusing to take any part in the conflict. Are those volunteers protected too?
If they are "volunteers" in the sense of not receiving payment, then they are not mercenaries under the Conventions' definition, and so are fully entitled to their protections, although again being entitled to be protection and being protected are not quite the same thing. Also, R could, hypothetically speaking, set up puppet governments that are, officially speaking, separate sovereign and which haven't signed the Conventions, and claim that these separate entities are killing the volunteers of their own accord.
