This question is investigated here. If we take the refugee question off the table for a moment, we need only consider the
Third Geneva Convention Relative to the Treatment of Prisoners of War which is the international law that governs prisoners of war.
No provision of that treaty requires a detaining power to, in any fashion, consult with a prisoner regarding selection for a prisoner exchange. Therefore, "I'd rather stay here" has no legal force.
Article 7 may be an important procedural requirement for a POW: "Prisoners of war may in no circumstances renounce in part or in entirety the rights secured to them by the present Convention, and by the special agreements referred to in the foregoing Article, if such there be". The rationale behind this is the presumption that the prisoner may well have been coerced into some renunciation, so Art 7 makes such coercion ineffective. Art. 21 says that a POW can be interned (etc.) but also
Prisoners of war may be partially or wholly released on parole or
promise, in so far as is allowed by the laws of the Power on which
they depend. Such measures shall be taken particularly in cases where
this may contribute to the improvement of their state of health. No
prisoner of war shall be compelled to accept liberty on parole or
promise.
(This does not refer to repatriation, it refers to "letting them roam about").
The one exception of note regarding the principle that prisoners get no say is Art. 109 which states that
No sick or injured prisoner of war who is eligible for repatriation
under the first paragraph of this Article, may be repatriated against
his will during hostilities
where the first-paragraph condition is being a seriously wounded or
seriously sick prisoners of war who is has been treated to be fit for travel.
The convention states an unconditional requirement to repatriate prisoners at the end of hostilities, so we must assume that since this is a prisoner exchange that hostilities persist. As the analysis article states,
Those benefiting from the protective status of prisoners of war are,
in principle, not entitled to the international protection granted to
refugees. However, prisoners of war who refuse to be repatriated at
the end of the hostilities can apply for asylum in the detaining
State....
The 1951 Refugee Convention is not applicable to ex-combatants during
their period of internment
Thus a line can be draws between prisoner exchanges and repatriation at the cessation of hostilities. However,
From a refugee law perspective, a former combatant in a neutral State
who has been disarmed and interned can be a refugee
In summary, in the case of a POW held by an enemy state (not a neutral state), one who is not seriously injured, and while hostilities exist, there is no clear path to refuse repatriation. The refugee door is open at the cessation of hostilities, where one asks if a person is "includable" (might be persecuted) and is not "excludable" (war criminals are excluded). Being an enemy combatant is a legitimate ground for exclusion (there is a long exegesis of the 1951 Refugee Convention here, see Art. 9).
