The physics is the same in both reference frames. What changes is what you call "energy". This is a subtle point, so feel free to ask as many questions as needed. I'll try to be as clear as possible.
In the end of the day, our theories are merely theories. They are descriptions of the physical world, but that's all. Descriptions. Hopefully they catch useful information and turn it into more useful information. To be sure we are talking about reality and not only some mambo jumbo, we need to perform experiments.
With this in mind, the question you posed becomes: if I try to measure the energy in this situation you described, which outcome will I get? And the answer is that it depends on how you perform your experiment.
I won't get to details on how to measure the energy, just assume you have an apparatus that can measure the energy of your system of interest. Maybe, to make it simpler, we can consider two balls, $A$ and $B$, colliding and then measuring their energy. Your question becomes to simply consider which energy is correct: that given in the frame of reference of $A$ or $B$?
This depends on how we set up the apparatus. Suppose we set up the energy-measuring-apparatus at rest with respect to $A$. Then this will give the result in $A$'s frame of reference. If we set it up at rest with respect to $B$, we get the result in $B$'s frame of reference. Hence, both energies are correct, but in different frames of reference. Why is this so? Because to get each of the energies, we had to set up different experiments. While we call both quantities "energy of the system", the physical processes required to measure them are different. Hence, they are not the same quantity. We just give them the same name.
The physics does not depend on the reference frame. It is the same regardless of the observer (at least in classical physics) and the reference frame. However, our description does depend on the reference frame. The numbers and such might change, but the damage to your body is the same in both reference frames. We might describe the damage in different manners, but we agree on the damage.
Important Remark
Notice that if you assume the wall gets to rest on your reference frame, then the reference frame in non-inertial and gets sort of weird. If you assume the inertial reference frame that still has the wall moving at a velocity $v$ after the collision, then you actually gain energy due to the force exerted by the wall on you. There's also some energy loss when we consider the wall has a finite mass and will move a little due to the collision. The energy analysis is a bit more complicated than you mentioned (notice, for example, that your computations do not conserve momentum if you assume the wall to have finite mass).