I suggest the answer may be "yes, but no jury would convict him." As others have mentioned, wearing a shirt that may be a joke, unfunny as it may be to most people, it's hard to see how that would be considered sufficient to constitute consent, legally. OTOH, if someone did such a thing, and requested a jury trial (as one may, anywhere in the U.S., at least), I suggest that most juries would refuse to find the puncher guilty, along lines similar to "if he's stupid enough to wear that shirt, he deserves to get punched," or, at least, "someone who takes him up on it shouldn't go to jail."
NOTE that, again, as mentioned previously by others, this entire discussion is assuming there is no significant injury. If the assailant were to punch him "in the gut" hard enough to rupture his spleen, and the victim perished (this is how Harry Houdini died), some jail time would be expected.