Let's suppose I push my desk into the wall. It doesn't move at all, but I get tired.
The sensation of getting tired has almost nothing to do with how much external work you do. A human is a very inefficient machine and tiredness is a very non-specific measurement. Whenever you are formulating a question in those terms it is likely not relevant to the underlying physics.
The easiest “fix” is to ask about a spring instead of a human. The amount of energy stored in a spring is easy to calculate exactly and such questions can be answered clearly on the basis of the underlying physics.
A spring can push your desk into a wall for as long as you want without getting tired. It’s energy depends only on the distance pushed and not on the time.
imagine there is a train in front of me and I'm on a train following behind pushing with a force of 100 Newtons for one minute. I can't imagine getting twice as tired if the train is moving a 40km/h instead of 20km/h.
Similarly, we will replace you with a spring compressed to provide 100 N force. As long as the distance between the train cars is constant then the 100 N will be constant and the energy in the spring will be constant.
As the trains move at 20 kph the spring will deliver a certain amount of mechanical power (work per unit time) to the front train. The rear train will deliver that same amount of power to the spring. The internal energy of the spring does not change and all energy that enters from the rear train leaves to the front train. Because the spring’s internal energy is constant the spring does not “get tired”.
As the trains move at 40 kph the situation is similar. Twice as much power (work/time) is indeed delivered to the front train from the spring and twice as much power is delivered to the spring from the rear train. The spring’s internal energy is again constant.