You are right. It does sound silly to say entropy is the reason time cannot run backward. But there is a bit more to this than meets the eye.
There are many things that work forward, but not backward. You can burn wood to ash, but not run a fire backwards to turn ash into wood. You can drop a cup on the floor and break it into pieces, but not throw pieces together to form a whole cup. These things tend to go in the direction or more disorder, or higher entropy.
Each of these is made of many microscopic chemical interactions between atoms. And those interactions can run both ways.
- Wood contains many molecules, such as CH4. In a fire, CH4 + 2 O2 -> CO2 + 2 H2O. This reaction can run backwards.
- A cup is made of glass, many SiO2 molecules stuck together. When a cup breaks, bonds between molecules are broken. Bonds between SiO2 molecules can be created.
Nobody has found any fundamental physics interaction that cannot run both ways.
So the question is why do large numbers of both-way interactions always form a one-way situation?
The only answer is probability. There are many ways for a large number of interactions to produce something more disordered than at the start, but only a few ways to make it more ordered. The odds are overwhelming in real situations that the outcome is more disordered.
But this is a little dissatisfying. It seems to say that the only reason time runs forward is because running backward is very improbable. People have been asking questions about this for a long time. E.G. Why does time not run backwards inside a refrigerator?