I can only speak from the point of view as a goldfish keeper, and only from my own experience. Bear in mind that goldfish are pretty much domesticated, fancy goldfish even more so and they'll have certain behaviours associated with that.
Banging on the tank is probably akin to noisy neighbours. From loud music to extensive DIY, depending on the extent of tapping/slapping/banging on the tank. Fish definitely respond/startle to loud noises outside their tank, and can act scared afterwards. It does depend on the fish, though. Conversely, they can also be "trained" to respond to a feeding bell (which makes monitoring individual fish in a pond or complicated tank setting easier). So if loud noises mean good things, then it's not necessarily bad for their mental health.
Some goldfish are also quite able to recognise different people and respond differently to the person that feeds them and the person that bangs on the tank. Some of my own goldfish can also recognise the net (which they despise).
Sad to say, but in places like a doctor's office, the fish who really can't stand the tank being banged on probably don't last that long. With a reasonable population turnover, you probably get left with a few very hardy (or insensitive) fish.
So, essentially, how much do you want your fish to like your toddler? How fond of individual fish are you? How noisy a fish-neighbour are you willing to be? And (IMHO most importantly) how stable is your tank? Tank damage was my main worry when I had toddlers. (Pre-children) I'd previously had a pane of glass smash under the gravel and noticed when the tank leaked a little. I rehomed my fish quickly, it could have been much, much worse. I'm led to believe that small weaknesses in brittle glass can rupture under a little stress, and that would be one big mess. We ended up using a Baby Dan baby den (other brands are probably available) which cordoned off the fish tank, the fire place and the TV until our children were less feral.