This is actually one of the things that is easier thought of using forces rather than energies. And some of the answer is that "fluid pressure is the same in all directions" is a simplification. It's a good simplification, but I think the steps they take might be causing you issues.
First off, let's consider pressure without gravity. Perhaps its fluid inside a container on the International Space Station. Fluid pressure is the statistical measure of the effect of myriad collisions between the fluid molecules and the environment (such as the tank holding the fluid). The reason why pressure is the same in all directions is because the molecules are all bouncing off each other unpredictably. Were there to be a difference in pressure on one side versus the other, then that environment (e.g the wall) would be applying an unbalanced force to the fluid, and statistically, the fluid would accelerate away from the wall. In practice, unless you're dealing with impulses near the speed of sound in the fluid, we ignore the statistical bit and instead just treat the fluid as one body which gets accelerated. So there's the first piece of the puzzle: if pressure were to be different from different directions, the fluid would accelerate. The rule that pressure is the same in all directions assumes no acceleration.
But what if there is a force? Take the same fluid container, and now put it on Earth. Now gravity is affecting every molecule. In this case, we find that the pressures must not be equal on all sides of the container. The bottom must have more pressure because it must apply a force to counteract the force of gravity -- else the fluid would need to accelerate. This force is computed exactly as you say: $P=\rho g h$, multiplied by area to yield $F=\rho g h A$ where $A$ is the area of the bottom surface.
So how do we get from "the forces on top and bottom are different" to "the forces are the same in all directions?" First we consider a small slug of fluid. There's no obligation that our "container" be the outside walls of something solid. It could just be a small volume of fluid, where the pressure pushing in from the nearby volume of fluid is exactly equal to the pressure pushing out from our own volume of fluid. The net behavior is the same as in the example enclosed by a physical tank: the fluid does not accelerate because there is no net force on the fluid. Were there to be a net force, it would quickly be distributed from the molecules closest to the source of the force to the rest of the body (the rate this happens is actually the speed of sound in that material).
What if we make this volume shallow? What if we take a thin slice parallel to the ground (perpendicular to the force of gravity?) Let's consider the difference between the pressures at the top and the pressures at the bottom. With a simple subtraction we see that $\Delta P = \rho g \Delta h$. So the smaller the difference in height between the top and bottom of your volume, the less the difference between the pressures.
When we say "fluid pressure is the same in all directions," we typically mean this to be true at a point. A point's height is 0, so we trivially get $\Delta P = 0$. The forces top and bottom are the same. And since the only asymmetry in the problem is the top-vs bottom difference due to gravity, it's not a big stretch to see that pressure should be the same in all directions (handwaving math a bit).
The weirdness is when you try to figure out how, if fluid pressure is the same in all directions, you somehow get a difference in pressure across top and bottom. It feels like adding 0 a bunch of times cannot possibly result in a positive value. To resolve this, one needs calculus. Calculus deals which such infinitesimal values rigorously, yielding the intuitive physics we expect. In calculus, we might write $dP=\rho g dh$, where the lower case "d" has the same meaning as $\Delta$ did in the earlier paragraphs, except it's an infintessimally small amount (or, even more rigorously, we'd write $\frac{dP}{dh}=\rho g$, but I chose to write the form that better mirrors the other equation). We can see that the "limit" of the difference in pressure as $dh$ goes to 0 is, in fact 0. This lines up with the idea that fluid pressure is the same in all directions. However, if we sum them all up across some real height, using what calculus calls an integral, we end up calculating $\Delta P=\int \rho g dh$, which accounts for the fact that we're summing an unlimited number of infinitely small slices.
If you haven't taken calculus, you may just have to take my word for it. It quite literally took millennia to invent (I like to measure it from Zeno's paradoxes being posed to when they finally were resolved by Newton and Leibnitz). Calculus does bridge these two apparently paradoxical definitions (pressure on a volume of fluid is greater on the bottom than the top vs. pressure is the same in all directions). That bridge just happens right near the $\Delta h = 0$ mark where math got really squirrely until calculus straightened it out.