Let's suppose Alice flies a rocket ship into a black hole. Poor Alice unknowingly crosses the event horizon. Too late, she realizes something is wrong as the tidal forces begin to grow, and she takes an evasive maneuver to try to get out of the gravitational field. Does she turn her ship left or right?
Note -- if you get bothered when free will gets brought into physics conversations, you can replace the above example with a robotic ship that will turn right or left with 50% probability depending on whether a radioactive atom decays or not, at some point after the ship crosses the horizon. Free will isn't a relevant issue here, Alice's spaceship is just meant to be a vivid example of information beyond the event horizon that is inaccessible to an outside observer in classical GR.
That information "exists" in spacetime, even though it's not directly apparent to an observer outside the black hole. Now, if I asked you "where" that information existed, you would probably say something like the information about whether Alice moved left or right "exists" on the trajectory of Alice's rocket -- in other words, the information exists in a region inside the event horizon. But that kind of intuition is precisely what leads to the degree of freedom counting where a black hole's entropy is proportional to its volume. If the holographic principle is correct, then that intuition must be wrong, and somehow the information about what Alice did after crossing the horizon gets encoded on the boundary of the black hole.
Now in classical GR, Alice's situation is strange, but not a problem. There's no real need to invoke the holographic principle in classical GR, so we might as well just assume the information about Alice's "right or left" choice is lost beyond the event horizon, inaccessible to outside observers who (like you said) never see the black hole form at all. Sure, there is some information in the spacetime that the outside observers will never be able to access, but, so what?
The issue arises when we include Hawking radiation. Then the black hole evaporates. So the outside observer will be able to access information about what happened inside the black hole, through the Hawking radiation. If unitary evolution of the quantum mechanical state is correct, then in principle it should be possible to evolve the quantum state back in time with unitary evolution and learn exactly what happened inside the black hole (however, this is completely impractical, since you would need to know a huge amount of information spread out in subtle ways over all of spacetime, and also you can never never learn the full state by measuring one copy of a quantum system). Various calculations of the entropy of the Hawking radiation show that it is proportional to the black hole's area. In order for the degrees of freedom of the black hole to be consistent with the entropy of the Hawking radiation, it must be the case that Alice's choice about whether to turn left or right did not just occur inside the black hole, but also must somehow have happened on the black hole horizon as well, even though Alice was nowhere near the horizon. That's certainly not something you would expect in classical GR, since an outside observer would never be able to learn anything that happened beyond the event horizon.
This is just a silly example, but maybe it helps elucidate why the area law is so weird.