I'm wondering if I were to fall from a 100 meter tall building into a 100 meter deep hole of balloons, what would happen?
This is kind of a devil-in-the detail thing. I don't think you'd get a definitive answer short of testing it (and I don't recommend that :-) ).
What are the balloons made of ? At what pressure ? How big ? How are they packed ? I means those questions are the tip of the iceberg.
Stunt men normally use inflated devices to slow them down relatively gently. These devices are not, however, sealed. They are specifically designed to allow gas to get out, which allows the device to deform slowly and absorb energy. Balloons are sealed. But note the height ratings for those devices on that link - nowhere near the 100m mark. The record (AFAIK) for a free fall into an airbag like this was by Dar Robinson and was some 95m, so a 100m jump might be possible by a pro into ideal safety equipment.
But balloons ... ?
At a guess I'd say best case scenario, the balloons initially burst as you hit them. This would soak up some energy from your fall without (maybe ?) injuring you (much ?). With luck you'd walk away, with less you'd be injured. You'd need some luck to get away with this because at some point the energy you still have won't be enough to break the balloons and then they're likely to act like solid, non-deformable rocks packed together tightly. You'd need to have slowed down enough before that to not suffer serious injury.
Worse case scenario. Balloons are built of something strong and at high pressure and packed tight. That would be little different from hitting the ground as they probably could not deform to absorb energy and they'd present a basically solid surface. Death probably, certainly extreme injury.
Would I suffer irrecoverable injuries and die a horrible death? Or would the balloons stop my fall slowly?
Maybe. Devil in the details. But I'd be inclined to say quite probably the former and less likely the later.
If you were injured it would greatly depend on the availability of very quick medical assistance.
If a 100 meter deep hole of balloons doesn't work, how deep would it have to be?
Impossible to say. Testing this sounds like a job for Beaker.