The eardrum is damaged if it is stretched too far. The stretching is caused by an imbalance of pressure on either side of the eardrum. Blowing into the ear creates a stream of air that tries to enter the ear. When the blowing starts, some air enters the ear canal, which raises the pressure and pushes on the eardrum. However, the change in pressure is limited because the increased pressure in the ear pushes away any more air from entering. This is like trying to pour water into an already full cup. Human lungs are not capable of creating a powerful enough stream of air to greatly increase the pressure on the eardrum (only enough to be very annoying). Compressed air tanks with nozzles can do this, as can blanks fired from a gun, which is why caution needs to be exercised when using either of these or any other high-pressure system.
Now, why does sound damage the ear? When sound travels through air, the overall motion of air particles is back and forth, with no net movement of air. When a sound enters your ear canal, the amount of air in there does not change. The average pressure inside the ear canal is the same as the outside air. So, there is no impediment to sounds entering the ear. This means that louder sounds are not prevented from entering the ear and pushing on the ear drum. Even though the average pressure has not changed, the momentary peaks and troughs in the wave can be very much greater or very much lesser than ambient air pressure. Peak pressures push the ear inward, trough pressures pull the ear outwards. The louder the sound, the greater the peak pressure on the ear drum, and the greater chance for permanent damage--either by killing sensory cells or even tearing the ear drum.
As for different frequencies, a quick search yielded two papers. This somewhat gruesome one describes subjecting animals to very loud sounds and then examining the damage afterwards. Loud, low-frequency (125 Hz) sounds seemed to do damage to a large area of the cochlea, resulting in hearing loss in a wide range of frequencies. Loud high-frequency (4000 Hz) noise seemed to only damage a small area of the cochlea, resulting in hearing loss in a narrower range of frequencies. This paper with human subjects finds that loud (but not damaging) low-frequency sounds do affect a person's perception of sound for several minutes after the exposure.
It looks like all frequencies do damage to the ear when the noise is too loud. The only difference is whether the range of frequencies affected is wide or narrow. One last note: sounds you can't hear, whether ultrasonic or infrasonic, can do damage to your hearing if they are loud enough.