The short answer: it isn't absent, it is only absent classically, and then only for certain initial conditions.
This is the old (and nowadays solved) puzzle of the electromagnetic arrow of time, which was a subject of a three-opinion paper in the early 20th century, with Einstein expressing the correct opinion, and two other people expressing other opinions. I don't remember the paper (or the other authors), I just remember Einstein's position.
Resolving the classical advanced wave paradox
The paradox with advanced waves is simply a thermodynamic issue. The central consistency problem with classical electromagnetism is that there is no thermal equilibrium--- you can't really imagine what an equilibrium state of the field looks like. It's a statistical field with probability for E and B given by the exponential of the field energy. At any finite temperature, this thermal equilibrium ensemble has infinite energy, it has $kT$ in every independent field mode, an infinite energy density overall. This is the ultraviolet catastrophe.
If you look at the classical vacuum of electromagnetism, no field, this state is completely ridiculous on thermodynamic grounds--- it can absorb arbitrarily large amounts of entropy, even entropy per unit volume, with no problem and go back to zero temperature, once it shunts the entropy into tiny wavelength modes. This means that the system of a mechanical charges interacting with a field is not thermodynamically sensible, all the entropy goes away into the field.
So there is an arrow of time in the classical theory which is simply defined by the direction in which the field entropy inexorably increases, as the field winds its way down to the smallest wavelengths. The reason we use retarded waves is because when we consider a system of charges radiating, the situation described by the advanced propagator is impossible to conceive: the classical field is set up in a conspiracy, just-so to have converging waves on every radiating body in the past. The reason it is impossible to concieve is exactly because the situation is so thermodynamically improbable.
The energy paradoxes you note aren't real. Both the advanced and retarded waves carry positive energy, the retarded wave carries it out, and the advanced wave carries it in. The radiation damping for a classical charged ball happens as the retarded wave emitted from one part of the ball acts on another part of the ball, and this happens at a slightly wrong time when the ball is accelerating, leading to friction. When you only have advanced waves, the friction turns into anti-friction, and you get speeding up of the ball. The ball is absorbing the radiation coming in and going faster.
But you can always consider forward in time retarded emission as a sum of advanced emission and a conspiratorial initial condition. If you imagine a normal situation with a charged ball, and it's shaking on a spring, and emitting retarded EM waves into the future, and you want to describe it with advanced waves, the shaking ball emits EM waves to the past. But this isn't the physical initial condition--- the condition is that you have no incoming radiation, so you need to set up extra waves in the initial conditions that exactly cancel all the past-emitted advanced waves. These extra waves exactly cancel the electromagnetic field emitted into the past by the particle, making the field at past null infinity zero, and produce the normal advanced wave of the particle going to future null infinity, and when you superpose the two fields, the emitted advanced wave and the conspiratorial initial condition wave, you get the exact same friction and slowing down from this field acting back on the particle.
So you can use any of the forms to do the classical theory, but any choice other than pure future-going wave is classically ridiculous, because it requires an infinitely improbable conspiracy in the initial conditions. Infinitely improbable because classical fields don't play well with thermodynamics.
The basic mathematical identity that demonstrates this is that the difference of the advanced and retarded wave solution is a solution to the source free Maxwell equations.
$$ G_F = G_R - G_A $$
So you are free to use either $G_R$ or $G_A$ as the response of a source, since the difference between them is a free-field solution, and can be added on and reproduced by appropriate free-field initial conditions.
I should point out here that if you have singular fields, if you allow the distributional thermal equilibrium classical electromagnetic field as an initial condition, then it wouldn't matter classically whether you used advanced or retarded waves. The distributional fields would be bringing energy in and out in equal amounts when the charged ball is close to thermal equilibrium. It's just that thermally equilibrated electromagnetic fields are distributional.
I should also say that classically, Feynman and Wheeler suggested to symmetrize the G function for the emission of a particle source by using half-advanced, half-retarded waves. For charged balls with a smooth charge density, this is equivalent to all retarded (or all advanced), any propagator is as good as any other, but the liked the time-symmetric formulation for some reason. The only reason they did this is to try to fix up the theory of the point-limit of classical electromagnetism, and this point limit is just not consistent.
As a side note, this point charge limit issue is completely fixed in GR, using charged black hole solutions. The charged black holes are not points, and they radiate and absorb classically free of any paradox related to zero size, which is incompatible with GR. But the black hole stays together gravitationally.
Quantum mechanics doesn't have this arrow
The full resolution to the electromagnetic arrow of time comes in quantum theory. Here, you can have a sensible finite energy thermal equilibrium state for the EM field, and in such a state, a particle at the same temperature as the field, interacting with the field, will not emit or absorb more. So the electromagnetic arrow of time is clearly the same as the thermodynamic arrow of time. The emission and absorption of photons does not carry an electromagnetic arrow of time.
The Feynman propagator for the quantum electromagnetic field in Feynman gauge is (in real space) proportional to
$$ {ig_{\mu\nu}\over (t^2 - x^2 +i\epsilon)} $$
The thing to note is that in the classical limit, you are integrating the thing over all possible times, and for long propagation distances, the result is concentrated on the light cone. You can figure out the delta-functions on the light cone from the Cauchy theorem fact that
$$ \delta(x) = {1\over 2\pi i} ({1\over x+i\epsilon} - {1\over x-i\epsilon})$$
and you can see that the propagator has a pole at $t=\pm|x|$ which gives delta-function contributions on the future and past light cone. This like a quantum half-advanced half-retarded solution, but it is determined by the Wick rotation which makes time slightly imaginary.
$$ G_F \propto g_{\mu\nu}( \delta(t^2 - x^2) + P.V {i\over (t^2 -x^2)}$$
and the P.V. part is an imaginary solution to the free Maxwell equations, while the delta-function part is a half-retarded half-advanced source propagator.
The retarded Leonard Weichert form is recovered from the delta-function part of the Feynman propagator, when you look at the part that goes into the future (times two), and ignore the part that goes into the past (this must be cancelled by classical zero field initial conditions, and this free field solution then goes on to double the retarded part). The additional imaginary part is not zero outside the light-cone, and is responsible for the Coulomb force in Feynman's description.
Understanding that this reproduces electromagnetism in the classical limit required sorting out the electromagnetic propagator business, which is implicit in all quantum field theory work post-dating Feynman's.
