When we measure the speed of light we get the same answer in all directions. This is taken to undermine the aether or absolute motion hypothesis and give support to the proposal that the speed of light is invariant, from which derives the theory of special relativity.
But doesn't the fact that we only measure speed of light 'there and back' undermine this conclusion? Wouldn't we expect this result through an aether?
There-and-back measurements still show the effect of an aether, because you can compare the results in different directions. For example, there-and-back along the direction of motion would show a different speed from there-and-back across it. This was the approach taken in the Michelson-Morley experiment.
The bottom line is that this is the wrong question to ask. You don't ever prove an axiom in physics.
You're not quite right about the ether: while the first order effect cancels out in "there and back again" experiments, the second order effect doesn't, which is why the Michelson-Morley experiment stood a chance of testing for the ether. But Michelson-Morley wasn't the end of ether theory, because you could always add fixes to account for the results. For example, the famous Lorentz contraction in special relativity was originally invented for ether theory; the idea was that flowing ether physically squeezed objects smaller. Einstein just took effects like these more seriously.
Ether theory limped on for another 40 years, getting progressively more complicated as more results came in. Ether was not abandoned because it was disproven by experiments or because special relativity was proven by experiments, because this never happens. It was abandoned by the 1930s it could only explain experiments using tons of epicycles, while special relativity just worked perfectly out of the box.
That the one-way speed of light is not observable plays a large role in keeping various relativistically-correct aether theories alive. Like general relativity, these new aether theories locally reduce to special relativity in the absence of nearby massive objects. Unlike general relativity, these new aether theories have a preferred universal frame of reference, typically a frame co-moving with the cosmic microwave background radiation.
The $x$ direction is not any different physically than the $-x$ direction. So light travelling towards $x-> \infty$ should travel in the same way it travels towards $x-> -\infty$ wether theres an aether or not. What should make a difference is if we measure the speed of light in the direction the body that emitted the light is moving through the aether and any other direction, for example, the direction perpendicular to this movement to see the difference a bit easier. However no difference in arrival times was measured.
