This is by no means a complete solution, but it may help in conjunction with other measures: There is an I/O scheduler much like a process scheduler, and it can be tweaked.
Most notably, you can actually choose amongst different schedulers:
~# cat /sys/block/sda/queue/scheduler
noop anticipatory deadline [cfq]
~# echo "deadline" > /sys/block/sda/queue/scheduler
~# cat /sys/block/sda/queue/scheduler
noop anticipatory [deadline] cfq
~#
deadline may help you get more strongly reproducible results.
noop, as its name implies, is insanely dumb, and will enable you to wreck absolute havoc on I/O performance with little effort.
anticipatory and cfq both try to be smart about it, though cfq is generally the smarter of the two. (As I recall, anticipatory is actually the legacy scheduler from right before the kernel started supporting multiple schedulers.)