This isn't quite how rainbows work. The standard explanation is that light bounces around inside each droplet, and getting reflected once, and exiting at an angle:
Image source
However, the real picture is a little bit more complicated. When sunlight hits a water droplet, the rays will
- refract when they come in,
- (partially) reflect back when they hit the back of the droplet, and then
- (partially) refract on their way out.
For each droplet, though, there are a bunch of rays hitting the droplet at different locations, and each of them will bounce around differently and exit at a different angle, so that the end result looks like this:
Because there is a reflection inside the droplet, the light is mostly sent backwards, and because there are two steps where refraction happens, the angles are a bit wonky. But here's the important thing: the angle at which the light exits increases, has a maximum, and then decreases again, a fact which is clearly visible by following the dots as they go down from the negative-$x$ axis, stop, and then go back up again.
This means that if the relative angle between the Sun, the droplet, and your head is smaller than a certain maximal angle $\theta_\mathrm{max}$, usually equal to about $\theta_\mathrm{max}\approx 42°$, then the droplet will appear bright to you (and, since this isn't an individual droplet but a misty conglomerate, the mist will have a diffuse glow), and if the angle is larger than that, then there will be no extra light going towards your eyes from those droplets.
In other words, then, this process will produce a disk that's bright, centered at the anti-solar point (i.e. where your eyes receive the on-axis reflections in the diagram above) and with diameter $\theta_\mathrm{max}\approx 42°$, and this is precisely what's observed, particularly when the rainbow happens against a darker background:
Image source
Notice, in particular, that the inside of the (primary) rainbow is much brighter than the outside.
Moreover, notice that the brightness of this disk increases as you go from the center to the edge: this is caused because the rays cluster at the turning point at $\theta_\mathrm{max}$ (notice in the ray diagram that there's many more dots in that region than there are near the axis). This clustering means that, for each color, the disk of light has a particularly bright edge, called a caustic.
So what's with the colors?
Although your diagram's geometry is off, as you correctly note, the standard diagram (the first figure in this answer) is kind of misleading, because for it kind of implies that for every red ray that hits your eyes, there will be another droplet at another angle sending a yellow ray (or green, blue, orange, indigo, and so on) on the same path ─ and that is indeed correct! This is what happens inside this disk of light.
The thing with this process, though, is that the maximal angle of aperture of the cone of light that's reflected by each droplet depends very sensitively on the refractive index of the water that makes up the droplet, and this refractive index also depends on the wavelength of the light, so that the size of the disk increases with the wavelength, with the red disk being the largest, then the orange, yellow, green, blue, indigo and violet being successively smaller.
This means that, at the edge of the disk produced by the red light, where it is the brightest, there is no light of other colours to compete with it, so the light looks red there.
A bit closer in, at the edge of the orange disk, there is no light of yellow, green, or blue colors, since those disks are smaller ─ and, also, the light from the red disk is fainter, because it's not at the maximal-brightness edge and the orange disk does have its maximum shine there. Thus, at that location, the orange light wins out, and the light looks overall orange.
And so on down the line: for each color in the spectrum, the edge of the disk is brighter than the larger disks, and the smaller disks don't contribute at all, so the edge of each disk shines with its respective color.
For further reading on the creation of rainbows see e.g. this excellent previous Q&A.
And finally, to address the subquestion: why aren't the different colours blurred together once they reach the retina? Basically, because in the human eye the retina is not exposed directly to the air $-$ the human eye is a fairly sophisticated optical re-imaging system, which uses a lens at the front of the eye to focus the incoming light onto the retina:
If this lens was not present (say, if the retina was where the dashed gray line is, and the lens had no effect) then you would indeed have light of different colors hitting every cell of the retina, and the retina would report a big jumbled uniformly-coloured mess to the brain.
Luckily, of course, the lens is present, and the effect of the lens is to re-focus the light, so that (at least, when the eye is focused at infinity) light coming in collimated from different angles will be focused at different lateral positions in the retina. Since the different colors are coming in at different angles, collimated from the rainbow which is effectively at infinity, this means that all the red light will be focused onto certain retina cells, and the blue light will be focused onto different retina cells at a different location, and so on.
It's extremely important to note that this has nothing to do with the fact that what you're seeing is a rainbow, and this re-imaging scheme coming from the focusing by the lens at the front of the eye (and the potential blurring problem we'd have if the lens wasn't present) is universal to seeing any objects at all, colored or not, rainbows or not.
For more details of how the eye works, see your favourite optics textbook.
