The main force slowing down a fast moving bicycle is air drag.
When you're rolling down a slope, gravity is speeding up your bike while drag is slowing it down. The force of gravity is proportional to your mass (plus that of your bicycle) while the drag force is proportional to your frontal area and approximately proportional to the square of your speed relative to the air.
Yes, there's also some friction in the wheel bearings and some rolling drag between the wheels and the ground. Like gravity, rolling drag is typically also proportional to weight, so if it was the only thing slowing your bike down, you would indeed expect the weight of the rider not to matter.
However, at speeds above about 20 km/h or so air drag will exceed rolling drag, and even at lower speeds (unless you're going very slowly or driving in deep mud or snow, or unless your bike is super rusty and never oiled) the effect of air drag is still noticeable. (That's also why it's so much harder to drive a bicycle against the wind than with the wind behind you.)
Thus there is still a significant component to the total drag force that's not proportional to your mass but to your frontal area.
For two objects of similar shape and composition, such as two humans, mass is approximately proportional to volume, and thus to height cubed, while frontal area is only proportional to height squared. Thus, the ratio of gravitational to drag force (at a given velocity) for a bicyclist rolling down a hill is approximately proportional to their height, or to the cube root of their weight. (This is yet another example of a square–cube law.)
While this relationship is only approximate (since obviously human bodies are not all the same shape and two equally tall people can have different body weights), it still holds pretty well, especially when comparing a small child to an adult (or an adolescent).
So if e.g. your little sister is 120 cm tall and you're 180 cm, your gravity-to-air-drag ratio at a given speed is about 1.5 times that of your sister (and your equilibrium speed at which drag and gravity are balanced will be about $\sqrt{1.5} ≈ 1.2$ times that of your sister's, assuming a sufficiently steep hill for air drag to dominate rolling drag).
In other words, your dad was right. In an atmosphere, all else being equal, heavier objects really do fall (and roll) down faster.