Here are some good resources to read about cat behavior:
For this answer I will recite the article The social structure of cat life
A cat’s territory consists of a core area, or den, where it feels secure enough to sleep, eat, play and potentially enjoy social interaction. This forms the hub of the territory which is the area beyond the core area that the cat actively defends against invasion from others. Beyond this lies the home or hunting range that constitutes the extent of the area over which the cat will roam.
This means every cat has their own territory and will fight off any cat comming into their core territory. From your description, I'd say your tomcat has declared your home to his territory and fights off all of his brothers.
Feral cats (cats which live without help from man) can and will form small colonies based around available food sources. This does not inevitably happen, and some will live singly, but it is not uncommon for small groups of co-operating females and kittens (matrilinear colonies) to develop.
[...]
Male cats are not commonly part of the small colonies, and they tend to exist on the periphery with large territories that may overlap several groups of females.
This means that female cats sometimes live together, but male cats usually don't. If they are neutered, they don't produce as much male hormones anymore and act more like a female, so neutered male cats may live together in a group.
Aggression most commonly occurs as male kittens reach sexual maturity and are excluded from the group, or between males and females as the tom ‘patrols’ his territory. [...] Mature males interact differently, and this is usually characterised by agonistic interaction, avoidance or tolerance, and not by friendly interactions.
This is what happened with your cats. One (the troublemaker) probably has more male hormones, so he is more dominant and aggressive to his siblings. He decided that he doesn't want to live with his family anymore, but that your house is his territory. Instead of leaving the house and letting the other cats live with you, he makes everyone else leave the house.