Thank you for the extra information, it was realy helpful in finding the likely reason for the swelling.
You say he has had this before, and the vet gave antibiotics to treat it and that this did not help.
Your cat was later given prednisone and that this helped your cat (prednisone is an anti-inflammatory steroid that supresses the body's immune response).
This makes me think your cat might have an allergy. You are living in Holland and if I am right it is early spring there now, so the trees have started to pollinate now.
It can be hard to find exactly what your cat is allergic to, but your vet can take a blood sample and test it for possible allergens that might cause your cats reaction.
You need to ask your vet if allergy might be the reason for the swelling, and ask if your vet thinks this might be something that needs to be treated.
Another possibility is your cat has been bitten by an insect and it's the reaction to it.
As for neutering that I've asked about in the comments - by neutering I mean having the cat's testicles removed, so he cannot reproduce and will have a lower level of testoterone in his body. Neutered males fight less and are calmer.