You need to change setting called rgba from rgb to bgr.
<fontconfig>
<match target="font">
<edit name="rgba" mode="assign">
<const>bgr</const> <!-- here -->
</edit>
</match>
</fontconfig>
I'm on Arch, and for me it was in /usr/share/fontconfig/conf.avail.infinality/10-base-rendering.conf. On your system it might be different, just grep for rgba in all *.conf files. A reboot/X restart might also be required.
EDIT
I just realized that a package upgrade might revert this back to the default. So the right location for this is etc/fonts/local.conf, you can create it if it doesn't exist.