So, I wonder why is it usually said that photons do not interact, or hardly interact?
As far as we know photons do not directly interact with each other.
Mathematically, this is manifest in the fact that the equations of motion for electromagnetism are linear: given two sources A and B of electromagnetic radiation, the resulting EM field is precisely the same as if you take the field which would arise from source A alone, the field from source B alone, and just sum them together.
In other words
$$\text{Field from}(A+B) = \text{Field from}(A) + \text{Field from}(B) \, .$$
Yet another way to say this is that if a photon is moving along, the existence of a second photon has absolutely no influence over the first photon's path.
Things get more interesting when you have some charged particles involved.
If a photon moves past a charged particle, the electromagnetic field pushes that charged particle around.
The resulting motion of the charged particle creates new electromagnet radiation (a.k.a. new photons).
So, taking the full system into account, photons now actually do interact.
Of course, it's not a direct interaction; the interaction is mediated by the charges.
Still, you can say that when you have charges involved photons do wind up interacting with one another, and their dynamics is no longer linear.
In nonlinear optics people find materials (usually crystals or atoms in a resonant cavity) which interact strongly enough with photons to result in a strong effective photon-photon interaction.
This is pretty hard because light interacts pretty weakly with matter (charged particles) in general.
optical nonlinearities are inherently weak, because they are governed by photon–photon interactions enabled by materials.
This means what you guessed: the effective photon-photon interaction is weak because it's mediated by a weak photon-matter interaction.
In short, your guess is correct!