If the dipole moment is not unique, how can the probability of absorption depend on the angle between the dipole moment and the electric-field?

Question

For systems with non-zero net charge, the dipole moment depends on the choice of origin (see Does the dipole moment depend on the choice of origin?). How then, can the probability of emission/absorption of light by an atom depend on the angle between the dipole moment and the electric-field component of the light wave?