In studying general relativity (GR) we learn that the Einstein-Hilbert (EH) action $S_{EH}=\int_{M}\mathrm{d}v_{g}R$ (where $\mathrm{d}v_{g}=\mathrm{d}^{4}x\sqrt{-g}$, with $g$ the metric tensor) is diffeomorphism invariant, i.e. under a diffeomorphsim $\phi^{\ast}:M\to M$ (where $(M,g)$ is the spacetime manifold): $\phi^{\ast}S_{EH}=S_{EH} \iff \mathcal{L}_{\xi}S_{EH}=0$, where $\xi$ is the vector field generating the diffeomorphism. As far as I understand it, one can show this is true as follows: $$\delta_{\xi}S_{EH}=\epsilon\mathcal{L}_{\xi}S_{EH}=\epsilon\int_{M}\big[\mathcal{L}_{\xi}(\mathrm{d}v_{g})R+\mathrm{d}v_{g}\mathcal{L}_{\xi}(R)\big]=\epsilon\int_{M}\big[\mathrm{d}v_{g}\nabla_{\mu}\xi^{\mu}R+\mathrm{d}v_{g}\xi^{\mu}\nabla_{\mu}R\big]\\ =\epsilon\int_{M}\mathrm{d}v_{g}\nabla_{\mu}(\xi^{\mu}R)=0\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\quad\;\;$$ assuming that $\xi$ has compact support, such that it vanishes on the boundary $\partial M$.
This is all well and good, but then how does one show that the action of a theory in the context of special relativity (SR) is not diffeomorphism invariant? Take, for example, electromagnetism (EM), which has the following action $$S_{EM}=-\int\mathrm{d}^{4}x\frac{1}{4}F^{\mu\nu}F_{\mu\nu}$$ where we have noted that $\sqrt{-\eta}=1$. Isn't it still the case that $\mathcal{L}_{\xi}(\mathrm{d}v_{\eta})=\mathrm{d}v_{\eta}\partial_{\mu}\xi^{\mu}$ and $\mathcal{L}_{\xi}(F^{\mu\nu}F_{\mu\nu})=\xi^{\mu}\partial_{\mu}(F^{\mu\nu}F_{\mu\nu})$? In which case it would follow that $\phi^{\ast}S_{EM}=S_{EM}$, which would contradict the statement that SR is not diffeomorphism invariant? My understanding of diffeomorphism invariance is that solutions to the Einstein Field equations (EFEs) $G=\text{Ric}(g)-\frac{1}{2}gR(g)=0$ are mapped into solutions, i.e. if $g$ is a solution to the EFEs, related to $g'$ via a diffeomorphism $g=\phi^{\ast}g'$, then $$G(g)=G(\phi^{\ast}g')=(\phi^{\ast}G)(g')=0.$$ However, SR only holds under isometries $\phi^{\ast}\eta=\eta\iff\mathcal{L}_{\xi}\eta =0$.
I have gotten myself very confused over the issue. Any help on the matter would be very much appreciated.
P.S. I get that so-called active (which we are discussing here) and passive diffeomorphisms are essentially the same in coordinate form (since any infinitesimal diffeomorphism can be expressed in coordinate form as an infinitesimal coordinate transformation), however, the two are very different (at least in a physical sense). There must be a way to understand this without resorting to the coordinate argument?!
