Electromagnetic radiation can indeed take away angular momentum. Thinking about this just from a classical point of view, the flux of energy carried by electromagnetic waves (in vacuum) is
$$ \vec{S} = \frac{1}{\mu_0} \vec{E} \times \vec{B}\ ,$$
where $\vec{S}$ is the Poynting vector.
The linear momentum associated with this is given by
$$ \vec{p} = \epsilon_0 \int \vec{E} \times \vec{B}\ dV\ = \frac{1}{c^2} \int \vec{S}\ dV\ , $$
where the integral is over a volume and the angular momentum is
$$ \vec{L} = \frac{1}{c^2} \int \vec{r} \times \vec{S}\ dV\ .$$
If the waves are not perfectly transverse - and the total fields of say a rotating magnetic dipole (the usual model for a neutron star magnetic field) are not - then angular momentum can be carried away.
Gravitational waves are unlikely to be a factor at all since gravitational wave radiation only arises from any time-dependent quadrupole mass moment and this will be zero if the neutron star has axial symmetry. aLIGO does now have observational limits on this for some pulsars - for example the rate at which the Crab and Vela pulsars are losing rotational kinetic energy now can only have contributions of $<0.2$% and $<1$% resepectively from gravitational wave radiation (Abbott et al. 2017).
It is of interest to look back at Ostriker & Gunn (1969) who give a detailed examination of the rotation-powered pulsar model, where they do consider that both a rotating magnetic dipole or a rotating mass quadrupole could be responsible for the spin-down of neutron stars (via electromagnetic and gravitational wave radiation respectively). Although it appears that the Crab and Vela pulsars are mainly braked by electromagnetic radiation now, it is worth noting that whilst the rate of change of angular momentum due to electromagnetic radiation scales as $\omega^3$, the equivalent losses for gravitational waves scale as $\omega^5$. That means that any gravitational wave spin-down could have been much more important at higher (by factors of $\geq 10$) spin rates than observed for the Crab and Vela pulsars now.