What is the general definition of symmetry in quantum mechanics?

Question

Consider a quantum system with Hilbert space $\mathcal{H}$ and Hamiltonian $H$. Let $G$ be a Lie group and $U$ a unitary representation of $G$ on $H$. What are the most general conditions that $H$, $G$ and $U$ must fulfil so that $G$ can be considered a symmetry of the system?

I know, for example, that $G$ is a symmetry if $$ [U(g), H] = 0\quad\forall g\in G\quad, $$ but this is not the most general condition. In a relativistic QFT the representations of Lorentz boosts do not commute with the Hamiltonian. In the case of Lorentz symmetry the criterium seems to be that $H$ must transform as the time component of a four-vector, as explained, for instance, in https://physics.stackexchange.com/a/568141/197448. But how does this generalise to arbitrary groups? Is there a broader notion of symmetry in QM for which both Lorentz symmetry in QFT and $[U(g), H]=0$ are just special examples?

For example, would it be valid to say that the system is symmetric under $G$ if $H$ is a linear combination of the generators of the representation $U$ (so that it transforms as a vector in the adjoint representation of $G$)? This would cover the case of Lorentz boosts in QFT if we take $G$ to be the Poincare group. It would also cover the cases where $H$ commutes with $U(g)$ if we let $G$ be the product of the time translations and some other group of transformations which do not involve time.

Answer 1

I would answer that there is no such a thing as "general definition". The problem is that symmetries in quantum mechanics have more than one inspiration and more than one purpose.

For example, rotation and translational symmetries are defined through the principle of correspondence and classical analogy. It's inspired by the intuition we have dealing with these symmetries in classical physics. The commutation with the Hamiltonian is just a matter of stability under time evolution. There would be no meaning to study a system using rotation symmetric states if it's evolution does not satisfies it. But, on the other hand, it can happen that you have a spontaneous symmetry break, and the symmetries of the Hamiltonian are not enough to explain what states are accessible for the system.

Permutation symmetry on the other hand are not based on correspondence or any analogy. It's not assumed by a matter of convenience or simplicity. It is assumed by first principles. It not only constrain the possible states the system could access, but also the possible Hamiltonians could describe it evolutions. Observe that it is defined in a very different way of the symmetries above, and we impose different conditions.

There are symmetries that we define through physical reasoning. For example Gauge symmetry. Since the Hamiltonian of a charged particle under the action of a EM field depends on the potentials, it would have no meaning to expect that the Hamiltonian should remain invariant under Gauge transformations. But in this case we impose that observable quantities should remain invariant and a gauge transformation of EM field translate to a local phase transformation in quantum theory.

There are symmetries that could not be represented by unitary operations like the time-reversal symmetry. But again we appeal to observable invariance.

In resume, it depends on the motivation behind the symmetry in question. Some transformations could be thought as physical transformations, like rotations and translations, but other ones are simply defined as mathematical properties the system should satisfy (permutation, time-reversing, etc., gauge transform.).

Answer 2

The problem with the hamiltonian based definition of symmetry is that a hamiltonian, unlike the lagrangian, is not a Lorentz covariant object. Symmetry operations can be defined by requiring that the lagrangian is invariant under their operation.