Motivation: the symmetries of a system as unitary operators and representations
In quantum mechanics, we usually consider two different kind of symmetries, as explained in Valter Moretti's answer there: (Coming from Wigner's Theorem): What is a Symmetry in QFT?, namely Wigner symmetries and Dynamical symmetries. Wigner symmetries are transformations acting on states, which are rays in the projective space of the Hilbert space of your system.
- A unitary operator automatically defines a Wigner symmetry transformation (on your projective space). This is explained in @ZeroTheHero reply.
- A converse of this fact is given by Wigner's theorem: we can associate to every symmetry transformation (in ray space) a unitary transformation on your Hilbert space (which is defined up to a complex phase factor).
This is the physical motivation, and is explained in What are unitary representations used for in physics? and Needs of unitary representation for QFT.
Decomposition of these representations
Now, the set of these symmetry operators forms a group, and one is often interested in the representation of this symmetry group, and more specifically in the simple representations, or irreducible representations of these representations. And in a way we are lucky, because unitary representations are easy to decompose, both for finite groups and compact groups. This is shown using the following theorems.
For finite groups, Maschke theorem is usually stated in two different ways:
- Every representation is equivalent to a unitary representation, i.e is related to a unitary representation by a change of basis. This is what @user1379857 says in his answer. This is actually a lemma combined with the fact that:
- Every unitary representation of a finite group is semisimple/decomposable/completely reducible, i.e is a direct sum of irreducible/simple representations (the orthogonal complement of every closed invariant subspace under the representation is again a closed invariant subspace).
Giving Maschke theorem, telling us that every representation of a finite group is semisimple, or completely reducible. This is what @Janusz Przewocki explains in his reply.
An equivalent but more general statement applies for compact groups (not necessarily finite), Peter-Weyl theorem:
A unitary representation of a compact group on a complex Hilbert space splits into an orthogonal direct sum of irreducible finite-dimensional unitary representations.
I am quite unsure about how I formulated these ideas, and I invite people to correct me wherever I might be wrong, and to add any extra useful information.