In order to understand what kind of representations you need, you have to look at a fundamental theorem about symmetries in quantum physics: Wigner's theorem.
A Hilbert space $\mathcal H$ is technically not the space of pure states, as there is a redundancy. Two vectors related by a complex number
$$ \psi' = \lambda\psi, \qquad \lambda\in\mathbb C$$
describe the same state. Or if you work with normalized states, then $\lambda = e^{i\theta}$. Therefore, you need to identify $\psi'\sim \psi$ all such vectors and what you get is a ray
$$ [\psi] = \{\psi'\in S\mathcal H \, \big| \, \exists e^{i\theta}\in U(1), \text{ s.t. } \psi' = e^{i\theta}\psi\}. $$
Here by $S\mathcal H$ I mean the submanifold of unit vectors (sphere in $\mathcal H$).
Alternatively, a pure state can be described by a density matrix
$$ \rho_\psi = \frac{|\psi\rangle\langle\psi|}{\langle\psi|\psi\rangle}.$$
The space of such rays is a projective space $\mathbb P\mathcal H$, and thus not linear. Now, on this space you can define the transition probability between two different states (rays), which is nothing but the Born rule
$$ P\left([\psi]\rightarrow [\phi]\right) = |\langle\psi|\phi\rangle|^2 = tr(\rho_\psi\rho_\phi).$$
A symmetry can now be defined as a bijection on the space of physical states $\mathbb P\mathcal H\rightarrow \mathbb P\mathcal H$ that preserves the transition probabilities. The set of such symmetries forms a group $G$.
However, we prefer to work with the (linear) Hilbert space $\mathcal H$ than the space of rays $\mathbb P\mathcal H$. The question is now, how should such symmetries be represented on the Hilbert space when lifted up from $\mathbb P\mathcal H$?
Wigner's theorem states that symmetries must be represented as projective unitary or anti-unitary representations on $\mathcal H$.
The projective part is why we have to allow spinor representations (fermions), for example why we work with $Spin(3)=SU(2)$ (universal cover of SO(3)) when we really have $SO(3)$ symmetry. Or with $\mathbb R^4\rtimes Spin(1,3)=\mathbb R^4\rtimes SL(2,\mathbb C)$ when we have Poincare symmetry $\mathbb P = \mathbb R^4\rtimes SO(1,3)$.
Anti-unitary representations only really show up when considering time-reversal symmetry and it's usually a combination of complex conjugation and regular unitary representation.
For topologically compact Lie groups like $SO(3)$, we have finite dimensional unitary representations. For example we can decompose the infinite dimensional space of square integrable functions on the sphere
$$L^2(S^2) = \{\,\psi:S^2\rightarrow\mathbb C,\,\big| \, \int d\theta d\phi\,|\psi(\theta,\phi)|^2<\infty\,\} $$
into infititely many finite-dimensional spaces
$$ L^2(S^2) = \bigoplus_{s=0}^\infty \mathcal H_s,$$
where each space $\mathcal H_s$ corresponds to the irreducible spin-$s$ representation and spanned by Spherical Harmonic functions.
For topologically non-compact groups like the Lorentz group $SO(1,3)$, all unitary representation are infinite dimensional. There are finite dimensional representations, but they are not unitary (or anti-unitary).
