We are interested in unitary/anti-unitary transformations of the underlying Hilbert space when considering symmetries because symmetry transformations are those transformations that preserve the angles and lengths between all points in the Hilbert space.
Recall the dot product
$$ \mathbf{a}\cdot \mathbf{b} =\left\|\mathbf{a}\right\| \left\| \mathbf{b}\right\|\cos(\theta_{ab}).$$
The dot product allow formal definitions of intuitive geometric notions, such as lengths and angles [0]. Each vector is a point in the vector space. Imagine we took the dot product of every pair of points (i.e., every pair of vectors) in the vector space. Then, we would a have a geometric notions of the how all the points in the vector space look like when considered as a whole. Maybe the points together appear like the constellation Sagitarius, Ursa Major, Cygnus, or your favorite euhedral crystal.
We now wish to focus our attention on those transformations that preserve the geometry of this shape (i.e., those that preserve lengths and angles). Well these transformations have a name: these are the symmetry transformations.
A Hilbert space is constituted by a set of vectors. Each of these vectors is a point in the Hilbert space. Now, it is the inner product (or something akin to it [see below]) that allows a formal definitions of intuitive geometric notions, such as lengths and angles. Looking at [1], we see that
Definition [Symmetry Transformation] A bijective ray transformation $T : \mathbf{P}(H) \to \mathbf{P}(H)$ is called a symmetry transformation if, and only if,
$$
T \underline{\Psi} \cdot T\underline{\Phi} = \underline{\Psi} \cdot \underline{\Phi},\tag{1}$$
for all $\underline\Psi, \underline\Phi \in \mathbf{P}(H)$.
We see that the inner product is preserved by the symmetry transformation for all pairs of points in the Hilbert space. If we initially have a collection of point in the Hilbert space that together appear like the constellation Sagitarius, Ursa Major, Cygnus, or your favorite euhedral crystal, then after transformation the collection of points still appear this way.
The last bit comes in now. With regard to Equation 1, the binary operation $\cdot$ is called the ray correlation or ray product. This is defined as
Definition [Ray product]
$$\underline{\Psi} \cdot \underline{\Phi} = \frac{\left|\left\langle\Psi, \Phi\right\rangle\right|}{\|\Phi\|\|\Psi\|} $$
With the defintion of ray product we rewrite Equation 1 as
$$
\frac{\left|\left\langle T\Psi,T \Phi\right\rangle\right|}{\|T\Phi\|\|T\Psi\|}
= \frac{\left|\left\langle\Psi, \Phi\right\rangle\right|}{\|\Phi\|\|\Psi\|}.
\tag{2}$$
Finally, consider what types of matrices can satisfy the relation in Equation 2. Upon reflection, you will find that of all the types of matrices that exist [2], only unitary transformations, antiunitary transformations, scaled unitary transformations, or scaled antiunitary transformations satisfy Equation 2 [3].
Bibliography
[0] https://en.wikipedia.org/wiki/Inner_product_space
[1] https://en.wikipedia.org/wiki/Wigner%27s_theorem
[2] https://en.wikipedia.org/wiki/List_of_named_matrices
[3] It may not be typical to include the scaled unitary transformation or scaled anti-unitary transformation. Yet, they satisfy the definitions given. In the case of scaled transformations, though angles are preserved, only the relative lengths are preserved. Thus, the aforementioned constellations would appear larger or smaller upon transformation. If we restrict the Hilbert spaces to include only the set of vectors with norm 1 (which is standard in quantum mechanics), then the scaled unitary transformations and the scaled anti-unitary transformations need be excluded from consideration.
