Well there are a few ways to think about it:
a) Spin-1/2 is an irreducible representation of $SU(2)$, so Schur's Lemma implies that there is no operator $H$ you can write down such that $[H, U] = 0$ for all $U \in \mathrm{SU}(2)$, except $H=0$.
b) More concretely, any state of a spin-1/2 can be visualized as a point on a sphere (the "Bloch sphere"), and $SU(2)$ acts on the sphere by rotation, and any point of the sphere can be rotated into a different point. This means that there is no state of a spin-1/2 that is $SU(2)$ invariant, so the ground-state must be at least twofold degenerate.
c) Another way is to define $X = e^{i(\pi/2)\sigma^x}$ and $Z = e^{i(\pi/2)\sigma^z}$ (these are $\pi$ rotations about the $x$ and $z$ axes). They anti-commute, $XZ = -ZX$. But any nondegenerate ground state $|\Psi\rangle$ must be invariant under both, so we must have $X |\Psi\rangle = \alpha |\Psi\rangle$ and $Z = \beta |\Psi\rangle$ for some phase factors, $\alpha$ and $\beta$. Then we find that $\alpha \beta |\Psi\rangle = XZ |\Psi\rangle = -ZX |\Psi\rangle = -\beta \alpha |\Psi\rangle$, which is a contradiction.