The Helstrom measurement is the measurement that has the minimum error probability when trying to distinguish between two states.
For example, let's imagine you have two pure states $|\psi\rangle$ and $|\phi\rangle$, and you wish to know which it is that you have. If $\langle\psi|\phi\rangle=0$, then you can specify a measurement with three projectors
$$
P_{\psi}=|\psi\rangle\langle\psi|\qquad P_{\phi}=|\phi\rangle\langle\phi|\qquad \bar P=\mathbb{I}-P_{\psi}-P_{\phi}.
$$
(For a two-dimensional Hilbert space, $\bar P=0$.)
The question is what measurement should you perform in the case that $\langle\psi|\phi\rangle\neq0$? Specifically, let's assume that $\langle\psi|\phi\rangle=\cos(2\theta)$, and I'll concentrate just on projective measurements (IIRC, this is optimal). In that case, there is always a unitary $U$ such that
$$
U|\psi\rangle=\cos\theta|0\rangle+\sin\theta|1\rangle\qquad U|\phi\rangle=\cos\theta|0\rangle-\sin\theta|1\rangle.
$$
Now, those states are optimally distinguished by $|+\rangle\langle +|$ and $|-\rangle\langle -|$ (you get $|+\rangle$, and you assume you had $U|\psi\rangle$). Hence, the optimal measurement is
$$
P_{\psi}=U^\dagger|+\rangle\langle+|U\qquad P_{\phi}=U^\dagger|-\rangle\langle-|U\qquad \bar P=\mathbb{I}-P_{\psi}-P_{\phi}.
$$
The success probability is
$$
\left(\frac{\cos\theta+\sin\theta}{\sqrt{2}}\right)^2=\frac{1+\sin(2\theta)}{2}.
$$
More generally, how do you distinguish between two density matrices $\rho_1$ and $\rho_2$? Start by calculating
$$
\delta\rho=\rho_1-\rho_2,
$$
and finding the eigenvalues $\{\lambda_i\}$ and corresponding eigenvectors $|\lambda_i\rangle$ of $\delta\rho$. You construct 3 measurement operators
$$
P_1=\sum_{i:\lambda_i>0}|\lambda_i\rangle\langle\lambda_i|\qquad P_2=\sum_{i:\lambda_i<0}|\lambda_i\rangle\langle\lambda_i|\qquad P_0=\mathbb{I}-P_1-P_2.
$$
If you get answer $P_1$, you assume you had $\rho_1$. If you get $P_2$, you had $\rho_2$, while if you get $P_0$ you simply guess which you had. You can verify that this reproduces the pure state strategy described above. What's the success probability of this strategy?
$$
\frac12\text{Tr}((P_1+P_0/2)\rho_1)+\frac12\text{Tr}((P_2+P_0/2)\rho_2)
$$
We can expand this as
$$
\frac14\text{Tr}((P_1+P_2+P_0)(\rho_1+\rho_2))+\frac14\text{Tr}((P_1-P_2)(\rho_1-\rho_2))
$$
Since $P_1+P_2+P_0=\mathbb{I}$ and $\text{Tr}(\rho_1)=\text{Tr}(\rho_2)=1$, this is just
$$
\frac12+\frac14\text{Tr}((P_1-P_2)(\rho_1-\rho_2))=\frac12+\frac14\text{Tr}|\rho_1-\rho_2|.
$$
