Suppose Alice has a qubit $|\phi\rangle=\alpha|0\rangle+\beta|1\rangle$ and measures it. Bob knows the initial state but not the result of her measurement.
So after the measurement, Alice knows what state her qubit is in:
$$\rho_A=|0\rangle\langle0|$$
or:
$$\rho_A=|1\rangle\langle1|$$
I don't really understand what Bob thinks the state of the qubit is now though. Is it a mixed state? But he knows the qubit is not really in a mixed state because it must have collapsed when Alice measured it?
Yes, from Bob's perspective, the state Alice holds is: $$\rho=|\alpha|^2 |0\rangle\!\langle0| + |\beta|^2 |1\rangle\!\langle1|=\begin{pmatrix}|\alpha|^2 & 0\\ 0 & |\beta|^2 \end{pmatrix}.$$ As you've mentioned, this is indeed a mixed state (except if $|\psi\rangle$ is a computational basis state of course). However, you also said:
But he knows the qubit is not really in a mixed state because it must have collapsed when Alice measured it?
It is important to make the distinction between a state in superposition and a mixed state, something you can read about here.
But how do we get to the previous expression for $\rho$? Well, Bob knows that Alice got the outcome $|0\rangle$ with probability $|\alpha|^2$, and the outcome $|1\rangle$ with probability $|\beta|^2$. Putting things differently, the post-measurement state is $|0\rangle\!\langle0|$ with probability $|\alpha|^2$, and $|1\rangle\!\langle1|$ with probability $|\beta|^2$. That is, Bob has a classical uncertainty on Alice's state, which is directly described in the density matrix formalism by a convex combination of the possible outcomes, with the weights being the associated probabilities.
More generally, if Bob knows that the post-measurement state is some pure state $\left|\psi_i\right\rangle\!\left\langle\psi_i\right|$ with probability $p_i$, then from his perspective, the state Alice holds is: $$\rho=\sum_ip_i\left|\psi_i\right\rangle\!\left\langle\psi_i\right|.$$
Pure states are objective and mixed states are subjective. The classical probabilistic weights in a mixed state needn't come from quantum mechanics at all.
For example: Alice tells Bob that she is going to prepare a qubit in the state $|0\rangle$ or $|1\rangle$ but doesn't tell him which. Bob has been a close friend of Alice for years, and predicts from what he knows of her that there's a 2/3 chance that she will pick $|0\rangle$. He can and should represent the prepared qubit's state as the density matrix $\text{diag}(\frac23,\frac13)$, since that best represents his knowledge of the world, including the Alice part of it.
Another example: Bob watches Alice prepare a qubit in the $|{+}\rangle$ state and measure it in the computational basis. He can't see the result because the screen is facing away from him, but he does see Alice's reaction, and he happens to know she was hoping for one result over the other, and based on that information he decides there's a 2/3 chance that she saw $0$ on the screen. He should use the density matrix $\text{diag}(\frac23,\frac13)$, even though the Born-rule probabilities of the outcomes were $\frac12,\frac12$. It would only be correct to use the Born-rule probabilities if he had no information except that the measurement had taken place (such as during the fraction of a second after the measurement before Alice reacts).
