In this paper, there's the following sentence:
...and the factor 1/2 takes into account that the dipole moment is an induced, not a permanent one.
Without any further explanation. I looked through Griffiths' electrodynamics to see if this was a standard sort of thing, but couldn't find anything. I was thinking it might be because the field of the dipole itself opposes the inducing field, but that doesn't quite seem right for some reason.
The force on a dipole placed in an electrical field is given by $\mathbf{F} = (\mathbf{p}\cdot \nabla)\mathbf{E}$ (see, e.g., Griffiths, 3rd edition, eq. 4.5). Recall that, $$ \nabla(\mathbf{p}\cdot\mathbf{E}) = \mathbf{p}\times (\nabla\times \mathbf{E}) + \mathbf{E}\times(\nabla\times \mathbf{p})+(\mathbf{p}\cdot\nabla)\mathbf{E} + (\mathbf{E}\cdot\nabla)\mathbf{p} $$ Assume $\nabla\times \mathbf{E} = 0$ (I'll justify this at the end, trust me for now). If the dipole moment is a permanent one, $\mathbf{p} = \mathrm{const}.$, the second and fourth terms above are zero, and the expression for the force can be rewritten, $$ \mathbf{F}=\nabla(\mathbf{p}\cdot\mathbf{E})\quad\Rightarrow\quad U=-\int_{r_a}^{r_b} \mathbf{F}\cdot d\mathbf{r} = -\mathbf{p}\cdot \mathbf{E} |_{r_a}^{r_b} $$ However, if $\mathbf{p}$ is not a constant, but rather $\mathbf{p} = \alpha\mathbf{E}$, where $\alpha$ is the polarizability, the fourth term is not zero, and $$\begin{split} \nabla(\mathbf{p}\cdot\mathbf{E}) &= 2\alpha\mathbf{E}\times (\nabla\times \mathbf{E}) + 2\alpha(\mathbf{E}\cdot\nabla)\mathbf{E}\\ &= 0+2(\mathbf{p}\cdot\nabla)\mathbf{E} \end{split} $$ Therefore, $$ U=-\int_{r_a}^{r_b} \mathbf{F}\cdot d\mathbf{r} = -\int_{r_a}^{r_b}\frac{1}{2}\nabla(\mathbf{p}\cdot\mathbf{E})=-\frac{1}{2} \mathbf{p}\cdot\mathbf{E} |_{r_a}^{r_b} $$ The only outstanding issue is justifying the assumption $\nabla\times\mathbf{E}=0$. From the relevant Maxwell equation, $\nabla\times\mathbf{E}= - \frac{\partial \mathbf{B}}{\partial t}$. If your fields are static, $\frac{\partial \mathbf{B}}{\partial t} = 0$, and we're done.
In an optical trap, the application discussed in the paper above, the field is not static and we have to be a bit more careful. An optical trap is arranged by counterpropagating two identical laser beams. Assuming beam fronts are approximately planar, $$ \mathbf{E} = \mathbf{E_1} + \mathbf{E_2}\\ \mathbf{B} = \mathbf{B_1} + \mathbf{B_2} = (\frac{1}{c}\mathbf{\hat{k}}\times\mathbf{E_1}) + (\frac{1}{c}(\mathbf{-\hat{k}})\times\mathbf{E_2}) $$ If the beams are arranged so that they're in phase ($\mathbf{E_1} = \mathbf{E_2}$), we have $\mathbf{B} = 0$ at all times and so $\nabla\times\mathbf{E}=0$.
That's the math, then, but what's the intuition? To first order, there are two contributions to any change in the quantity $-\mathbf{p}\cdot \mathbf{E}$: the change in $\mathbf{p}$ at constant $\mathbf{E}$ and the change in $\mathbf{E}$ at constant $\mathbf{p}$. But there is actually no force opposing the first of these changes: strictly speaking, the energy of the dipole should just be the integral of the second of them. For a permanent dipole, the first change is zero, so we get away with writing the energy as $-\mathbf{p}\cdot \mathbf{E}$. But for an induced dipole, this is no longer the case. Linear polarizability gave us a factor of $1/2$, but more general relations between $\mathbf{p}$ and $\mathbf{E}$ may give you more complicated answers.
Because the black area is half the box below.
To explain: move the dipole from an area of no field to an area of field strength E. As you do, there's a force proportional to the dipole moment and to the gradient of E. For a fixed dipole, this force depends only on the gradient (horizontal dashed line). But for an induced dipole, the dipole moment depends on E and grows linearly as you move from zero field to full strength, so on average it is only half as strong during that movement (solid diagonal line).
