We are given the following commutator expression: $$ \langle p | [x, p^2] | p \rangle = \langle p | x p^2 - p^2 x | p \rangle $$ Two possible results are given:
$$ \langle p | [x, p^2] | p \rangle = \langle p | xp^2 | p \rangle - \langle p | p^2x | p \rangle = 0 $$
$$ \langle p | [x, p^2] | p \rangle = 2i \hbar \langle p | p | p \rangle = 2i \hbar p \delta(0) $$
Which one is correct and why?
Neither is correct. Because $|p\rangle$ is not a normalizable state, you can't really compute the expectation value in this state. The correct quantity that you can consider is $$ \langle p'|[X,P^2]|p\rangle $$ Let us evaluate this two different ways, \begin{align} \langle p'|[X,P^2]|p\rangle &= \langle p'| X P^2 |p\rangle - \langle p'| P^2 X |p\rangle \\ &= p^2 \langle p'| X |p\rangle - p'^2 \langle p'| X |p\rangle \\ &= ( p^2 - p'^2 ) \langle p'| X |p\rangle \\ &= -i\hbar ( p^2 - p'^2 ) \frac{d}{dp} \langle p' |p\rangle \\ &= -i\hbar ( p^2 - p'^2 )\frac{d}{dp} \delta(p-p') \\ &= 2i\hbar p \delta(p-p') \end{align} Alternatively, \begin{align} \langle p'|[X,P^2]|p\rangle &= 2 i \hbar \langle p'|P|p\rangle \\ &= 2 i \hbar p \langle p'|p\rangle \\ &= 2 i \hbar p \delta(p-p') \end{align} All is well.
