- Answer of the mathematician:
Consider the Hilbert space $\mathcal{H}=L^2(R)$ and the selfadjoint unbounded linear operator $P$ defined on the dense domain $\mathcal{D}(P)=\{\psi \in \mathcal{H} | \psi \, \text{ is absolutely continuous and} \, \psi^\prime \in \mathcal{H} \}$ with $(P\psi)(x)=-i \hbar \psi^\prime(x)$ (see e.g. W. Thirring, Quantum Mathematical Physics, Springer, ch. 2.5). The function $\phi(x)=c_I(x)/\sqrt{2a}$ (where $c_I$ is the characteristic function of the interval $I=[-a,a]$) is an element of $\mathcal{H}$, but not of the domain $\mathcal{D}(P)$, as it fails to be absolutely continuous (which is needed for partial integration). The fact that $\phi^\prime(x)=0$ almost everywhere (except for the set of measure zero consisting of the two points $\pm a$) is not sufficient for $\phi$ being an element of the domain of $P$.
For the mathematical purist this might already suffice as an answer. For those who whish to look behind the scenes, consider a sequence $\{\phi_n\}_{n \in N}$ of unit vectors in $\mathcal{D}(P^2)$ converging to $\phi$ in the $L^2$ norm (it is easy to construct such functions by just smearing out the edges of $\phi$ at $\pm a$). The expection values $\langle \phi_n | P^2 \phi_n \rangle$ are now finite but unbounded in the limit $n \to \infty$.
As the Fourier transform $\mathcal{F}: \mathcal{H} \to \mathcal{H}$ defined by $\tilde{\psi}(p) = (\mathcal{F}\psi)(p)=\int_{-\infty}^{+\infty}dx\frac{e^{-ip x /\hbar}}{\sqrt{2 \pi \hbar}} \psi(x)$ is a unitary transformation, all observations obtained in the "x-representation" must have a counterpart in the "p-representation": the "momentum operator" $\tilde{P} = \mathcal{F} P \mathcal{F}^{-1}$ is now simply a multiplication operator $\tilde{P} \tilde{\psi}(p)=p \tilde{\psi}(p)$ with domain $\mathcal{D}(\tilde{P})=\{\tilde{\psi} \in \mathcal{H} | p \tilde{\psi}(p) \in \mathcal{H}\}$.
The Fourier transform of $\phi$ is now given by $\tilde{\phi}(p) =\sqrt{\frac{\hbar}{\pi a}} \frac{1}{p} \sin(p a/\hbar)$, being square-integrable and thus an element of $\mathcal{H}$ (as to be expected as a result of a unitary transformation). On the other hand, $\tilde{\phi}$ is not in the domain of $\tilde{P}$ as $p \tilde{\phi}(p)$ is not square-integrable over $R$ and we recover all the observations of the previous discussion.
- Answer of the theoretical physicist:
She/he is not frightened by leaving the cosy atmosphere of Hilbert space, remarking that $\phi^\prime (x) = (\delta(x+a)-\delta(x-a))/ \sqrt{2 a}$. A detailed discussion along these lines can be found in answers to related questions at Physics Stack Exchange (see e.g. Rectangular window $\psi$ wave-function and the calculus of $\langle p^2\rangle$ for it)
- Answer of the mathematical physicist:
She/he secretly performs a calculation like 2 but publicly presents the result like 1. This impresses the uninitiated.