My question is how to prove $$\int|x\rangle \langle x| dx=1$$ where $|x\rangle$ is a eigenstate of a self-adjoint operator $X$ whose spectrum is continuous?
I want to have a rigorous mathematical proof. Any book recommendation is also appreciated, but please indicate the page of the proof relate to the above fact.
Let $A$ be a generic observable, that is, a self-adjoint operator on a Hilbert space $\mathcal H$. In general, there might be many values $\lambda$ in the spectrum of $A$ which do not have a corresponding eigenvector in $\mathcal H$. However, physicists often still choose to write the corresponding “eigenket” $|\lambda\rangle$, even if it's just a formal expression. For example, one often sees the integral: \begin{equation} \int f(\lambda) \; |\lambda\rangle\langle\lambda| \; \mathrm{d}\lambda \tag{1}\label{int1} \end{equation} Although the individual symbols “$|\lambda\rangle$” and “$\mathrm{d}\lambda$” aren't well-defined, the integral as a whole surprisingly is!
It is a deep result in functional analysis, the so called Spectral theorem, that for every normal operator $A$ there is a (operator-valued) measure $E_A$ with these properties:
For every operator $A$ there is precisely one such $E_A$ and we call it the spectral measure of $A$.
If you take an “infinitesimal slice” of the spectrum $\Omega = [\lambda, \; \lambda + \mathrm{d}\lambda]$, then the measure will return a projector to that “infinitesimal eigenspace” $E_A(\Omega) = |\lambda\rangle\langle\lambda|$. Therefore it makes sense to identify the integral \eqref{int1} with the rigorously defined Lebesgue integral with measure $E_A$: \begin{equation} \int f(\lambda) \; |\lambda\rangle\langle\lambda| \; \mathrm{d}\lambda \quad := \quad \int_{\mathbb C} f(\lambda) \; \mathrm{d}E_A(\lambda) \end{equation} Now, it should be obvious from the property “2.” of the spectral measure, that the integral $\int |\lambda\rangle\langle\lambda|\mathrm{d}\lambda$ is equal to the identity for every normal operator. Your question is just a specific case of this integral, with $A=\hat x$. Since the math can be a little too abstract, you'd maybe want to see the spectral measure for $\hat x$ – well, in position representation it's just the characteristic function: \begin{equation} \big( E_{\hat x}(\Omega) \; \psi \big)(x) = \chi_\Omega(x) \, \psi(x) = \begin{cases} \psi(x) \text{ for } x \in \Omega \\ \hspace{7pt} 0 \hspace{8pt} \text{ for } x \notin \Omega \end{cases} \end{equation}
As explained in the other answer, your "completeness relation" can be interpreted as the spectral theorem. Here's another approach:
The equation \begin{equation} \frac{1}{(2\pi)^n}\int|k\rangle\langle k|\,\mathrm{d}k=1 \end{equation} is usually used as a shorthand for the Fourier inversion theorem: \begin{equation} \psi(x)=\langle x|\psi\rangle=\frac{1}{(2\pi)^n}\int\langle x|k\rangle\langle k|\psi\rangle\,\mathrm{d}k=\frac{1}{(2\pi)^n}\int\mathrm{e}^{\mathrm{i}kx}\langle k|\psi\rangle\,\mathrm{d}k \end{equation}
We can give a precise meaning to the equation \begin{equation} \int|x\rangle\langle x|\,\mathrm{d}x=1 \end{equation} using Gelfland triples (see the answer to this question). However, this is not so popular. But \begin{equation} \langle\phi|\psi\rangle:=\int\overline{\phi(x)}\psi(x)\,\mathrm{d}x=\int\langle\phi|x\rangle\langle x|\psi\rangle\,\mathrm{d}x \end{equation} for all square integrable functions $\phi,\psi$.
$^1$In addition, if we regard the vector space of differentiable functions as the domain of the momentum operator $P$, $|k\rangle$ is a genuine eigenvector of $P$: $P|k\rangle=\hbar k|k\rangle$.
