If vectors $|\vec{r}⟩$ and $|\vec{p}⟩$ are defined as
$$ \hat{\vec{r}} |\vec{r}⟩ = \vec{r} |\vec{r}⟩ \\ \hat{\vec{p}} |\vec{p}⟩ = \vec{p} |\vec{p}⟩ $$
then one can see that products like
$$ ⟨\vec{r}|\vec{r}⟩ \\ ⟨\vec{p}|\vec{p}⟩ $$
do not converge, which means that $|\vec{r}⟩$ and $|\vec{p}⟩$ do not belong to a Hilbert space as it is necessary that $\int |\psi|^2 < \infty$. So how should one understand these objects and what space do they belong to?
Physicists usually generously relax the condition that the norm should be finite and they sometimes say that $|\vec r\rangle,|\vec p\rangle$ belong to the "Hilbert space". It's exactly the same "generous" language that allows physicists to say that $\delta(x)$ is a "function", the delta-function, even though its values around $x=0$ are infinite or "ill-defined", so it's not really a function.
Rigorously mathematically, those objects don't belong to the Hilbert space (because their norm isn't finite) but (in analogy with the concept of "distributions" that include "functions" like $\delta(x)$) there exists a mathematical concept that includes such non-normalizable vectors, the rigged Hilbert space.
The idea that is that one may define a subspace $\Phi$ of the Hilbert space that contains smooth enough functions. The dual space to the Hilbert space $H$ is $H$ itself. But the dual space to the subspace $\Phi$ is $\Phi^*$ that is, on the contrary, larger than $H$, and it's this $\Phi^*$ that is called the rigged Hilbert space and that contains objects such as $|\vec r\rangle$. Formally mathematically, the whole triplet $(\Phi^*,H,\Phi)$ is usually referred to as "the rigged Hilbert space".
Just to add to what's been said and to address the comment by @LightnessRacesinOrbit, you can indeed have perfectly good functions that are only defined on some subset of the real numbers. The reason the Dirac delta is not a function is not that it is only well-defined for some real numbers and not others; in fact, it is not defined as a function anywhere! It is instead defined as a sort of "hypothetical" function, call it $\delta(x)$, that, if it existed, would satisfy the property that when you integrated it against any other (sufficiently nice) function $f(x)$, you would get
$\int_{\mathbb{R}} \delta (x) f(x) \, dx = f(0) $.
There is no well-defined function that actually does this, but there are well-defined functions that come as close as you want: for example, functions we can call $\varphi_\varepsilon$ that are equal to zero everywhere except in a small interval $(-\varepsilon, \varepsilon)$ around zero, and are equal to $\frac{1}{2\varepsilon}$ for $x \in (-\varepsilon, \varepsilon)$. Notice that if $f(x)$ is smooth, then
$\int_{\mathbb{R}} \varphi_\varepsilon (x) f(x) \, dx \approx f(0) $.
So we can get close to having a function that behaves the way we want a Dirac delta to, but not quite (that equality is only approximate, because $f(x)$ may vary over the interval $(-\epsilon,\epsilon)$). Rather than a function, the Dirac delta is really a linear operator that takes in a smooth function and gives you back the value of that function at $x=0$. Notationally we still pretend that there is a "function" $\delta(x)$ that embodies this operation when you integrate other functions against it, but more properly it is an element of the so-called "dual space" to the space of smooth functions that are zero outside a bounded interval - that is, it is an element of the space of linear operators on such functions (as a technicality: note that it is a continuous linear operator with respect to the uniform norm on smooth functions of compact support).
This is to supplement @Idempotent's answer addressing the "functionness" of $\delta$ functions (discussed in the comments); I'd make it a comment but I don't have the reputation here for that yet. The quote by Dirac, below, is potentially of direct interest to the OP as it regards the interpretation of some of Dirac's notation.
Nicholas Wheeler has a helpful note, "Simplified Dirac Delta," in the "Miscellaneous Math" section of his online notes, here: Nicholas Wheeler | Documents. He emphasizes that the $\delta$ function only really makes sense in the context of integrals, describing Schwartz's generalized functions (aka distributions, including the Dirac $\delta$) as new mathematical entities "that live always in the shade of an implied integral sign." He quotes Dirac on this point:
There are a number of elementary equations which one can write down about $\delta$ functions. These equations are essentially rules of manipulation for algebraic work involving $\delta$ functions. The meaning of any of these equations is that its two sides give equivalent results [when used] as factors in an integrand.
So Dirac himself realized these entities were not "real" functions, but rather a kind of formal device. A few years after Dirac introduced them, Laurent Schwartz put them on a firm mathematical footing with the notion of distributions (distinct from but related to the notion of distributions in probability theory) or generalized functions.
