Consider the 1-D infinite square well problem (say of length L). The space of eigenfunctions all satisfy the boundary conditions that $\psi(0)=\psi(L)=0$ and the ground state wavefunction is given by
$\psi(x)= A*sin(\frac{\pi * x}{L})$ where $A$ is some factor.
Now if we take the momentum operation on our ground state we will get a function of the form $\cos\left(\frac{\pi x}{L}\right)$ which clearly doesn't satisfy the boundary conditions.
So why do we still call this as a "Linear operator" and not a " Linear transformation" ?
An operator ($\hat A$) on a Hilbert space ($\mathcal{H}$) is a linear map from a subset of the Hilbert space $\mathcal D(A)$ (called the domain of the operator) to the Hilbert space. This is a simply a definition in quantum mechanics. The linearity is included in the definition, as is the assertion that the operator can map to the entire Hilbert space, not necessarily the same subset of the Hilbert space as its domain.
As an example, suppose that the Hilbert space is the set of square integrable functions on $[0,1]$
$$
\mathcal{H} = L^2([0,1],dx)
$$
Then we can define an operator $\hat P$ by:
Specifying the operator's action as
$$
P: \psi(x) \to -i\hbar\frac{d\psi}{dx}\;;,
$$
which is linear, since the deriviative is linear; and
Specifying the domain of the operator to be the set of functions in the Hilbert space that are zero at the boundaries, and such that the derivative is also in the Hilbert space
$$
\mathcal{D}(P) = \left\{\psi\in\mathcal{H}\;|\; \psi(0)=\psi(1)=0\;\text{and}\; \frac{d\psi}{dx}\in\mathcal{H}\right\}\;.
$$
We still call $\hat P$ a "linear operator" even if it does not map $\mathcal D(P)$ to $\mathcal{D}(P)$, since by definition a "linear operator" can map some subset of the Hilbert space to the whole Hilbert space. Indeed, $\hat P$ does not map $\mathcal{D}(P)$ to $\mathcal{D}(P)$, as illustrated below.
As a concrete example, consider the function
$$
\psi_0(x)=\sin(\pi x)\;,
$$
which is in $\mathcal{H}$ and is in $\mathcal{D}(P)$.
But the function
$$
P\psi_0 = -i\hbar\pi\cos(\pi x)\;,
$$
is in $\mathcal{H}$ but is not in $\mathcal{D}(P)$. Nevertheless we still call $\hat P$ a "linear operator."
And since the image of the transformation doesn't beling to our Hilbert Space
The image of the transformation does belong to our Hilbert Space, since our Hilbert Space is $L^2([0,L], dx)$.
The boundary conditions are used in the definition of the domain of the operator $\hat P$, not in the definition of the Hilbert Space.