Why the following interaction, in QFT, $$\displaystyle{\cal L}_{\rm int} ~=~\frac{\lambda}{4!}\phi^4$$ where $\lambda$ is positive, represents a theory that is unstable (or unbounded from below as it is usually said in textbooks).
How can can one show that explicitly?
$\cal H \sim \frac{\lambda}{4!} \phi ^4 $ (note that this term goes with the opposite sign in the Lagrangian). $\lambda$ has to be real because of unitarity and has to be positive because of vacuum-stability or, equivalently, since the Hamiltonian must be bounded from bellow. If $\lambda$ were negative, the larger the value of $\phi$, the more negative the Hamiltonian and therefore non-vacuum state (or ground state) could exist.
Now, at the quantum level, even if $\lambda$ is positive at the classical level the vacuum can be unstable or metastable. This can happen if the renormalization group drives the classical positive value to a negative one. To know if this is the case you have to know the complete theory. For instance, the measured value of the quartic self-coupling of the Higgs field leads to a non-stable vacuum at high enough energies. See this: Measured Higgs mass and vacuum stability
The reason attractive $\lambda \phi^4$ is unphysical is because a sufficient density of the $\phi$ particles has a self interaction which compensates for their mass-energy, so it is less energy to make a condensate of particles with a large density than to leave the vacuum alone.
This means that the vacuum will spontaneously decay by a monstrous explosion in a bubble into a state where the field is rolling off to plus or minus infinity. To see this, you can consider the energy of the classical field state
$$ \phi = C $$
which is
$$ a C^2 - \lambda C^4 $$
and is unbouded below. The problem with turning this rigorous (although it is completely persuasive) is that it is hard to write down wave-functions for quantum fields which are finite energy density. You usually use the path-integral to define this. While it is physically obvious that a wavefunction for the field $\phi$ which is peaked at a large constant value will have arbitrarily negative energy, constructing such a thing is a nightmare, because you need to control the short-distance correlations in the wavefunction to make sure that they don't have infinite energy in he ultraviolet, which is a pain.
But there is a simple way around this which is how everyone analyzes vacuum stability today, after Coleman. Use the path integral to show that there is an instanton which leads to vacuum decay. In this case, the Euclidean action is
$$ |\partial_t \phi|^2 + |\nabla \phi| + a \phi^2 - \lambda \phi^4 $$
You then note that this can be viewed as a classical system with a potential
$$ V(\phi) = -|\nabla\phi|^2 - a \phi^2 + \lambda \phi^4 $$
and the classical equations of motion for this thing has a closed zero energy solution where $\phi$ oscillates to a big value in a region, until it hits the $\lambda\phi^4$ wall, and comes back. The contribution of the instanton is to give a rate for nucleation out of the $\phi=0$ vacuum, in a way calculated in detail in Coleman's "Aspects of Symmetry". The essential point is that fluctuations around the zero energy solution have one negative eigenvalue, so a negative determinant, so that the square root of the determinant has an imaginary part, leading to slow oscillatory behavior in imaginary time, which is a decay rate in real time.
But I will use it in a much easier way to argue that the theory doesn't have a stable vacuum. Supposing the vacuum is stable, then the vacuum wavefunction is the probability of finding a given $\phi$ configuration in any constant time-slice in imaginary time, using the imaginary time action. This is a well-known path-integral relation.
But the imaginary time probability distribution for field values is of the form $e^{- S}$ where S is unbounded below! So the field $\phi$ has no ground state wavefunction which is normalized in the usual sense. I gave this answer, even though Drake's was sufficient, because you don't seem convinced.
