You are true for the case when the body is not in equilibrium with the surrounding i.e. when the temperature of the body is not the same as of the surrounding. Let me explain.
We start by considering the the Stephan Boltzmann Law:
$$P=a\sigma AT_b^4$$
where, $P$ is the energy emitted per unit time by the body at temperature $T_b$ emitting or absorbing from a surface area $A$.
Also, here $a$ is the absorptance of that body is defined as : when $Q$ amount of energy(heat) falls on that body and it absorbs $Q_a$ out of it,
$$a=\frac{Q_a}{Q}$$
Now, it is well known that for any body , the emittance and absorptance are same i.e $$a=e≤1$$
Also , $e=\frac{E}{E_b}$, where $E$ is the emissive power of the body and$E_b$ is the emissive power of the blackbody under same conditions.
So, the Stephan Boltzmann Law can be written as:
$$P=a\sigma AT^4=e\sigma AT^4$$
This shows that the law can be used to calculate even the absorption of energy by the body.
With that, let's begin the calculations:
CASE 1: The body is in equilibrium with the surrounding
If the temperature of the body and the surrounding are $T_b$ and $T_s$, then
$$T_b=T_s=T(say)$$
Energy emitted per unit time by the body is:
$$P_b=e\sigma AT^4 \tag{1}$$
And, we consider the surrounding to emit like a blackbody as it has abundant area to emit.So, the energy emitted per unit time per unit area (let denote by $E$) by the surrounding is:
$$E_s=\sigma T^4$$
This energy would be incident on the body and the body would absorb it accordingly its area and absorptance , so energy absorbed by the body per unit time:
$$P_a=a\sigma AT^4\tag{2}$$, where A is the area of the body in consideration.
Clearly, from equation$(1)$ and $(2)$, the energy emitted and absorbed per unit by a body in equilibrium with the surrounding is same (since, $a=e$ ) and hence it's temperature doesn't change.
CASE 2: When the body is not in equilibrium with the surrounding
Then, energy emitted by the body per unit time:
$$P_b=e\sigma AT_b^4$$
And,energy absorbed per unit time (using CASE 1 discussion):
$$P_a=a\sigma AT_s^4$$
So, net energy lost (if $T_b>T_s$) from the body per unit time is:
$$P_{net}=P_b-P_s=e\sigma A(T_b^4-T_s^4)$$
Then , the body will lose heat in this case and it's temperature would decrease that is governed by the Newton's law of Cooling.
Also , if $T_b<T_s$, the scenario would be the other way around and the body would gain heat from the surrounding and it's temperature would elevate.