I) The solution to the time-dependent Schrödinger equation (TDSE) is
$$ \Psi(t_2) ~=~ U(t_2,t_1) \Psi(t_1),\tag{A}$$
where the (anti)time-ordered exponentiated Hamiltonian
$$\begin{align} U(t_2,t_1)~&=~\left\{\begin{array}{rcl}
T\exp\left[-\frac{i}{\hbar}\int_{t_1}^{t_2}\! \mathrm{d}t~H(t)\right]
&\text{for}& t_1 ~<~t_2 \cr\cr
AT\exp\left[-\frac{i}{\hbar}\int_{t_1}^{t_2}\! \mathrm{d}t~H(t)\right]
&\text{for}& t_2 ~<~t_1 \end{array}\right.\cr\cr
~&=~
\underset{N\to\infty}{\lim}
\exp\left[-\frac{i}{\hbar}H(t_2)\frac{t_2-t_1}{N}\right] \cdots\exp\left[-\frac{i}{\hbar}H(t_1)\frac{t_2-t_1}{N}\right]\end{align}\tag{B} $$
is formally the unitary evolution operator, which satisfies its own two TDSEs
$$ i\hbar \frac{\partial }{\partial t_2}U(t_2,t_1)
~=~H(t_2)U(t_2,t_1),\tag{C} $$
$$i\hbar \frac{\partial }{\partial t_1}U(t_2,t_1)
~=~-U(t_2,t_1)H(t_1),\tag{D} $$
along with the boundary condition
$$ U(t,t)~=~{\bf 1}.\tag{E}$$
II) The evolution operator $U(t_2,t_1)$ has the group-property
$$ U(t_3,t_1)~=~U(t_3,t_2)U(t_2,t_1). \tag{F}$$
The (anti)time-ordering in formula $(\text{B})$ is instrumental for the (anti)time-ordered exponential $(\text{B})$ to factorize according to the group-property $(\text{F})$.
III) The group-property $(\text{F})$ plays an important role in the proof that formula $(\text{B})$ is a solution to the TDSE $(\text{C})$:
$$\begin{array}{ccc} \dfrac{U(t_2+\delta t,t_1) - U(t_2,t_1)}{\delta t} &\stackrel{(\text{F})}{=}&
\dfrac{U(t_2+\delta t,t_2) - {\bf 1} }{\delta t}U(t_2,t_1)\cr\cr
\downarrow & &\downarrow\cr\cr
\dfrac{\partial }{\partial t_2}U(t_2,t_1)
&& -\dfrac{i}{\hbar}H(t_2)U(t_2,t_1).\end{array}\tag{G}$$
Remark: Often, the (anti)time-ordered exponential formula $(\text{B})$ does not make mathematical sense directly. In such cases, the TDSEs $(\text{C})$ and $(\text{D})$, along with boundary condition $(\text{E})$, should be viewed as the indirect/descriptive defining properties of the (anti)time-ordered exponential $(\text{B})$.
IV) If we define the unitary operator without the (anti)time-ordering in formula $(\text{B})$ as
$$ V(t_2,t_1)~=~\exp\left[-\frac{i}{\hbar}\int_{t_1}^{t_2}\! \mathrm{d}t~H(t)\right],\tag{H}$$
then the factorization $(\text{F})$ will in general not take place,
$$ V(t_3,t_1)~\neq~V(t_3,t_2)V(t_2,t_1). \tag{I}$$
There will, in general, appear extra contributions, cf. the BCH formula. Moreover, the unitary operator $V(t_2,t_1)$ will in general not satisfy the TDSEs $(\text{C})$ and $(\text{D})$. See also the example in section VII.
V) In the special (but common) case where the Hamiltonian $H$ does not depend explicitly on time, the time-ordering may be dropped. Then formulas $(\text{B})$ and $(\text{H})$ reduce to the same expression
$$ U(t_2,t_1)~=~\exp\left[-\frac{i}{\hbar}\Delta t~H\right]~=~V(t_2,t_1), \qquad \Delta t ~:=~t_2-t_1.\tag{J}$$
VI) Emilio Pisanty advocates in a comment that it is interesting to differentiate eq. $(\text{H})$ w.r.t. $t_2$ directly. If we Taylor expand the exponential $(\text{H})$ to second order, we get
$$ \frac{\partial V(t_2,t_1)}{\partial t_2}
~=~-\frac{i}{\hbar}H(t_2) -\frac{1}{2\hbar^2} \left\{ H(t_2), \int_{t_1}^{t_2}\! \mathrm{d}t~H(t) \right\}_{+} +\ldots,\tag{K} $$
where $\{ \cdot, \cdot\}_{+}$ denotes the anti-commutator. The problem is that we would like to have the operator $H(t_2)$ ordered to the left [to compare with the TDSE $(\text{C})$]. However, resolving the anti-commutator may, in general, produce unwanted terms. Intuitively, without the (anti)time-ordering in the exponential $(\text{H})$, the $t_2$-dependence is scattered all over the place, so when we differentiate w.r.t. $t_2$, we need afterwards to rearrange all the various contributions to the left, and that process generates non-zero terms that spoil the possibility to satisfy the TDSE $(\text{C})$. See also the example in section VII.
VII) Example. Let the Hamiltonian be just an external time-dependent source term
$$ H(t) ~=~ \overline{f(t)}a+f(t)a^{\dagger}, \qquad [a,a^{\dagger}]~=~\hbar{\bf 1},\tag{L}$$
where $f:\mathbb{R}\to\mathbb{C}$ is a function. Then, according to Wick's Theorem
$$ T[H(t)H(t^{\prime})] ~=~ : H(t) H(t^{\prime}): ~+ ~C(t,t^{\prime}), \tag{M}$$
where the so-called contraction
$$ C(t,t^{\prime})~=~ \hbar\left(\theta(t-t^{\prime})\overline{f(t)}f(t^{\prime})
+\theta(t^{\prime}-t)\overline{f(t^{\prime})}f(t)\right) ~{\bf 1}\tag{N}$$
is a central element proportional to the identity operator. For more on Wick-type theorems, see also, e.g. this, this, and this Phys.SE posts. (Let us, for notational convenience, assume that $t_1<t_2$ in the remainder of this answer.) Let
$$ A(t_2,t_1)~=~-\frac{i}{\hbar}\int_{t_1}^{t_2}\! \mathrm{d}t~H(t)
~=~-\frac{i}{\hbar}\overline{F(t_2,t_1)} a -\frac{i}{\hbar}F(t_2,t_1) a^{\dagger} ,\tag{O}$$
where
$$ F(t_2,t_1)~=~\int_{t_1}^{t_2}\! \mathrm{d}t ~f(t). \tag{P}$$
Note that
$$
\frac{\partial }{\partial t_2}A(t_2,t_1)~=~-\frac{i}{\hbar}H(t_2), \qquad
\frac{\partial }{\partial t_1}A(t_2,t_1)~=~\frac{i}{\hbar}H(t_1).\tag{Q} $$
Then the unitary operator $(\text{H})$ without (anti)time-order reads
$$\begin{align}
V(t_2,t_1)~&=~e^{A(t_2,t_1)}
\\
~&=~\exp\left[-\frac{i}{\hbar}F(t_2,t_1) a^{\dagger}\right]\exp\left[\frac{-1}{2\hbar}|F(t_2,t_1)|^2\right]\exp\left[-\frac{i}{\hbar}\overline{F(t_2,t_1)} a\right].\tag{R}
\end{align}$$
Here, the last expression in $(\text{R})$ displays the normal-ordered for $V(t_2,t_1)$. It is a straightforward exercise to show that formula $(\text{R})$ does not satisfy TDSEs $(\text{C})$ and $(\text{D})$. Instead, the correct unitary evolution operator is
$$\begin{align}
U(t_2,t_1)~&\stackrel{(\text{B})}{=}~T\exp\left[-\frac{i}{\hbar}\int_{t_1}^{t_2}\! \mathrm{d}t~H(t)\right]
\\~&\stackrel{(\text{M})}{=}~:\exp\left[-\frac{i}{\hbar}\int_{t_1}^{t_2}\! \mathrm{d}t~H(t)\right]:~ \exp\left[\frac{-1}{2\hbar^2}\iint_{[t_1,t_2]^2}\! \mathrm{d}t~\mathrm{d}t^{\prime}~C(t,t^{\prime})\right]
\\ ~&=~ e^{A(t_2,t_1)+D(t_2,t_1)}~=~V(t_2,t_1)e^{D(t_2,t_1)}\tag{S},
\end{align}$$
where
$$ D(t_2,t_1)~=~\frac{{\bf 1}}{2\hbar}\iint_{[t_1,t_2]^2}\! \mathrm{d}t~\mathrm{d}t^{\prime}~{\rm sgn}(t^{\prime}-t)\overline{f(t)}f(t^{\prime})\tag{T}$$
is a central element proportional to the identity operator. Note that
$$\begin{align}
\frac{\partial }{\partial t_2}D(t_2,t_1)~&=~\frac{{\bf 1}}{2\hbar}\left(\overline{F(t_2,t_1)}f(t_f)-\overline{f(t_2)}F(t_2,t_1)\right)
\\ ~&=~\frac{1}{2}\left[ A(t_2,t_1), \frac{i}{\hbar}H(t_2)\right]~=~\frac{1}{2}\left[\frac{\partial }{\partial t_2}A(t_2,t_1), A(t_2,t_1)\right].\tag{U}
\end{align}$$
One may use identity $(\text{U})$ to check directly that the operator $(\text{S})$ satisfies the TDSE $(\text{C})$.
References:
- Sidney Coleman, QFT lecture notes, arXiv:1110.5013; p. 77.