Start from the definitions
$D(\alpha)\equiv\exp(\alpha a^\dagger - \alpha^* a)$
and
$S(\xi)\equiv\exp[\frac12(\xi a^{\dagger 2}-\xi^* a^2)]$.
Remember the identity
$$e^A B e^{-A}=e^{\operatorname{ad}_A}B\equiv \sum_{k=0}^\infty\frac{1}{k!}[\underbrace{A\cdots A}_k,B],$$
with $[\underbrace{A\cdots A}_k,B]\equiv \operatorname{ad}_A^k B$ denoting the iterated commutator of $A$ with $B$ (e.g. $[AA,B]\equiv[A,[A,B]]$).
We then have
$$S(\xi)D(\alpha)S^\dagger(\xi)=\sum_{k=0}^\infty\frac{1}{k!} \operatorname{ad}_A^k D(\alpha)$$
with $A\equiv \frac12(\xi a^{\dagger 2}-\xi^* a^2)$. Observe that
\begin{align}
[A,\alpha a^\dagger -\alpha^* a] &=
(\xi\alpha^*) a^\dagger - (\xi\alpha^*)^* a =
(\xi\alpha^*) a^\dagger - (\xi^*\alpha) a, \\
[A,[A,\alpha a^\dagger -\alpha^* a]] &=
|\xi|^2(\alpha a^\dagger - \alpha^* a).
\end{align}
From these, we can deduce the more general rules:
\begin{align}
\operatorname{ad}_A^{2k}(\alpha a^\dagger-\alpha^* a) &=
|\xi|^{2k} (\alpha a^\dagger-\alpha^* a), \\
\operatorname{ad}_A^{2k+1}(\alpha a^\dagger-\alpha^* a) &=
|\xi|^{2k} [(\xi\alpha^*) a^\dagger-(\xi\alpha^*)^* a],
\end{align}
and therefore
$$
S(\xi)D(\alpha)S^\dagger(\xi) =
\sum_{k=0}^\infty |\xi|^{2k} \left[
\frac{1}{(2k)!} (\alpha a^\dagger -\alpha^* a) +
\frac{1}{(2k+1)!} (\xi\alpha^* a^\dagger -\xi^*\alpha a)
\right] \\
= (\alpha a^\dagger -\alpha^* a) \cosh(|\xi|) +
(e^{i\theta}\alpha^* a^\dagger -e^{-i\theta}\alpha a) \sinh(|\xi|)
\\= (\alpha \cosh(|\xi|) + e^{i\theta}\alpha^* \sinh(|\xi|))a^\dagger - c.c. \\
= D(\alpha \cosh(|\xi|) + e^{i\theta}\alpha^* \sinh(|\xi|)).
$$
We conclude that
$$S(\xi)D(\alpha) = D(\alpha \cosh(|\xi|) + e^{i\theta}\alpha^* \sinh(|\xi|)) S(\xi).$$