Given the unitary operator $U=1+i\varepsilon F$ (where $\varepsilon$ is an infinitesimal scalar), in order to prove that $F$ is Hermitian:
$$\begin{align} UU^{\dagger} &= 1 \\ (1+i\varepsilon F) (1-i\varepsilon F^\dagger) &= 1 \\ 1+i\varepsilon F-i\varepsilon F^\dagger +\varepsilon^2 FF^\dagger &= 1 \end{align}$$
It seems that $F$ must be equal to $F^\dagger$ to satisfy that expression, but how can the remaining term be equal to zero? $(\varepsilon^2 FF^\dagger\overset{\large\text{?}} = 0)$
The whole point here is that $U$ is an infinitesimal unitary transformation and so we neglect terms of order $\mathcal{O}(\varepsilon^2)$ since they are by definition negligible to $\mathcal O(\varepsilon)$. Thus,
$$UU^\dagger=(1+i\varepsilon F)(1-i\varepsilon F) = 1+ i\varepsilon(F-F^\dagger) + \mathcal O(\varepsilon^2) = 1$$
from which we see $F= F^\dagger$, i.e. $F$ must be Hermitian. Note however it is not always the case that we keep terms of linear order only.
For example, in Sakurai's derivation of the commutation relations for infinitesimal rotations, terms of order $\mathcal O(\epsilon^2)$ are kept, so it depends somewhat on context.
On the other hand, if you want keep terms to $\epsilon^2$, you need to include those in you expansion as well, i.e. $U\approx 1+i \epsilon F - \frac{1}{2}\epsilon^2 F^2$ as those will have contributions through cross terms when you expand $$ UU^\dagger\approx 1 +i \epsilon (F-F^\dagger) -\frac{\epsilon^2}{2} \left(F^2 - FF^\dagger - F^\dagger F + (F^\dagger)^2\right)+\ldots $$
Since $\epsilon$ is infinitesimal, the $\epsilon^2FF^+$ term can be neglected. Only keep terms to first order in $\epsilon$.
You can think of this whole derivation in a different way: you have a family $U(t)$ of unitary operators (continuously) parametrised by $t$, such that $U(t_1+t_2) = U(t_1)U(t_2)$ (think translation). It follows that $U(0)=1$. Furthermore, you call $U'(0)=i F$ (derivative!), such that $U(\epsilon)$'s first-order Taylor expansion looks like $U(\epsilon) = 1 + i\epsilon F$. You want to show that $F = -i U'(0)$ is hermitian, so you differentiate $U(t)U^\dagger(t) = 1$ at $t=0$: $$ 0 = U'(0)U^\dagger(0) + U(0)U^{\dagger\prime}(0) = U'(0)1 + 1U^{\dagger\prime}(0) = iF + (-iF^\dagger) = i (F - F^{\dagger}) $$
This way, you show that the Lie algebra $\mathfrak{u}(n)$ of the Lie group $U(n)$ of unitary matrices consists of anti-hermitian matrices (= $i$ times hermitian matrices).
