How does one show that the bracket of elements in the Lie algebra of $SO(n,m)$ is given by
$$[J_{ab},J_{cd}] ~=~ i(\eta_{ad} J_{bc} + \eta_{bc} J_{ad} - \eta_{ac} J_{bd} - \eta_{bd}J_{ac}),$$
where $$J_{ab}=-J_{ba},\qquad a,b\in\{1,\ldots,n+m\},$$
and where $\eta$ has is the definite symmetric form with signature $(n,m)$?
By definition, the metric tensor $\eta_{ij}$ transforms trivially under the defining rep of $SO(n,m)$. $$ \eta_{ij}=[D(g^{-T})]_{i}^{\ k}[D(g^{-T})]_{j}^{\ l}\eta_{kl} =[D(g^{-1})]^{k}_{\ i}[D(g^{-1})]^{l}_{\ j}\eta_{kl} $$ and this holds for all $g\in SO(n,m)$. Consider a one-parameter subgroup of the defining rep with matrices $D(g)=e^{tJ}$ where $J^{i}_{\ j}$ is an element of the Lie algebra and $t$ is a real parameter. Substitute into the above equation, $$ \eta_{ij}=[e^{tJ}]^{k}_{\ i}[e^{tJ}]^{l}_{\ j}\eta_{kl} $$ and differentiate wrt $t$ at the identity $t=0$. $$ 0=J^{k}_{\ i}\delta^{l}_{\ j}\eta_{kl}+\delta^{k}_{\ i}J^{l}_{\ j}\eta_{kl} =J^{k}_{\ i}\eta_{kj}+J^{k}_{\ j}\eta_{ik} $$ This is the condition that the elements of the Lie algebra must obey. The Lie algebra elements can be generated by an antisymmetrized pair of vectors $x^{i}$, $y^{j}$. $$ J^{i}_{\ j}=x^{i}y_{j}-y^{i}x_{j} $$ where lowering is performed by the metric tensor $x_{i}=\eta_{ij}x^{j}$. The Lie algebra condition is automatically satisfied by generating the Lie algebra elements in this way. The Lie algebra elements $J_{ab}$ in the question are just made by choosing the vectors $x$ and $y$ as the basis vectors $x^{i}=\delta{^i}_{a}$, $y_{i}=\eta_{ij}\delta^{j}_{b}=\eta_{ib}$. $$ [J_{ab}]^{i}_{\ j}=\delta^{i}_{a}\eta_{jb}-\delta^{i}_{b}\eta_{ja} $$ Now compute the commutator (hopefully two different uses of square brackets is not too confusing), $$ [J_{ab},J_{cd}]^{i}_{\ j}=[J_{ab}]^{i}_{\ k}[J_{cd}]^{k}_{\ j}-[J_{cd}]^{i}_{\ k}[J_{ab}]^{k}_{\ j} $$ and a few lines of straightforward calculation gives, $$ [J_{ab},J_{cd}]^{i}_{\ j}=\eta_{bc}[J_{ad}]^{i}_{\ j}-\eta_{ac}[J_{bd}]^{i}_{\ j}-\eta_{bd}[J_{ac}]^{i}_{\ j}+\eta_{ad}[J_{bc}]^{i}_{\ j} $$ as the commutator for the defining rep. The Lie algebra is the same for all the group reps. The question asks for the commutator for a unitary rep of the group. To do this, the one-parameter unitary subgroup is $D(g)=e^{itJ}$ and so the Lie algebra elements of the defining rep are redefined as belonging to a unitary rep by the replacement $J\rightarrow iJ$. The commutator now becomes, $$ [iJ_{ab},iJ_{cd}]=\eta_{bc}iJ_{ad}-\eta_{ac}iJ_{bd}-\eta_{bd}iJ_{ac}+\eta_{ad}iJ_{bc}\\ -[J_{ab},J_{cd}]=i\eta_{bc}J_{ad}-i\eta_{ac}J_{bd}-i\eta_{bd}J_{ac}+i\eta_{ad}J_{bc} $$ which is the commutator in the question apart from an overall sign change. This is easily fixed by changing the definition of the Lie algebra elements of the defining rep to, $$ [J_{ab}]^{i}_{\ j}=\delta^{i}_{b}\eta_{ja}-\delta^{i}_{a}\eta_{jb} \ . $$
Here we will sketch a possible derivation.
Let $\eta\in {\rm Mat}_{n\times n}(\mathbb{R})$ be a real symmetric matrix of signature $(p,q)$, where $n=p+q$.
Define the Lie group $$O(p,q)~:=~ \{ \Lambda\in {\rm Mat}_{n\times n}(\mathbb{R}) \mid \Lambda^T\eta \Lambda = \eta \}, $$ where $\Lambda^T$ denotes the transposed $\Lambda$ matrix. Prove for fun that $O(p,q)=O(q,p)$.
Prove that if $\Lambda_1, \Lambda_2 \in O(p,q)$, then the matrix product $\Lambda_1 \Lambda_2\in O(p,q)$.
Prove that if $\Lambda \in O(p,q)$, then the inverse matrix $\Lambda^{-1} \in O(p,q)$.
Define the Lie algebra $$ o(p,q)~:=~ \{ M\in {\rm Mat}_{n\times n}(\mathbb{R}) \mid M^T\eta + \eta M = 0\}. $$
Prove that if $M_1, M_2 \in o(p,q)$, then the matrix commutator $$[M_1, M_2]~:=~M_1M_2-M_2M_1 ~\in ~o(p,q).$$
If $O(p,q)\ni \Lambda= {\bf 1}_{n\times n}+ M$, where $M\in {\rm Mat}_{n\times n}(\mathbb{R})$ is infinitesimal, prove that $M\in o(p,q)$.
Define generators $J^{ij}= -J^{ji}\in{\rm Mat}_{n\times n}(\mathbb{R})$ as $$ (J^{ij})^k{}_{\ell}~:=~\eta^{ik}\delta^j_{\ell} - (i \leftrightarrow j). $$
Prove that $J^{ij}\in o(p,q)$.
Prove that $$ [J^{ij},J^{k\ell}]~=~ \left(\eta^{jk}J^{i\ell} - (i \leftrightarrow j)\right) - (k \leftrightarrow \ell). $$
The above convention makes the Lie algebra $o(n)$ the set of skewsymmetric real $n\times n$ matrices, which are anti-Hermitian. If you would like the Lie algebra $o(n)$ to instead be the set of Hermitian $n\times n$ matrices, modify the above definitions with appropriate factors of the imaginary unit $i$.
