One way to answer this question is to look at all of the elemental abundances

By Swift - Own work, CC0, Link
and information about the elements’ origins,

By Cmglee - Own work, CC BY-SA 3.0, Link
Tantalum is way out in the tail of the other heaviest elements. It’s rare for the same reason that all of the heavies are rare: they have to be produced from iron by the s-process and the r-process.
Deuterium, lithium, and boron are mostly not produced in stellar fusion at all. Deuterium is produced by from protium by neutron capture, but has a binding energy of only 2.2 MeV. An environment with enough neutrons to produce deuterium probably also has lots of thermal photons with enough energy to dissociate the deuterium. And there is not a low-temperature fusion pathway to produce lithium or boron. The earliest “zero-metallicity” stars produced carbon in the triple-alpha process, which enabled CNO-catalyzed fusion at lower temperatures in their higher-metallicity descendants. Some lithium was produced in the big bang, but no boron. The figure suggests that essentially all of the universe’s boron was produced as cosmic-ray spallation fragments.
In this picture, nitrogen outnumbers the other odd-odd isotopes just because it’s easier to make nitrogen in main-sequence low-mass stars than it is the other elements.
Another way to ask why nitrogen is the most abundant of the low-mass odd-odd nuclei is to ask what happens to each species in the kind of high-radiation environment where s-process element production might be taking place:
- Deuterium will form from hydrogen, but be subject to photodissociation.
- There is no s-process path from helium to lithium, because helium-5 isn’t stable.
- Lithium-6 will absorb neutrons and fragment, $\require{mhchem}\ce{^6Li(n,\alpha){}^3He}$. Lithium-6 has a much higher neutron cross section than lithium-7 (though Li-7 is also destroyed by neutron capture), so a high-radiation environment will deplete the lithium-6.
- Boron-10 likewise has a much higher neutron absorption/fragmentation cross section than boron-11, with significant contributions from $\ce{^{10}B(n,\gamma){}^{11}B}$ and $\ce{^{10}B(n,\alpha)^{7}Li}$.
- Nitrogen under neutron irradiation, however, undergoes $\ce{^{14}N(n,p)^{14}C}$. After a few thousand years, the new carbon nucleus turns back into nitrogen by beta decay. (This is why there’s carbon-14 in Earth’s atmosphere; the neutrons come from spallation by cosmic rays.)
I don’t know if there’s a good, intuitive explanation for why nitrogen-14 is a hundred times more likely to undergo (n,p) than (n,$\gamma$). I suspect that, if the cross sections for those two processes were reversed, then the s-process would also deplete N-14 relative to N-15. But whether the s-process is actually relevant here, or whether it’s just a useful shorthand for “a place where element production happens and there are lots of neutrons” which happens to support a just-so story, isn’t something I’m terribly clear on.