3

I'm considering getting into NLP research (that is AI research into Transformers specifically) but my friend made this point which got me concerned, "he said NLP research is dead b/c of ChatGPT". We all know ChatGPT is enormous and highly versatile. Is it realistic for NLP researchers to make meaningful contributions without having the resources of: google, amazon, ivy leagues, facebook, OpenAI/Microsoft etc...

For example is it meaningful to compete with models in certain "resource classes", e.g. trying to get the best BERT model given a fixed amount of compute or parameters?

P.S. For example recently saw the Vega2 paper 2nd place on SuperGlue leader board used 320 A100s for > 30 days. Looking at rental prices online that should cost about $300,000!

profPlum
  • 496
  • 2
  • 10

2 Answers2

2

While LLMs like ChatGPT have indeed reshaped NLP, they have not killed NLP research. Instead, the field has shifted in terms of focus and the right niche opportunities available for researchers, even those outside the biggest organizations.

LLMs like ChatGPT are powerful but often brittle when faced with out-of-distribution (OOD) inputs or nuanced domain-specific higher-level rule-based traditional symbolic AI and NLP tasks. For instance, many languages and dialects are underrepresented or near-extinct and NLP research for these is critical and often does not require massive resources. Real world NLP applications often require domain-specific fine-tuning. For example, legal, medical, and financial NLP involve challenges that general-purpose models like ChatGPT cannot solve out of the box.

Multi-Modal NLP to combine text with other modalities (e.g., vision, audio) is another evolving domain where new contributions are welcome. Similarly combining NLP with fields like social sciences, mathematics, cognitive science, or healthcare opens new avenues that are less reliant on current ChatGPT like LLMs.

Finally addressing imminent and difficult NLP issues related to explainability and interpretability, fairness, bias, societal risks and ethical concerns is also a burgeoning area of study.

cinch
  • 11,000
  • 3
  • 8
  • 17
2

It is difficult. At my previous company I (NLP Engineer) was replaced by an LLM, and looking for NLP jobs, everything was geared towards experience with LLMs. "Traditional" NLP methods are not really en vogue enough at present, it seems.

However, my own view is that this is a temporary bubble. There is a lot of hype, and over-promises about the capabilities of LLMs. At some point, I expect this to calm down, and there to be a falling back on tried and tested methods which aren't as "sexy", but do actually work.

I do realise this is perhaps not a very popular view, but LLMs are basically just smoke and mirrors, albeit very sophisticated.

So, for the near future it might be difficult to find a job in the NLP area with more traditional skills, but my prediction would be that this will change in the longer term; and even now there are companies who don't want to use LLMs for various reasons (compliance, data protection, etc). Academia might be a better bet for now than industry, as it seems less susceptible to the hype.

Oliver Mason
  • 5,477
  • 14
  • 32