4

One of the biggest strengths of ChatGPT is that it generates fitting text with respect to the input query. It usually stays on topic, anwers the question completely and especially does not start talking gibberish or repeating itself.

This behaviour is different when comparing this to older LLMs. For example: GPT2 would usually only stop generating text when it hit the token limit or a predefined stop sequence. Also, it had a much bigger problem with giving repeating answers. Newer models (especially instruction tuned ones) do not suffer from this problems (e.g. llama 2).

So I have 2 questions: What mechanisms/techniques are used in current language models such that...

  1. ...the models know when to stop generating text.
  2. ...the models do not repeat themselfes and stay on topic.

I suspect it might have alot to do with instruction tuning but I am happy to hear from you.

Ricu
  • 41
  • 3

2 Answers2

1

TLDR: if you're using a model without instruction-tuning, you'll likely need to do some additional work during inference to make sure that your outputs are coherent regardless of how old your model is. But even on GPT-2 you can get decent generations with the right technique.

You're correct in that this has a lot to do with instruction-tuning: GPT-2 wasn't trained to give helpful chatbot-style responses, it was trained to generate random web-text. Yes, there are EOS tokens that indicate the end of a sequence, but during pre-training these separate documents, which tend to be pretty long and likely aren't very coherent as chat-bot responses.

This also means that newer models that aren't instruction-tuned (like Llama-2) will suffer from similar issues, compared to their instruction-tuned variant (Llama-2 chat).

Better responses through prompting

However, it's important to note that, at least for larger models, training isn't necessary to get this kind of behavior. Lin et al. explore various methods for eliciting helpful chatbot-like responses through zero-shot prompting (instructions + a markdown-style prompt), ICL (providing examples of queries & responses), retrieval augmented ICL (ICL but retrieving examples that are most similar to the query), and their proposed method which adds a system prompt and uses specially-crafted ICL samples.

From Lin et al. (https://ai-plans.com/fs/arxiv/arxiv.org_abs_2312.01552v1.pdf)

My guess is that this wouldn't work as well on less powerful models like GPT-2, as they are much worse at learning from context.

Repetition penalty

@Eponymous mentioned that there are mechanisms to downweigh repetition during sampling, but it's important to add that people have been working on developing inference-time techniques to improve generation quality since GPT-2 (and before neural models, for that matter). e.g., this famous 2019 paper, proposing nucleus sampling (and targeting GPT-2 Large & XL): Holtzman et al. (https://arxiv.org/pdf/1904.09751)

With the right techniques, you can definitely get these older models to generate coherent text (see Figure 11 in Appendix B in the above paper for some examples of different sampling techniques + the resulting generations).

Alexander Wan
  • 1,409
  • 1
  • 12
0
  1. LLMs have by themself a token used for end of string, so they themself are the one that decide when a generation should stop
  2. This kind of questions are usually pretty opinionated, but the main reasons are bigger models, and more training data

However, tuning the length might also be a bias from the RLHF training some model have received

Alberto
  • 2,863
  • 5
  • 12