How is LLM generated content moderated?

Question

I'm looking for references (articles) about how LLM generated content is moderated.

From a technical point of view, what makes the difference between a so-called "uncensored" LLM, such as Pygmalion 7B, and a "censored" one?

Does a LLM always generate text that gets moderated later, or is it pre-trained / fine-tuned to generate moderated content by default?

Answer 1

TL;DR; There is no model inherently censored, as big models have big training data which can't be all modified to not include explicit content. Also, we don't want to not-include the explicit content lest we want to take away the abstractions that the NN might learn from that explicit content also. There are multiple layers to causal reasoning involved to design commercially available products like OpenAI's models.

That's an interesting one. Let's first differentiate between different terms being used. A foundational model is a model that is pre-trained on a set of training data and can be fine-tuned to perform downstream tasks, whereas consequentially a fine-tuned model is one that can perform specific tasks like summarization, topic extraction, NER, etc.

Now, let's talk about the GPT-3 family for a bit which houses both the Davinci model and ChatGPT (GPT-3.5) model. If you read about it a little bit, OpenAI lets you fine-tune Davinci but not ChatGPT. Davinci was fine-tuned to perform better generations/completions. On the other hand, ChatGPT was fine-tuned to perform humanesque chat conversations and consequentially started outputting more uncensored content (because of-course human texts tend to get witty).

Then comes the concept of causal reasoning. You can understand it most easily as a conditional statement which is an added layer over the output of the language model. Essentially it forces the model to take certain actions given certain scenarios. One example might be a prompt saying to use a different selection strategy (it becomes greedy at temperature=0) if the next generated token leads to a vulgar word/slang. Another might be pre-added instructions to model input requesting it to return 'I can't respond to this query as it pertains to copyright information'. It's basically an elaborate if-else statement that works with language models.

Coming back to our example of OpenAI's models, while not completely uncensored, DaVinci is susceptible to producing more uncensored content than ChatGPT as evidenced in their blog asking users to use a layer of causal reasoning to best meet their market requirements and that OpenAI will not be held liable to any explicit content generated.

UPDATE (21 July 2023) - According to this UC Berkley study, this causal reasoning might be actually reducing the effectiveness of ChatGPT due to increasing limitations on the content that it can produce.

Answer 2

The better formulation of this question is “How do LLMs get aligned with human values?”. Which humans and what values is an interesting societal question. Let’s say, for the purpose of this conversation, it’s with a set of humans who don’t like to hear curse words.

There are 4 main forms of achieving alignment. The layer at which each is implemented is shown in [brackets].

1. [model] Pre-train your model on the data set you want to learn from. If you don’t want curse words, strip all the training data of curse words laden content, and the model is better aligned with your “no curse words” values.
2. [model] Fine-tuning & reinforcement learning w/ human feedback (RLHF). If you have a base model that is good in performance but needs alignment, you can take a smaller set of data than what was used to pre-train, and do a post-train step. This fine-tuning data is often very carefully curated to achieve specific outcomes or match a specific data distribution. Fine-tuning on that data often “re-wires” the neurons (via updating the weights of the neurons) to reduce the curse words. Reinforcement learning can further achieve alignment using a more complicated process of creating a reward model that the base model is further tuned on.
3. [application layer] At the prompt layer, few-shot learning and system prompts can be used to prevent the model from using certain words, by way of giving examples. This is also called “in-context learning” and can be used for many outcomes, alignment is one of them.
4. Post-generation of the model, the final result can be passed through another model or ensemble of models. These are typical classifier models that are looking for specific things. You could put any kind of “model” in this step from something naive as keyword filtering, to more sophisticated models like “hate speech”, “violence” or “sexually explicit material’ detection. The best practice is to pass prompt inputs (from users) through moderation systems, but the outputs from the LLM can also be checked.