My rough understanding of RLHF as used for ChatGPT in a nutshell is this:
A reward model is trained using comparisons of different responses to the same prompt. Human trainers rank these responses based on quality.
The reward model is a neural network that learns to predict these human rankings. It essentially learns the "policy" that human trainers use to rank responses.
An initial policy, which is a language model, is fine-tuned using Proximal Policy Optimization (PPO) with the reward model providing the reward signal. This process is iterative, with the policy and reward model being updated alternately.
The policy is then used to generate responses to prompts. The reward model assesses these responses and provides a reward signal, which is used to further fine-tune the policy, i.e. the language model.
My main question is the first one, the others are just for giving context:
1. What's the architecture and size of the neural-network-based reward model?
Is it pretrained, too? Is it possibly another pretrained (foundational) language model?
By how many samples labelled by human trainers is the reward model trained?
By how many prompts and rewarded completions is the language model trained later? (Which prompts, by the way?)
These numbers I'd like to compare with the numbers of pretrained ChatGPT:
Transformer-based ChatGPT has 175 billion weights.
It was pretrained on 500 GB of text data, distributed over an unknown number of "documents" (from single tweets to the Holy Bible) with roughly 500B tokens over all. During training ChatGPT was exposed to a multiple of 500B samples (assuming that all 500B tokens were used for training).
I assume that during RLHF foundational ChatGPT was exposed to a much smaller number of prompts to complete (and to be rewarded).
