Instruction Tuning and RLHF

Instruction Tuning

Instruction tuning fine-tunes LLMs on datasets of instruction-response pairs, enabling models to generalize across tasks by following natural language instructions. Unlike traditional fine-tuning, which targets specific tasks, instruction tuning teaches models to interpret and execute diverse commands, such as “summarize this text” or “solve this math problem.” This approach, popularized by models like InstructGPT, leverages curated datasets to improve zero-shot and few-shot performance.

Reinforcement Learning with Human Feedback (RLHF)

RLHF aligns LLMs with human values by fine-tuning them using reinforcement learning, where human feedback serves as a reward signal. RLHF addresses limitations in supervised fine-tuning, such as overfitting to biased datasets, by optimizing models for helpfulness, truthfulness, and safety. The RLHF pipeline, used in InstructGPT and ChatGPT, combines supervised fine-tuning, reward modeling, and policy optimization.

RLHF Pipeline

The RLHF pipeline consists of three stages:

1. Supervised Fine-Tuning (SFT): The pretrained LLM is fine-tuned on a high-quality dataset of instruction-response pairs to establish a baseline model capable of following instructions.
2. Reward Modeling: A separate reward model is trained to predict human preferences by ranking model outputs (e.g., which response is more helpful). Humans provide feedback by comparing pairs of outputs, creating a dataset of preference rankings.
3. Policy Optimization: The LLM is optimized using reinforcement learning (typically PPO) to maximize the reward model’s score while staying close to the SFT model to prevent overfitting or degradation.

This pipeline, detailed in Training Language Models to Follow Instructions with Human Feedback, ensures alignment with complex human objectives.

InstructGPT

InstructGPT, introduced by OpenAI in the above paper (2022), is a GPT-3 variant fine-tuned with RLHF to prioritize helpfulness and truthfulness. It uses a dataset of human-written prompts and responses for SFT, followed by RLHF to refine outputs based on human rankings. InstructGPT outperforms GPT-3 in user satisfaction, reducing harmful or biased responses.

Key Innovations:

• Demonstrated that smaller models (e.g., 1.3B parameters) with RLHF can outperform larger, unaligned models (e.g., 175B).
• Balanced alignment with performance using a KL-divergence penalty to prevent deviation from the pretrained model.

Results: Human evaluators preferred InstructGPT over GPT-3 in 70-80% of cases, with significant improvements in instruction-following and factual accuracy.

Reward Modeling

Reward modeling trains a model to assign scores to LLM outputs based on human preferences. Given a prompt and multiple responses, humans rank the responses (e.g., better/worse), creating a dataset of pairwise comparisons. The reward model, often a smaller LLM, is trained to predict these rankings, providing a scalar reward for RL optimization. This approach, used in InstructGPT, captures nuanced preferences like clarity or appropriateness.

Challenges:

• Sparse Feedback: Human rankings are costly, requiring efficient data collection.
• Reward Hacking: Models may exploit reward model flaws, generating high-scoring but undesirable outputs.

Solutions: Regularization (e.g., KL-divergence) and diverse feedback datasets mitigate these issues, as discussed in Learning to Summarize with Human Feedback.

Proximal Policy Optimization (PPO)

PPO, a reinforcement learning algorithm, is used in RLHF to optimize the LLM’s policy (output distribution) to maximize the reward model’s score. PPO balances exploration and stability by constraining policy updates, ensuring the model doesn’t diverge too far from its initial behavior. In InstructGPT, PPO incorporates a KL-divergence penalty to maintain similarity to the SFT model, preventing overfitting to the reward model.

Advantages:

• Stable training compared to older RL algorithms like TRPO.
• Effective for high-dimensional action spaces like text generation.

Results: PPO in RLHF improved InstructGPT’s alignment, reducing toxic outputs by 10-15% compared to SFT alone.

Supervised Fine-Tuning (SFT)

SFT initializes the RLHF pipeline by fine-tuning the pretrained LLM on a curated dataset of prompt-response pairs, typically human-written or high-quality synthetic data. SFT aligns the model with basic instruction-following capabilities, serving as a starting point for RLHF. For InstructGPT, SFT used a dataset of ~13k prompt-response pairs, focusing on diverse tasks like summarization and question answering.

Results: SFT alone improved GPT-3’s instruction-following but was less effective than RLHF for nuanced alignment, highlighting the need for reward-driven optimization.

Alignment Objectives

Alignment objectives ensure LLMs produce outputs that are helpful, safe, and aligned with human values. RLHF formalizes alignment by optimizing for:

• Helpfulness: Responses should address user intent (e.g., InstructGPT’s focus on clear answers).
• Truthfulness: Outputs should be factually accurate, reducing hallucinations.
• Safety: Models avoid harmful, biased, or toxic content, critical for real-world deployment.

Alignment is challenging due to subjective human preferences and dataset biases. RLHF mitigates this through iterative feedback and regularization, as seen in Llama-2’s RLHF pipeline (Llama 2: Open Foundation and Fine-Tuned Chat Models).

Impact on Foundation Models

Instruction tuning and RLHF have transformed foundation models by:

• Enhancing Usability: InstructGPT and Llama-2 deliver user-friendly, instruction-following models for real-world applications.
• Improving Safety: RLHF reduces harmful outputs, as seen in ChatGPT, enabling safer deployment.
• Enabling Generalization: Instruction tuning allows models to handle diverse tasks with minimal retraining, broadening their utility.
• Setting Standards: RLHF’s success in aligning models has influenced designs like PaLM and Mixtral, emphasizing human-centric optimization.

These advancements, discussed in Multimodal Foundation Models: From Specialists to General-Purpose Assistants, underscore their role in making LLMs practical and ethical.