Generates candidate summaries
Remember RLHF
Full name: Reinforcement Learning from Human Feedback
Memory aid: Reinforcement Learns from Humans' Feedback.
Human preference signals can contain bias and do not guarantee truthfulness.
Study area: AI & machine learning · Batch 4
Relevant study for: Azure AI Fundamentals AI-901 · Azure AI Apps and Agents Developer AI-103 · Azure AI Fundamentals (retired) AI-900 · Azure AI Engineer (retired) AI-102
Practice with exam flashcards →AI Models & Assistants / Knowledge library
RLHF (Reinforcement Learning from Human Feedback)
Learning desired behavior from human preference feedback
Why learn RLHF (Reinforcement Learning from Human Feedback)?
Helps train for qualities people can compare more easily than they can specify as exact rules. Its results depend on whose preferences are collected and whether the learned reward reflects the intended behavior.
The problem it solves
Converts human judgments into a training signal for behavior that is difficult to express with a hand-written reward function.
Where it came from
RLHF uses human judgments to shape a learning system's reward signal. Work on learning from comparisons of behavior helped establish the approach before it was applied to instruction-following language models. A common pipeline learns a reward model from preferences and optimizes a policy against it; PPO is one possible optimizer, not the definition of RLHF.
Build your understanding
Work through these exercises in order. Try the task before opening a reference.
- Write a rubric for comparing two assistant responses on helpfulness and accuracy.
- Label ten response pairs and record disagreements between reviewers.
- Sketch the separate roles of the policy, reward model, and reference policy.
- Construct a response that could exploit a reward model's preference for length.
- Design a held-out human evaluation that is independent of the training reward.
Documentation and sources
- Deep reinforcement learning from human preferences (arxiv.org)
- InstructGPT: training with human feedback (arxiv.org)
Connect the ideas
Explore next
Reinforcement Learning (RL)SFT (Supervised Fine-Tuning)DPO (Direct Preference Optimization)Reinforcement Learning from AI Feedback (RLAIF)AI Alignment
Real World Example
Automated Legal Document Summarization Alignment
A mid-sized legal tech firm providing automated contract analysis software to corporate legal departments.
The business problem
The existing summarization model frequently hallucinates clauses or adopts an overly casual tone that fails to meet the formal, precise standards required by senior legal counsel, leading to low user trust.
How this topic solves it
By implementing Reinforcement Learning from Human Feedback (RLHF), the firm can align the model's output with expert legal preferences. Instead of relying on rigid, manually defined reward functions, the system learns a reward model based on pairwise comparisons provided by human legal experts, effectively capturing the nuance of professional legal drafting.
- Generate multiple candidate summaries for a single legal document using the base language model.
- Present pairs of these summaries to legal experts for comparative preference ranking.
- Train a reward model to predict which summary a human would prefer based on the collected ranking data.
- Optimize the summarization policy using reinforcement learning, where the reward model provides the signal for policy updates.
- Iteratively refine the policy by collecting new feedback on the updated model's outputs.
The flow shows how human preferences shape the model's behavior.
Ranks pairs of summaries
Learns to predict preferences
Updates model weights
- Base Model → provides pairs → Human Evaluator
- Human Evaluator → preference data → Reward Model
- Reward Model → reward signal → Policy Optimizer
- Policy Optimizer → updated policy → Base Model
Expected business outcome
The model produces summaries that consistently mirror the professional tone and structural precision preferred by the firm's legal experts, significantly increasing adoption rates.
Limitations and checks
A primary limitation is the potential for reward hacking, where the model learns to exploit the reward model by generating text that looks professional to humans but contains subtle, incorrect legal interpretations. This is checked by maintaining a held-out set of documents to audit for factual accuracy versus stylistic alignment.