Module 5 · Lesson 12

Once a model can generate many plausible responses, another question appears: which responses should it prefer?

That is the problem addressed by preference-based post-training methods such as Reinforcement Learning from Human Feedback, or RLHF.

Where the feedback comes from

Human evaluators can compare multiple responses to the same prompt and indicate which one is better according to criteria such as helpfulness, correctness, relevance, or safety.

Prompt
  ↓
Response A     Response B
             /
       preference
           ↓
post-training signal

In a classic RLHF pipeline, those preferences can train a reward model. The reward model then scores candidate outputs, and reinforcement learning adjusts the language model toward outputs that receive higher reward.

Not every modern preference method uses that exact pipeline

Preference optimization has evolved. Some methods, such as direct preference optimization, can train directly from preferred and rejected examples without first building a separate reward model and then running a traditional RL loop.

So RLHF is best understood as an influential family of feedback-driven post-training ideas, not the only alignment recipe.

What is RLAIF?

Reinforcement Learning from AI Feedback, or RLAIF, uses another AI system to provide some of the evaluations or preference signals that humans would otherwise produce manually. This can make feedback generation easier to scale, often with human-written principles or examples guiding the evaluator.

Feedback improves behavior; it does not guarantee perfection

It would be misleading to say RLHF “prevents” a model from producing harmful, false, or inappropriate content. Preference data can be incomplete. Evaluators can disagree. Reward signals can be exploited. New prompts can expose behaviors that were not represented in training.

Safety therefore relies on layers: training data, post-training, system instructions, tool permissions, filters where appropriate, monitoring, evaluation, and product-level controls.

Alignment is a continuing engineering problem

The deeper lesson is that model behavior is not determined by pre-training alone. We can shape behavior after pre-training using feedback, but the quality of that feedback and the surrounding system still matter.

Alignment is not a switch that turns a model from “wrong” to “right.” It is an ongoing process of defining desired behavior, measuring it, finding failures, and improving the system.

Demystifying AI series

← Pre-Training vs. Post-Training   ·   RAG Explained →