<?xml version="1.0" encoding="utf-8" standalone="yes"?><rssversion="2.0"xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Eric X. Liu's Personal Page</title><link>/</link><description>Recent content on Eric X. Liu's Personal Page</description><generator>Hugo</generator><language>en</language><lastBuildDate>Mon, 26 Oct 2020 04:47:36 +0000</lastBuildDate><atom:linkhref="/index.xml"rel="self"type="application/rss+xml"/><item><title>Some useful files</title><link>/posts/useful/</link><pubDate>Mon, 26 Oct 2020 04:14:43 +0000</pubDate><guid>/posts/useful/</guid><description><ul>
<?xml version="1.0" encoding="utf-8" standalone="yes"?><rssversion="2.0"xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Eric X. Liu's Personal Page</title><link>/</link><description>Recent content on Eric X. Liu's Personal Page</description><generator>Hugo</generator><language>en</language><lastBuildDate>Sat, 02 Aug 2025 15:46:24 -0700</lastBuildDate><atom:linkhref="/index.xml"rel="self"type="application/rss+xml"/><item><title>Some useful files</title><link>/posts/useful/</link><pubDate>Mon, 26 Oct 2020 04:14:43 +0000</pubDate><guid>/posts/useful/</guid><description><ul>
</ul></description></item><item><title>About</title><link>/about/</link><pubDate>Fri, 01 Jun 2018 07:13:52 +0000</pubDate><guid>/about/</guid><description/></item></channel></rss>
</ul></description></item><item><title>About</title><link>/about/</link><pubDate>Fri, 01 Jun 2018 07:13:52 +0000</pubDate><guid>/about/</guid><description/></item><item><title/><link>/posts/a-deep-dive-into-ppo-for-language-models/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/posts/a-deep-dive-into-ppo-for-language-models/</guid><description><p>Large Language Models (LLMs) have demonstrated astonishing capabilities, but out-of-the-box, they are simply powerful text predictors. They don&rsquo;t inherently understand what makes a response helpful, harmless, or aligned with human values. The technique that has proven most effective at bridging this gap is Reinforcement Learning from Human Feedback (RLHF), and at its heart lies a powerful algorithm: Proximal Policy Optimization (PPO).</p>
<p>You may have seen diagrams like the one below, which outlines the RLHF training process. It can look intimidating, with a web of interconnected models, losses, and data flows.</p></description></item></channel></rss>
<!doctype html><htmllang=en><head><title>· Eric X. Liu's Personal Page</title><metacharset=utf-8><metaname=viewportcontent="width=device-width,initial-scale=1"><metaname=color-schemecontent="light dark"><metaname=authorcontent="Eric X. Liu"><metaname=descriptioncontent="Large Language Models (LLMs) have demonstrated astonishing capabilities, but out-of-the-box, they are simply powerful text predictors. They don’t inherently understand what makes a response helpful, harmless, or aligned with human values. The technique that has proven most effective at bridging this gap is Reinforcement Learning from Human Feedback (RLHF), and at its heart lies a powerful algorithm: Proximal Policy Optimization (PPO).
You may have seen diagrams like the one below, which outlines the RLHF training process. It can look intimidating, with a web of interconnected models, losses, and data flows."><metaname=keywordscontent="software engineer,performance engineering,Google engineer,tech blog,software development,performance optimization,Eric Liu,engineering blog,mountain biking,Jeep enthusiast,overlanding,camping,outdoor adventures"><metaname=fediverse:creatorcontent><metaname=twitter:cardcontent="summary"><metaname=twitter:titlecontent="Eric X. Liu's Personal Page"><metaname=twitter:descriptioncontent="Large Language Models (LLMs) have demonstrated astonishing capabilities, but out-of-the-box, they are simply powerful text predictors. They don’t inherently understand what makes a response helpful, harmless, or aligned with human values. The technique that has proven most effective at bridging this gap is Reinforcement Learning from Human Feedback (RLHF), and at its heart lies a powerful algorithm: Proximal Policy Optimization (PPO).
You may have seen diagrams like the one below, which outlines the RLHF training process. It can look intimidating, with a web of interconnected models, losses, and data flows."><metaproperty="og:url"content="/posts/a-deep-dive-into-ppo-for-language-models/"><metaproperty="og:site_name"content="Eric X. Liu's Personal Page"><metaproperty="og:title"content="Eric X. Liu's Personal Page"><metaproperty="og:description"content="Large Language Models (LLMs) have demonstrated astonishing capabilities, but out-of-the-box, they are simply powerful text predictors. They don’t inherently understand what makes a response helpful, harmless, or aligned with human values. The technique that has proven most effective at bridging this gap is Reinforcement Learning from Human Feedback (RLHF), and at its heart lies a powerful algorithm: Proximal Policy Optimization (PPO).
You may have seen diagrams like the one below, which outlines the RLHF training process. It can look intimidating, with a web of interconnected models, losses, and data flows."><metaproperty="og:locale"content="en"><metaproperty="og:type"content="article"><metaproperty="article:section"content="posts"><metaproperty="article:modified_time"content="2025-08-02T15:46:24-07:00"><linkrel=canonicalhref=/posts/a-deep-dive-into-ppo-for-language-models/><linkrel=preloadhref=/fonts/fa-brands-400.woff2as=fonttype=font/woff2crossorigin><linkrel=preloadhref=/fonts/fa-regular-400.woff2as=fonttype=font/woff2crossorigin><linkrel=preloadhref=/fonts/fa-solid-900.woff2as=fonttype=font/woff2crossorigin><linkrel=stylesheethref=/css/coder.min.60f552a2c0452fcc0254c54f21c3e0728460c1ae85f97a9c35833a222ef8b884.cssintegrity="sha256-YPVSosBFL8wCVMVPIcPgcoRgwa6F+XqcNYM6Ii74uIQ="crossorigin=anonymousmedia=screen><linkrel=stylesheethref=/css/coder-dark.min.a00e6364bacbc8266ad1cc81230774a1397198f8cfb7bcba29b7d6fcb54ce57f.cssintegrity="sha256-oA5jZLrLyCZq0cyBIwd0oTlxmPjPt7y6KbfW/LVM5X8="crossorigin=anonymousmedia=screen><linkrel=icontype=image/svg+xmlhref=/images/favicon.svgsizes=any><linkrel=icontype=image/pnghref=/images/favicon-32x32.pngsizes=32x32><linkrel=icontype=image/pnghref=/images/favicon-16x16.pngsizes=16x16><linkrel=apple-touch-iconhref=/images/apple-touch-icon.png><linkrel=apple-touch-iconsizes=180x180href=/images/apple-touch-icon.png><linkrel=manifesthref=/site.webmanifest><linkrel=mask-iconhref=/images/safari-pinned-tab.svgcolor=#5bbad5></head><bodyclass="preload-transitions colorscheme-auto"><divclass=float-container><aid=dark-mode-toggleclass=colorscheme-toggle><iclass="fa-solid fa-adjust fa-fw"aria-hidden=true></i></a></div><mainclass=wrapper><navclass=navigation><sectionclass=container><aclass=navigation-titlehref=/>Eric X. Liu's Personal Page
7-minute read</span></div></div></header><divclass=post-content><p>Large Language Models (LLMs) have demonstrated astonishing capabilities, but out-of-the-box, they are simply powerful text predictors. They don’t inherently understand what makes a response helpful, harmless, or aligned with human values. The technique that has proven most effective at bridging this gap is Reinforcement Learning from Human Feedback (RLHF), and at its heart lies a powerful algorithm: Proximal Policy Optimization (PPO).</p><p>You may have seen diagrams like the one below, which outlines the RLHF training process. It can look intimidating, with a web of interconnected models, losses, and data flows.</p><p>![[Pasted image 20250730232756.png]]</p><p>This post will decode that diagram, piece by piece. We’ll explore the “why” behind each component, moving from high-level concepts to the deep technical reasoning that makes this process work.</p><h3id=translating-rl-to-a-conversation>Translating RL to a Conversation
<aclass=heading-linkhref=#translating-rl-to-a-conversation><iclass="fa-solid fa-link"aria-hidden=truetitle="Link to heading"></i>
<spanclass=sr-only>Link to heading</span></a></h3><p>The first step is to understand how the traditional language of reinforcement learning maps to the world of text generation.</p><ul><li><strong>State (<code>s_t</code>)</strong>: In a chat setting, the “state” is the context of the conversation so far. It’s the initial prompt (<code>x</code>) plus all the text the model has generated up to the current moment (<code>y₁, ..., y_{t-1}</code>).</li><li><strong>Action (<code>a_t</code>)</strong>: The “action” is the model’s decision at each step. For an LLM, this means generating the very next token (<code>y_t</code>). A full response is a sequence of these actions.blob:https://aistudio.google.com/872e746f-88c1-40ec-8e45-fa0efce97299</li><li><strong>Reward (<code>r</code>)</strong>: The “reward” is a numeric score that tells the model how good its full response (<code>y</code>) was. This score comes from a separate <strong>Reward Model</strong>, which has been trained on a large dataset of human preference comparisons (e.g., humans rating which of two responses is better). This reward is often only awarded at the end of the entire generated sequence.</li></ul><p>Let’s make this concrete. If a user provides the prompt <strong>(x)</strong>: <em>“The best thing about AI is”</em>, and the model generates the response <strong>(y)</strong>: <em>“its potential to solve problems.”</em>, here is how it’s broken down for training:</p><ul><li><strong>State 1</strong>: “The best thing about AI is”<ul><li><strong>Action 1</strong>: “its”</li></ul></li><li><strong>State 2</strong>: “The best thing about AI is its”<ul><li><strong>Action 2</strong>: " potential"</li></ul></li><li><strong>State 3</strong>: “The best thing about AI is its potential”<ul><li><strong>Action 3</strong>: " to"</li></ul></li><li>…and so on for every generated token.</li></ul><p>This breakdown transforms a single prompt-response pair into a rich trajectory of state-action pairs, which becomes the raw data for our learning algorithm.</p><h3id=the-cast-of-models-an-actor-critic-ensemble>The Cast of Models: An Actor-Critic Ensemble
<aclass=heading-linkhref=#the-cast-of-models-an-actor-critic-ensemble><iclass="fa-solid fa-link"aria-hidden=truetitle="Link to heading"></i>
<spanclass=sr-only>Link to heading</span></a></h3><p>The PPO process doesn’t rely on a single model but an ensemble where each member has a distinct role.</p><ol><li><strong>The Actor (Policy LM)</strong>: This is the star of the show—the LLM we are actively fine-tuning. Its role is to take a state (the current text) and decide on an action (the next token). We refer to its decision-making process as its “policy” (<code>π</code>).</li><li><strong>The Critic (Value Model)</strong>: This is the Actor’s coach. The Critic doesn’t generate text. Instead, it observes a state and estimates the <em>potential future reward</em> the Actor is likely to receive from that point onward. This estimate is called the “value” (<code>V(s_t)</code>). The Critic’s feedback helps the Actor understand whether it’s in a promising or a dead-end situation, which is a much more immediate learning signal than waiting for the final reward.</li><li><strong>The Reward Model</strong>: This is the ultimate judge. As mentioned, it’s a separate model trained on human preference data that provides the final score for a complete generation. Its judgment is treated as the ground truth for training both the Actor and the Critic.</li></ol><h3id=the-challenge-of-credit-assignment-generalized-advantage-estimation-gae>The Challenge of Credit Assignment: Generalized Advantage Estimation (GAE)
<aclass=heading-linkhref=#the-challenge-of-credit-assignment-generalized-advantage-estimation-gae><iclass="fa-solid fa-link"aria-hidden=truetitle="Link to heading"></i>
<spanclass=sr-only>Link to heading</span></a></h3><p>A key problem in RL is assigning credit. If a 20-token response gets a high reward, was it because of the first token, the last one, or all of them? The Critic helps solve this. By comparing the reward at each step with the Critic’s value estimate, we can calculate the <strong>Advantage (<code>Â</code>)</strong>.</p><p>A simple advantage calculation might be: <code>Advantage = reward + Value_of_next_state - Value_of_current_state</code>.</p><p>However, this can be noisy. PPO uses a more sophisticated technique called <strong>Generalized Advantage Estimation (GAE)</strong>. The formula looks complex, but the idea is intuitive:</p><p><code>Â(s_t, a_t) = Σ(γλ)^l * δ_{t+l}</code>
where <code>δ_t = r_t + γV(s_{t+1}) - V(s_t)</code></p><ul><li><strong>γ (gamma)</strong> is a discount factor (e.g., 0.99), which values immediate rewards slightly more than distant ones.</li><li><strong>λ (lambda)</strong> is a smoothing parameter that balances the trade-off between bias and variance. It creates a weighted average of advantages over multiple future time steps.</li></ul><p>In essence, GAE provides a more stable and accurate estimate of how much better a specific action was compared to the policy’s average behavior in that state.</p><h3id=the-heart-of-ppo-the-quest-for-stable-updates>The Heart of PPO: The Quest for Stable Updates
<aclass=heading-linkhref=#the-heart-of-ppo-the-quest-for-stable-updates><iclass="fa-solid fa-link"aria-hidden=truetitle="Link to heading"></i>
<spanclass=sr-only>Link to heading</span></a></h3><p>Now we arrive at the core innovation of PPO. We want to update our Actor model to take actions with higher advantages. The naive way to do this is to re-weight our training objective by an <strong>importance sampling ratio</strong>: <code>(π_new / π_old)</code>. This corrects for the fact that the data we are learning from was generated by a slightly older version of our policy.</p><p>However, this ratio is incredibly dangerous. If the new policy becomes very different from the old one, the ratio can explode, leading to massive, unstable gradient updates that destroy the model.</p><p>PPO solves this with its signature <strong>Clipped Surrogate Objective</strong>. The PPO loss function is:</p><p><code>L_CLIP(θ) = Ê_t [ min( r_t(θ)Â_t, clip(r_t(θ), 1 - ε, 1 + ε)Â_t ) ]</code></p><p>Let’s translate this from math to English:</p><ul><li><code>r_t(θ)</code> is the probability ratio <code>π_new(a_t|s_t) / π_old(a_t|s_t)</code>.</li><li>The goal is to increase the objective by an amount proportional to the advantage <code>Â_t</code>.</li><li><strong>The <code>clip</code> function is the crucial safeguard.</strong> It forbids the probability ratio from moving outside a small window (e.g., <code>[0.8, 1.2]</code>).</li></ul><p>This means the algorithm says: “Let’s update our policy to favor this good action. But if the required update would change the policy too drastically from the old one, we’ll ‘clip’ the update to a more modest size.” This creates a “trust region,” ensuring stable, incremental improvements.</p><h3id=avoiding-amnesia-the-pretraining-loss>Avoiding Amnesia: The Pretraining Loss
<aclass=heading-linkhref=#avoiding-amnesia-the-pretraining-loss><iclass="fa-solid fa-link"aria-hidden=truetitle="Link to heading"></i>
<spanclass=sr-only>Link to heading</span></a></h3><p>There’s one final problem. If we only optimize for the PPO loss, the model might learn to “hack” the reward model by generating repetitive or nonsensical text that gets a high score. In doing so, it could suffer from <strong>catastrophic forgetting</strong>, losing its fundamental grasp of grammar and facts.</p><p>To prevent this, we introduce a second loss term. As seen in the diagram, we mix in data from the original <strong>Pretraining Data</strong> (or the dataset used for Supervised Fine-Tuning). We calculate a standard next-token prediction loss (<code>LM Loss</code>) on this high-quality data.</p><p>The final loss for the Actor is a combination of both objectives:</p><p><strong>Total Loss = Loss_PPO + <code>λ_ptx</code> * Loss_LM</strong></p><p>This brilliantly balances two goals:</p><ol><li>The <code>Loss_PPO</code> pushes the model towards behaviors that align with human preferences.</li><li>The <code>Loss_LM</code> acts as a regularizer, pulling the model back towards its core language capabilities and preventing it from drifting into gibberish.</li></ol><h3id=the-full-training-loop>The Full Training Loop
<aclass=heading-linkhref=#the-full-training-loop><iclass="fa-solid fa-link"aria-hidden=truetitle="Link to heading"></i>
<?xml version="1.0" encoding="utf-8" standalone="yes"?><rssversion="2.0"xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Posts on Eric X. Liu's Personal Page</title><link>/posts/</link><description>Recent content in Posts on Eric X. Liu's Personal Page</description><generator>Hugo</generator><language>en</language><lastBuildDate>Mon, 26 Oct 2020 04:47:36 +0000</lastBuildDate><atom:linkhref="/posts/index.xml"rel="self"type="application/rss+xml"/><item><title>Some useful files</title><link>/posts/useful/</link><pubDate>Mon, 26 Oct 2020 04:14:43 +0000</pubDate><guid>/posts/useful/</guid><description><ul>
<?xml version="1.0" encoding="utf-8" standalone="yes"?><rssversion="2.0"xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Posts on Eric X. Liu's Personal Page</title><link>/posts/</link><description>Recent content in Posts on Eric X. Liu's Personal Page</description><generator>Hugo</generator><language>en</language><lastBuildDate>Sat, 02 Aug 2025 15:46:24 -0700</lastBuildDate><atom:linkhref="/posts/index.xml"rel="self"type="application/rss+xml"/><item><title>Some useful files</title><link>/posts/useful/</link><pubDate>Mon, 26 Oct 2020 04:14:43 +0000</pubDate><guid>/posts/useful/</guid><description><ul>
</ul></description></item><item><title/><link>/posts/a-deep-dive-into-ppo-for-language-models/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/posts/a-deep-dive-into-ppo-for-language-models/</guid><description><p>Large Language Models (LLMs) have demonstrated astonishing capabilities, but out-of-the-box, they are simply powerful text predictors. They don&rsquo;t inherently understand what makes a response helpful, harmless, or aligned with human values. The technique that has proven most effective at bridging this gap is Reinforcement Learning from Human Feedback (RLHF), and at its heart lies a powerful algorithm: Proximal Policy Optimization (PPO).</p>
<p>You may have seen diagrams like the one below, which outlines the RLHF training process. It can look intimidating, with a web of interconnected models, losses, and data flows.</p></description></item></channel></rss>
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
eric