diff --git a/404.html b/404.html index 91b4f21..e9fbd70 100644 --- a/404.html +++ b/404.html @@ -4,4 +4,4 @@ 2016 - 2025 Eric X. Liu -[5706ff7] \ No newline at end of file +[eba296f] \ No newline at end of file diff --git a/about/index.html b/about/index.html index 7a63b22..4097fbb 100644 --- a/about/index.html +++ b/about/index.html @@ -1,7 +1,7 @@ About · Eric X. Liu's Personal Page
\ No newline at end of file +[eba296f] \ No newline at end of file diff --git a/categories/index.html b/categories/index.html index f13e225..a25e4bd 100644 --- a/categories/index.html +++ b/categories/index.html @@ -4,4 +4,4 @@ 2016 - 2025 Eric X. Liu -[5706ff7] \ No newline at end of file +[eba296f] \ No newline at end of file diff --git a/index.html b/index.html index 4034f50..7ec884a 100644 --- a/index.html +++ b/index.html @@ -4,4 +4,4 @@ 2016 - 2025 Eric X. Liu -[5706ff7] \ No newline at end of file +[eba296f] \ No newline at end of file diff --git a/index.xml b/index.xml index d8dec86..680d753 100644 --- a/index.xml +++ b/index.xml @@ -1,4 +1,4 @@ -Eric X. Liu's Personal Page/Recent content on Eric X. Liu's Personal PageHugoenSun, 03 Aug 2025 04:20:20 +0000A Deep Dive into PPO for Language Models/posts/a-deep-dive-into-ppo-for-language-models/Sat, 02 Aug 2025 00:00:00 +0000/posts/a-deep-dive-into-ppo-for-language-models/<p>Large Language Models (LLMs) have demonstrated astonishing capabilities, but out-of-the-box, they are simply powerful text predictors. They don&rsquo;t inherently understand what makes a response helpful, harmless, or aligned with human values. The technique that has proven most effective at bridging this gap is Reinforcement Learning from Human Feedback (RLHF), and at its heart lies a powerful algorithm: Proximal Policy Optimization (PPO).</p> +Eric X. Liu's Personal Page/Recent content on Eric X. Liu's Personal PageHugoenSun, 03 Aug 2025 06:02:48 +0000A Deep Dive into PPO for Language Models/posts/a-deep-dive-into-ppo-for-language-models/Sat, 02 Aug 2025 00:00:00 +0000/posts/a-deep-dive-into-ppo-for-language-models/<p>Large Language Models (LLMs) have demonstrated astonishing capabilities, but out-of-the-box, they are simply powerful text predictors. They don&rsquo;t inherently understand what makes a response helpful, harmless, or aligned with human values. The technique that has proven most effective at bridging this gap is Reinforcement Learning from Human Feedback (RLHF), and at its heart lies a powerful algorithm: Proximal Policy Optimization (PPO).</p> <p>You may have seen diagrams like the one below, which outlines the RLHF training process. It can look intimidating, with a web of interconnected models, losses, and data flows.</p>Mixture-of-Experts (MoE) Models Challenges & Solutions in Practice/posts/mixture-of-experts-moe-models-challenges-solutions-in-practice/Wed, 02 Jul 2025 00:00:00 +0000/posts/mixture-of-experts-moe-models-challenges-solutions-in-practice/<p>Mixture-of-Experts (MoEs) are neural network architectures that allow different parts of the model (called &ldquo;experts&rdquo;) to specialize in different types of inputs. A &ldquo;gating network&rdquo; or &ldquo;router&rdquo; learns to dispatch each input (or &ldquo;token&rdquo;) to a subset of these experts. While powerful for scaling models, MoEs introduce several practical challenges.</p> <h3 id="1-challenge-non-differentiability-of-routing-functions"> 1. Challenge: Non-Differentiability of Routing Functions @@ -8,7 +8,7 @@ </a> </h3> <p><strong>The Problem:</strong> -Many routing mechanisms, especially &ldquo;Top-K routing,&rdquo; involve a discrete, hard selection process. A common function is <code>KeepTopK(v, k)</code>, which selects the top <code>k</code> scoring elements from a vector <code>v</code> and sets others to (-\infty) or (0).</p>An Architectural Deep Dive of T5/posts/t5-the-transformer-that-zigged-when-others-zagged-an-architectural-deep-dive/Sun, 01 Jun 2025 00:00:00 +0000/posts/t5-the-transformer-that-zigged-when-others-zagged-an-architectural-deep-dive/<p>In the rapidly evolving landscape of Large Language Models, a few key architectures define the dominant paradigms. Today, the &ldquo;decoder-only&rdquo; model, popularized by the GPT series and its successors like LLaMA and Mistral, reigns supreme. These models are scaled to incredible sizes and excel at in-context learning.</p> +Many routing mechanisms, especially &ldquo;Top-K routing,&rdquo; involve a discrete, hard selection process. A common function is <code>KeepTopK(v, k)</code>, which selects the top <code>k</code> scoring elements from a vector <code>v</code> and sets others to $-\infty$ or $0$.</p>An Architectural Deep Dive of T5/posts/t5-the-transformer-that-zigged-when-others-zagged-an-architectural-deep-dive/Sun, 01 Jun 2025 00:00:00 +0000/posts/t5-the-transformer-that-zigged-when-others-zagged-an-architectural-deep-dive/<p>In the rapidly evolving landscape of Large Language Models, a few key architectures define the dominant paradigms. Today, the &ldquo;decoder-only&rdquo; model, popularized by the GPT series and its successors like LLaMA and Mistral, reigns supreme. These models are scaled to incredible sizes and excel at in-context learning.</p> <p>But to truly understand the field, we must look at the pivotal models that explored different paths. Google&rsquo;s T5, or <strong>Text-to-Text Transfer Transformer</strong>, stands out as one of the most influential. It didn&rsquo;t just introduce a new model; it proposed a new philosophy. This article dives deep into the architecture of T5, how it fundamentally differs from modern LLMs, and the lasting legacy of its unique design choices.</p>Mastering Your Breville Barista Pro: The Ultimate Guide to Dialing In Espresso/posts/espresso-theory-application-a-guide-for-the-breville-barista-pro/Thu, 01 May 2025 00:00:00 +0000/posts/espresso-theory-application-a-guide-for-the-breville-barista-pro/<p>Are you ready to transform your home espresso game from good to genuinely great? The Breville Barista Pro is a fantastic machine, but unlocking its full potential requires understanding a few key principles. This guide will walk you through the systematic process of dialing in your espresso, ensuring every shot is delicious and repeatable.</p> <p>Our overarching philosophy is simple: <strong>isolate and change only one variable at a time.</strong> While numbers are crucial, your palate is the ultimate judge. Dose, ratio, and time are interconnected, but your <strong>grind size</strong> is your most powerful lever.</p>Some useful files/posts/useful/Mon, 26 Oct 2020 04:14:43 +0000/posts/useful/<ul> <li><a href="https://ericxliu.me/rootCA.pem" class="external-link" target="_blank" rel="noopener">rootCA.pem</a></li> diff --git a/posts/a-deep-dive-into-ppo-for-language-models/index.html b/posts/a-deep-dive-into-ppo-for-language-models/index.html index 91fe7b7..e3449b7 100644 --- a/posts/a-deep-dive-into-ppo-for-language-models/index.html +++ b/posts/a-deep-dive-into-ppo-for-language-models/index.html @@ -19,8 +19,8 @@ where δ_t = r_t + γV(s_{t+1}) - V(s_t)

  • γ (gam Link to heading

    There’s one final problem. If we only optimize for the PPO loss, the model might learn to “hack” the reward model by generating repetitive or nonsensical text that gets a high score. In doing so, it could suffer from catastrophic forgetting, losing its fundamental grasp of grammar and facts.

    To prevent this, we introduce a second loss term. As seen in the diagram, we mix in data from the original Pretraining Data (or the dataset used for Supervised Fine-Tuning). We calculate a standard next-token prediction loss (LM Loss) on this high-quality data.

    The final loss for the Actor is a combination of both objectives:

    Total Loss = Loss_PPO + λ_ptx * Loss_LM

    This brilliantly balances two goals:

    1. The Loss_PPO pushes the model towards behaviors that align with human preferences.
    2. The Loss_LM acts as a regularizer, pulling the model back towards its core language capabilities and preventing it from drifting into gibberish.

    The Full Training Loop -Link to heading

    Now, we can assemble the entire process into a clear, iterative loop:

    1. Collect: The current Actor policy π_k generates responses to a batch of prompts. These experiences—(state, action, probability, reward, value)—are stored in an Experience Buffer.
    2. Calculate: Once the buffer is full, we use the collected data to compute the advantage estimates Â_t for every single token-generation step.
    3. Optimize: For a few epochs, we repeatedly sample mini-batches from the buffer and update the Actor and Critic models. The Actor is updated using the combined PPO-clip Loss and LM Loss. The Critic is updated to improve its value predictions.
    4. Flush and Repeat: After the optimization phase, the entire experience buffer is discarded. The data is now “stale” because our policy has changed. The newly updated policy π_{k+1} becomes the new Actor, and we return to step 1 to collect fresh data.

    This cycle of collection and optimization allows the language model to gradually and safely steer its behavior towards human-defined goals, creating the helpful and aligned AI assistants we interact with today.

    References:

    1. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv preprint arXiv:1707.06347.
    2. Schulman, J., Moritz, P., Levine, S., Jordan, M., & Abbeel, P. (2015). High-Dimensional Continuous Control Using Generalized Advantage Estimation. arXiv preprint arXiv:1506.02438.
    3. Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35.
    © +Link to heading

    Now, we can assemble the entire process into a clear, iterative loop:

    1. Collect: The current Actor policy π_k generates responses to a batch of prompts. These experiences—(state, action, probability, reward, value)—are stored in an Experience Buffer.
    2. Calculate: Once the buffer is full, we use the collected data to compute the advantage estimates Â_t for every single token-generation step.
    3. Optimize: For a few epochs, we repeatedly sample mini-batches from the buffer and update the Actor and Critic models. The Actor is updated using the combined PPO-clip Loss and LM Loss. The Critic is updated to improve its value predictions.
    4. Flush and Repeat: After the optimization phase, the entire experience buffer is discarded. The data is now “stale” because our policy has changed. The newly updated policy π_{k+1} becomes the new Actor, and we return to step 1 to collect fresh data.

    This cycle of collection and optimization allows the language model to gradually and safely steer its behavior towards human-defined goals, creating the helpful and aligned AI assistants we interact with today.

    References:

    1. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv preprint arXiv:1707.06347.
    2. Schulman, J., Moritz, P., Levine, S., Jordan, M., & Abbeel, P. (2015). High-Dimensional Continuous Control Using Generalized Advantage Estimation. arXiv preprint arXiv:1506.02438.
    3. Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35.
    \ No newline at end of file +[eba296f]
    \ No newline at end of file diff --git a/posts/espresso-theory-application-a-guide-for-the-breville-barista-pro/index.html b/posts/espresso-theory-application-a-guide-for-the-breville-barista-pro/index.html index d33254c..4e53faa 100644 --- a/posts/espresso-theory-application-a-guide-for-the-breville-barista-pro/index.html +++ b/posts/espresso-theory-application-a-guide-for-the-breville-barista-pro/index.html @@ -16,8 +16,8 @@ Our overarching philosophy is simple: isolate and change only one variable at a Link to heading

    This is where the magic (and sometimes frustration) happens. Grind size is your main tool for controlling the resistance of the coffee puck, which directly dictates your brew time.

    The Dual Impact of Grinding Finer:

    1. Increases surface area: Allows for more efficient flavor extraction.
    2. Increases resistance: Slows down water flow and increases contact time.

    The Risk of Grinding Too Fine (Channeling):

    If the grind is too fine, the puck becomes so dense that high-pressure water can’t flow evenly. Instead, it “breaks” the puck and punches an easy path (a channel) through a weak spot. This results in a disastrous shot that is simultaneously:

    • Under-extracted: Most of the coffee is bypassed.
    • Over-extracted: The water that does flow blasts through the channel, extracting harsh, bitter compounds.
    • The Taste: A channeled shot tastes hollow, weak, sour, and bitter all at once.

    The Goal: You want to grind as fine as you possibly can without causing significant channeling. This is the sweet spot for maximizing surface area and resistance for high, even extraction.

    Grind Retention (Purging): Most grinders retain some old grounds. When you change your grind setting, always purge a few grams of coffee to ensure your dose is entirely at the new setting.

    Application for Your Breville Barista Pro:

    • Grinder Mechanism: The “Grind Amount” dial controls the TIME the grinder runs, not the weight. When you adjust the fineness, you must re-adjust the grind time to ensure you are still getting your target 18g dose.
    • Tackling Channeling: The Barista Pro is prone to channeling. To fight this, focus on excellent puck prep: use a WDT (Weiss Distribution Technique) tool to break up clumps and evenly distribute the grounds before tamping levelly.

    The Complete Dialing-In Workflow -Link to heading

    This systematic process will get you to a delicious shot from your Breville Barista Pro efficiently:

    1. Set Your Constants:
      • Dose: 18g.
      • Ratio: 1:2 (meaning a Yield of 36g).
      • Pre-infusion: Use a consistent method (e.g., manual 8-second hold).
    2. Make an Initial Grind:
      • Set the grinder to a starting point of 15.
      • Adjust the grind time until the grinder dispenses exactly 18g.
    3. Pull the First Shot:
      • Brew manually, stopping at 36g of liquid in the cup. Note the total brew time.
    4. Taste and Diagnose:
      • Fast & Sour? (<25s): Grind is too coarse.
      • Slow & Bitter? (>32s): Grind is too fine.
    5. Make ONE Adjustment - THE GRIND SIZE:
      • If fast/sour, adjust the grind finer (e.g., from 15 down to 13).
      • If slow/bitter, adjust the grind coarser (e.g., from 15 up to 17).
    6. Re-adjust and Repeat:
      • After changing the grind setting, purge a small amount of coffee.
      • Re-weigh your next dose and adjust the grind time to get back to exactly 18g.
      • Pull another 36g shot. Repeat this process until your shot tastes balanced and the time falls roughly between 25-32 seconds.

    Happy brewing! With patience and this systematic approach, you’ll be pulling consistently delicious espresso shots from your Breville Barista Pro in no time.

    © +Link to heading

    This systematic process will get you to a delicious shot from your Breville Barista Pro efficiently:

    1. Set Your Constants:
      • Dose: 18g.
      • Ratio: 1:2 (meaning a Yield of 36g).
      • Pre-infusion: Use a consistent method (e.g., manual 8-second hold).
    2. Make an Initial Grind:
      • Set the grinder to a starting point of 15.
      • Adjust the grind time until the grinder dispenses exactly 18g.
    3. Pull the First Shot:
      • Brew manually, stopping at 36g of liquid in the cup. Note the total brew time.
    4. Taste and Diagnose:
      • Fast & Sour? (<25s): Grind is too coarse.
      • Slow & Bitter? (>32s): Grind is too fine.
    5. Make ONE Adjustment - THE GRIND SIZE:
      • If fast/sour, adjust the grind finer (e.g., from 15 down to 13).
      • If slow/bitter, adjust the grind coarser (e.g., from 15 up to 17).
    6. Re-adjust and Repeat:
      • After changing the grind setting, purge a small amount of coffee.
      • Re-weigh your next dose and adjust the grind time to get back to exactly 18g.
      • Pull another 36g shot. Repeat this process until your shot tastes balanced and the time falls roughly between 25-32 seconds.

    Happy brewing! With patience and this systematic approach, you’ll be pulling consistently delicious espresso shots from your Breville Barista Pro in no time.

    \ No newline at end of file +[eba296f]
    \ No newline at end of file diff --git a/posts/index.html b/posts/index.html index e9e4c7f..faa45e3 100644 --- a/posts/index.html +++ b/posts/index.html @@ -9,4 +9,4 @@ 2016 - 2025 Eric X. Liu -[5706ff7] \ No newline at end of file +[eba296f] \ No newline at end of file diff --git a/posts/index.xml b/posts/index.xml index 1876b5e..d435e64 100644 --- a/posts/index.xml +++ b/posts/index.xml @@ -1,4 +1,4 @@ -Posts on Eric X. Liu's Personal Page/posts/Recent content in Posts on Eric X. Liu's Personal PageHugoenSun, 03 Aug 2025 04:20:20 +0000A Deep Dive into PPO for Language Models/posts/a-deep-dive-into-ppo-for-language-models/Sat, 02 Aug 2025 00:00:00 +0000/posts/a-deep-dive-into-ppo-for-language-models/<p>Large Language Models (LLMs) have demonstrated astonishing capabilities, but out-of-the-box, they are simply powerful text predictors. They don&rsquo;t inherently understand what makes a response helpful, harmless, or aligned with human values. The technique that has proven most effective at bridging this gap is Reinforcement Learning from Human Feedback (RLHF), and at its heart lies a powerful algorithm: Proximal Policy Optimization (PPO).</p> +Posts on Eric X. Liu's Personal Page/posts/Recent content in Posts on Eric X. Liu's Personal PageHugoenSun, 03 Aug 2025 06:02:48 +0000A Deep Dive into PPO for Language Models/posts/a-deep-dive-into-ppo-for-language-models/Sat, 02 Aug 2025 00:00:00 +0000/posts/a-deep-dive-into-ppo-for-language-models/<p>Large Language Models (LLMs) have demonstrated astonishing capabilities, but out-of-the-box, they are simply powerful text predictors. They don&rsquo;t inherently understand what makes a response helpful, harmless, or aligned with human values. The technique that has proven most effective at bridging this gap is Reinforcement Learning from Human Feedback (RLHF), and at its heart lies a powerful algorithm: Proximal Policy Optimization (PPO).</p> <p>You may have seen diagrams like the one below, which outlines the RLHF training process. It can look intimidating, with a web of interconnected models, losses, and data flows.</p>Mixture-of-Experts (MoE) Models Challenges & Solutions in Practice/posts/mixture-of-experts-moe-models-challenges-solutions-in-practice/Wed, 02 Jul 2025 00:00:00 +0000/posts/mixture-of-experts-moe-models-challenges-solutions-in-practice/<p>Mixture-of-Experts (MoEs) are neural network architectures that allow different parts of the model (called &ldquo;experts&rdquo;) to specialize in different types of inputs. A &ldquo;gating network&rdquo; or &ldquo;router&rdquo; learns to dispatch each input (or &ldquo;token&rdquo;) to a subset of these experts. While powerful for scaling models, MoEs introduce several practical challenges.</p> <h3 id="1-challenge-non-differentiability-of-routing-functions"> 1. Challenge: Non-Differentiability of Routing Functions @@ -8,7 +8,7 @@ </a> </h3> <p><strong>The Problem:</strong> -Many routing mechanisms, especially &ldquo;Top-K routing,&rdquo; involve a discrete, hard selection process. A common function is <code>KeepTopK(v, k)</code>, which selects the top <code>k</code> scoring elements from a vector <code>v</code> and sets others to (-\infty) or (0).</p>An Architectural Deep Dive of T5/posts/t5-the-transformer-that-zigged-when-others-zagged-an-architectural-deep-dive/Sun, 01 Jun 2025 00:00:00 +0000/posts/t5-the-transformer-that-zigged-when-others-zagged-an-architectural-deep-dive/<p>In the rapidly evolving landscape of Large Language Models, a few key architectures define the dominant paradigms. Today, the &ldquo;decoder-only&rdquo; model, popularized by the GPT series and its successors like LLaMA and Mistral, reigns supreme. These models are scaled to incredible sizes and excel at in-context learning.</p> +Many routing mechanisms, especially &ldquo;Top-K routing,&rdquo; involve a discrete, hard selection process. A common function is <code>KeepTopK(v, k)</code>, which selects the top <code>k</code> scoring elements from a vector <code>v</code> and sets others to $-\infty$ or $0$.</p>An Architectural Deep Dive of T5/posts/t5-the-transformer-that-zigged-when-others-zagged-an-architectural-deep-dive/Sun, 01 Jun 2025 00:00:00 +0000/posts/t5-the-transformer-that-zigged-when-others-zagged-an-architectural-deep-dive/<p>In the rapidly evolving landscape of Large Language Models, a few key architectures define the dominant paradigms. Today, the &ldquo;decoder-only&rdquo; model, popularized by the GPT series and its successors like LLaMA and Mistral, reigns supreme. These models are scaled to incredible sizes and excel at in-context learning.</p> <p>But to truly understand the field, we must look at the pivotal models that explored different paths. Google&rsquo;s T5, or <strong>Text-to-Text Transfer Transformer</strong>, stands out as one of the most influential. It didn&rsquo;t just introduce a new model; it proposed a new philosophy. This article dives deep into the architecture of T5, how it fundamentally differs from modern LLMs, and the lasting legacy of its unique design choices.</p>Mastering Your Breville Barista Pro: The Ultimate Guide to Dialing In Espresso/posts/espresso-theory-application-a-guide-for-the-breville-barista-pro/Thu, 01 May 2025 00:00:00 +0000/posts/espresso-theory-application-a-guide-for-the-breville-barista-pro/<p>Are you ready to transform your home espresso game from good to genuinely great? The Breville Barista Pro is a fantastic machine, but unlocking its full potential requires understanding a few key principles. This guide will walk you through the systematic process of dialing in your espresso, ensuring every shot is delicious and repeatable.</p> <p>Our overarching philosophy is simple: <strong>isolate and change only one variable at a time.</strong> While numbers are crucial, your palate is the ultimate judge. Dose, ratio, and time are interconnected, but your <strong>grind size</strong> is your most powerful lever.</p>Some useful files/posts/useful/Mon, 26 Oct 2020 04:14:43 +0000/posts/useful/<ul> <li><a href="https://ericxliu.me/rootCA.pem" class="external-link" target="_blank" rel="noopener">rootCA.pem</a></li> diff --git a/posts/mixture-of-experts-moe-models-challenges-solutions-in-practice/index.html b/posts/mixture-of-experts-moe-models-challenges-solutions-in-practice/index.html index b5183f2..6f59f07 100644 --- a/posts/mixture-of-experts-moe-models-challenges-solutions-in-practice/index.html +++ b/posts/mixture-of-experts-moe-models-challenges-solutions-in-practice/index.html @@ -7,9 +7,9 @@ The Problem: -Many routing mechanisms, especially “Top-K routing,” involve a discrete, hard selection process. A common function is KeepTopK(v, k), which selects the top k scoring elements from a vector v and sets others to (-\infty) or (0).">

2. Challenge: Uneven Expert Utilization (Balancing Loss) Link to heading

The Problem: Left unchecked, the gating network might learn to heavily favor a few experts, leaving others underutilized. This leads to:

  • System Inefficiency: Overloaded experts become bottlenecks, while underutilized experts waste computational resources.
  • Suboptimal Learning: Experts might not specialize effectively if they don’t receive diverse data.

Solution: Heuristic Balancing Losses (e.g., from Switch Transformer, Fedus et al. 2022) -An auxiliary loss is added to the total model loss during training to encourage more even expert usage.

(( \text{loss}{\text{auxiliary}} = \alpha \cdot N \cdot \sum{i=1}^{N} f_i \cdot P_i ))

Where:

  • (\alpha): A hyperparameter controlling the strength of the auxiliary loss.
  • (N): Total number of experts.
  • (f_i): The fraction of tokens actually dispatched to expert (i) in the current batch (B). -(( f_i = \frac{1}{T} \sum_{x \in B} \mathbf{1}{\text{argmax } p(x) = i} )) -((p(x)) here refers to the output of the gating network, which could be (s_{i,t}) in the DeepSeek/classic router. The (\text{argmax}) means it counts hard assignments to expert (i).)
  • (P_i): The fraction of the router probability mass allocated to expert (i) in the current batch (B). -(( P_i = \frac{1}{T} \sum_{x \in B} p_i(x) )) -((p_i(x)) is the learned probability (or soft score) from the gating network for token (x) and expert (i).)

How it works: -The loss aims to minimize the product (f_i \cdot P_i) when (f_i) and (P_i) are small, effectively pushing them to be larger (closer to (1/N)). If an expert (i) is overused (high (f_i) and (P_i)), its term in the sum contributes significantly to the loss. The derivative with respect to (p_i(x)) reveals that “more frequent use = stronger downweighting,” meaning the gating network is penalized for sending too much traffic to an already busy expert.

Relationship to Gating Network:

  • (p_i(x)) (or (s_{i,t})): This is the output of the learned gating network (e.g., from a linear layer followed by Softmax). The gating network’s parameters are updated via gradient descent, influenced by this auxiliary loss.
  • (P_i): This is calculated from the outputs of the learned gating network for the current batch. It’s not a pre-defined value.

Limitation (“Second Best” Scenario): -Even with this loss, an expert can remain imbalanced if it’s consistently the “second best” option (high (P_i)) but never the absolute top choice that gets counted in (f_i) (especially if (K=1)). This is because (f_i) strictly counts hard assignments based on argmax. This limitation highlights why “soft” routing or “softmax after TopK” approaches can be more effective for truly even distribution.

3. Challenge: Overfitting during Fine-tuning +An auxiliary loss is added to the total model loss during training to encourage more even expert usage.

$$ \text{loss}_{\text{auxiliary}} = \alpha \cdot N \cdot \sum_{i=1}^{N} f_i \cdot P_i $$

Where:

  • $\alpha$: A hyperparameter controlling the strength of the auxiliary loss.
  • $N$: Total number of experts.
  • $f_i$: The fraction of tokens actually dispatched to expert $i$ in the current batch $B$. +$$ f_i = \frac{1}{T} \sum_{x \in B} \mathbf{1}\{\text{argmax } p(x) = i\} $$ +($p(x)$ here refers to the output of the gating network, which could be $s_{i,t}$ in the DeepSeek/classic router. The $\text{argmax}$ means it counts hard assignments to expert $i$.)
  • $P_i$: The fraction of the router probability mass allocated to expert $i$ in the current batch $B$. +$$ P_i = \frac{1}{T} \sum_{x \in B} p_i(x) $$ +($p_i(x)$ is the learned probability (or soft score) from the gating network for token $x$ and expert $i$.)

How it works: +The loss aims to minimize the product $f_i \cdot P_i$ when $f_i$ and $P_i$ are small, effectively pushing them to be larger (closer to $1/N$). If an expert $i$ is overused (high $f_i$ and $P_i$), its term in the sum contributes significantly to the loss. The derivative with respect to $p_i(x)$ reveals that “more frequent use = stronger downweighting,” meaning the gating network is penalized for sending too much traffic to an already busy expert.

Relationship to Gating Network:

  • $p_i(x)$ (or $s_{i,t}$): This is the output of the learned gating network (e.g., from a linear layer followed by Softmax). The gating network’s parameters are updated via gradient descent, influenced by this auxiliary loss.
  • $P_i$: This is calculated from the outputs of the learned gating network for the current batch. It’s not a pre-defined value.

Limitation (“Second Best” Scenario): +Even with this loss, an expert can remain imbalanced if it’s consistently the “second best” option (high $P_i$) but never the absolute top choice that gets counted in $f_i$ (especially if $K=1$). This is because $f_i$ strictly counts hard assignments based on argmax. This limitation highlights why “soft” routing or “softmax after TopK” approaches can be more effective for truly even distribution.

3. Challenge: Overfitting during Fine-tuning Link to heading

The Problem: Sparse MoE models, despite only activating a few experts per token, possess a very large total number of parameters. When fine-tuning these models on smaller datasets, they are highly prone to overfitting. The model’s vast capacity allows it to memorize the limited fine-tuning data, leading to poor generalization performance on unseen validation data. This is evident when training loss continues to decrease, but validation loss stagnates or increases.

Solutions:

  • Zoph et al. Solution – Fine-tune non-MoE MLPs:

    • This strategy involves freezing a portion of the MoE model’s parameters during fine-tuning, specifically the large expert weights.
    • Instead, only the “non-MoE” parameters (e.g., attention layers, adapter layers, or the gating network itself) are updated.
    • This reduces the effective number of trainable parameters during fine-tuning, thereby mitigating the risk of overfitting on small datasets. It assumes the experts are already well-pre-trained for general tasks.

  • DeepSeek Solution – Use Lots of Data (1.4M SFT):

    • This approach tackles the problem by providing the model with a very large and diverse dataset for Supervised Fine-Tuning (SFT).
    • With abundant data (e.g., 1.4 million examples covering a wide range of tasks and languages), the model’s large capacity can be effectively utilized for specialized learning rather than memorization. The diversity and volume of data prevent individual experts from overfitting to specific examples.

Conclusion: MoE models offer significant advantages in terms of model capacity and computational efficiency, but their unique sparse activation pattern introduces challenges in training and fine-tuning. Overcoming non-differentiability in routing and ensuring balanced expert utilization are crucial for effective pre-training. During fine-tuning, managing the model’s vast parameter count to prevent overfitting on smaller datasets requires either strategic parameter freezing or access to very large and diverse fine-tuning data. -The Top-K routing mechanism, as illustrated in the provided image, is a core component in many modern Mixture-of-Experts (MoE) models. It involves selecting a fixed number (K) of experts for each input based on relevance scores.

Traditional Top-K (Deterministic Selection):

  • How it works:
    1. Calculate relevance scores (s_{i,t}) for each expert i and input t.
    2. Identify the K experts with the highest scores.
    3. Experts within the Top-K are assigned their scores (g_{i,t} = s_{i,t}).
    4. Experts outside the Top-K are assigned a score of 0 (g_{i,t} = 0).
    5. The output is a weighted sum of the selected experts’ outputs.
  • Pros: Predictable, deterministic, selects the “best” experts based on current scores.
  • Cons: Can lead to expert imbalance, where a few popular experts are always chosen, starving others of training.

Alternative: Sampling from Softmax (Probabilistic Selection):

  • How it works:
    1. Calculate relevance scores (s_{i,t}) which are treated as probabilities (after softmax).
    2. Randomly sample K unique expert indices from the distribution defined by these probabilities.
    3. Selected experts contribute; unselected experts do not.
  • Why it’s suggested:
    • Load Balancing: Prevents expert collapse by ensuring all experts get a chance to be selected, even those with slightly lower scores. This promotes more even training across the entire expert pool.
    • Diversity & Exploration: Introduces randomness, potentially leading to better generalization and robustness by exploring different expert combinations.
  • Pros: Better load balancing, prevents expert starvation, encourages exploration.
  • Cons: Stochastic (non-deterministic routing), can make debugging harder, might not pick the absolute “best” expert in a single instance (but better for long-term training).

Key Takeaway: While deterministic Top-K is simpler and directly picks the “highest-scoring” experts, sampling from the softmax offers a more robust training dynamic by ensuring that all experts receive training data, thereby preventing some experts from becoming unused (“dead experts”).

© +The Top-K routing mechanism, as illustrated in the provided image, is a core component in many modern Mixture-of-Experts (MoE) models. It involves selecting a fixed number (K) of experts for each input based on relevance scores.

Traditional Top-K (Deterministic Selection):

  • How it works:
    1. Calculate relevance scores (s_{i,t}) for each expert i and input t.
    2. Identify the K experts with the highest scores.
    3. Experts within the Top-K are assigned their scores (g_{i,t} = s_{i,t}).
    4. Experts outside the Top-K are assigned a score of 0 (g_{i,t} = 0).
    5. The output is a weighted sum of the selected experts’ outputs.
  • Pros: Predictable, deterministic, selects the “best” experts based on current scores.
  • Cons: Can lead to expert imbalance, where a few popular experts are always chosen, starving others of training.

Alternative: Sampling from Softmax (Probabilistic Selection):

  • How it works:
    1. Calculate relevance scores (s_{i,t}) which are treated as probabilities (after softmax).
    2. Randomly sample K unique expert indices from the distribution defined by these probabilities.
    3. Selected experts contribute; unselected experts do not.
  • Why it’s suggested:
    • Load Balancing: Prevents expert collapse by ensuring all experts get a chance to be selected, even those with slightly lower scores. This promotes more even training across the entire expert pool.
    • Diversity & Exploration: Introduces randomness, potentially leading to better generalization and robustness by exploring different expert combinations.
  • Pros: Better load balancing, prevents expert starvation, encourages exploration.
  • Cons: Stochastic (non-deterministic routing), can make debugging harder, might not pick the absolute “best” expert in a single instance (but better for long-term training).

Key Takeaway: While deterministic Top-K is simpler and directly picks the “highest-scoring” experts, sampling from the softmax offers a more robust training dynamic by ensuring that all experts receive training data, thereby preventing some experts from becoming unused (“dead experts”).

\ No newline at end of file +[eba296f]
\ No newline at end of file diff --git a/posts/t5-the-transformer-that-zigged-when-others-zagged-an-architectural-deep-dive/index.html b/posts/t5-the-transformer-that-zigged-when-others-zagged-an-architectural-deep-dive/index.html index f33df47..11dddda 100644 --- a/posts/t5-the-transformer-that-zigged-when-others-zagged-an-architectural-deep-dive/index.html +++ b/posts/t5-the-transformer-that-zigged-when-others-zagged-an-architectural-deep-dive/index.html @@ -26,8 +26,8 @@ But to truly understand the field, we must look at the pivotal models that explo Link to heading

Despite its differences, the “T5 v1.1” variant pioneered several techniques that are now standard practice in the most advanced LLMs:

  • RMSNorm: It was one of the first major models to adopt Root Mean Square Normalization instead of LayerNorm, a choice now used by LLaMA, Mistral, and others for its efficiency and stability.
  • Pre-Normalization: T5 applies the normalization layer before the attention and FFN blocks, a critical technique for enabling stable training of very deep networks.
  • No Bias Terms: T5 v1.1 removed the bias parameters from its normalization and FFN layers, a small but important optimization for memory and stability that modern models follow.
  • Gated Activations (GeGLU): While the original T5 used ReLU, T5 v1.1 adopted a Gated Linear Unit (GeGLU), presaging the move to GLU-family activations (like SwiGLU) that is now ubiquitous.

Conclusion: The Lasting Legacy -Link to heading

T5 represents a different evolutionary branch in the Transformer family tree. While the field has largely converged on the decoder-only architecture for its scalability in general-purpose models, T5’s design remains a masterclass in purpose-built engineering.

Its text-to-text framework was revolutionary, its encoder-decoder structure is still a go-to for tasks like translation, and its refined T5 v1.1 architecture laid the groundwork for many of the stability and efficiency tricks we see in today’s state-of-the-art models. T5 is more than just a model; it’s a crucial case study in the architectural trade-offs that continue to shape the future of artificial intelligence.

© +Link to heading

T5 represents a different evolutionary branch in the Transformer family tree. While the field has largely converged on the decoder-only architecture for its scalability in general-purpose models, T5’s design remains a masterclass in purpose-built engineering.

Its text-to-text framework was revolutionary, its encoder-decoder structure is still a go-to for tasks like translation, and its refined T5 v1.1 architecture laid the groundwork for many of the stability and efficiency tricks we see in today’s state-of-the-art models. T5 is more than just a model; it’s a crucial case study in the architectural trade-offs that continue to shape the future of artificial intelligence.

\ No newline at end of file +[eba296f]
\ No newline at end of file diff --git a/posts/useful/index.html b/posts/useful/index.html index f6e030c..9e12775 100644 --- a/posts/useful/index.html +++ b/posts/useful/index.html @@ -6,8 +6,8 @@ vpnclient.ovpn

Some useful files

-One-minute read
© +One-minute read
\ No newline at end of file +[eba296f]
\ No newline at end of file diff --git a/sitemap.xml b/sitemap.xml index 97059f8..661d8e9 100644 --- a/sitemap.xml +++ b/sitemap.xml @@ -1 +1 @@ -/posts/a-deep-dive-into-ppo-for-language-models/2025-08-03T03:28:39+00:00weekly0.5/2025-08-03T04:20:20+00:00weekly0.5/posts/2025-08-03T04:20:20+00:00weekly0.5/posts/mixture-of-experts-moe-models-challenges-solutions-in-practice/2025-08-03T03:49:59+00:00weekly0.5/posts/t5-the-transformer-that-zigged-when-others-zagged-an-architectural-deep-dive/2025-08-03T03:41:10+00:00weekly0.5/posts/espresso-theory-application-a-guide-for-the-breville-barista-pro/2025-08-03T04:20:20+00:00weekly0.5/posts/useful/2020-10-26T04:47:36+00:00weekly0.5/about/2020-06-16T23:30:17-07:00weekly0.5/categories/weekly0.5/tags/weekly0.5 \ No newline at end of file +/posts/a-deep-dive-into-ppo-for-language-models/2025-08-03T03:28:39+00:00weekly0.5/2025-08-03T06:02:48+00:00weekly0.5/posts/2025-08-03T06:02:48+00:00weekly0.5/posts/mixture-of-experts-moe-models-challenges-solutions-in-practice/2025-08-03T06:02:48+00:00weekly0.5/posts/t5-the-transformer-that-zigged-when-others-zagged-an-architectural-deep-dive/2025-08-03T03:41:10+00:00weekly0.5/posts/espresso-theory-application-a-guide-for-the-breville-barista-pro/2025-08-03T04:20:20+00:00weekly0.5/posts/useful/2020-10-26T04:47:36+00:00weekly0.5/about/2020-06-16T23:30:17-07:00weekly0.5/categories/weekly0.5/tags/weekly0.5 \ No newline at end of file diff --git a/tags/index.html b/tags/index.html index 6da50ea..74a50ea 100644 --- a/tags/index.html +++ b/tags/index.html @@ -4,4 +4,4 @@ 2016 - 2025 Eric X. Liu -[5706ff7] \ No newline at end of file +[eba296f] \ No newline at end of file