7.2: The Bias-Variance Trade Off

Double Descent Phenomenon

Double descent describes the non-monotonic relationship between model complexity and test error:

• First Descent (Classical Regime): As model complexity increases, the test error initially decreases as the model moves from high-bias (underfitting) to a better fit.

• The Critical Peak: As complexity grows further, error begins to rise sharply, reaching a peak at the interpolation threshold. This is the point where the model has just enough parameters to perfectly fit (interpolate) all training data, often capturing noise as if it were signal.

• Second Descent (Modern Regime): Surprisingly, increasing complexity beyond the interpolation threshold into the “overparameterized” regime causes the test error to decrease again. In this regime, even though the model perfectly fits the training data, its generalization on unseen data improves, often achieving even better results than the “optimal” model in the classical regime.

Why It Happens? While research is ongoing, several mechanisms are believed to drive this behavior:

• Inductive Bias & Implicit Regularization: Optimization algorithms like Stochastic Gradient Descent (SGD) tend to select the “simplest” or “smoothest” solution among the many possible functions that fit well with the training data.

• Smoothing Effect: In the highly overparameterized regime, the model has so many degrees of freedom that it can fit the training data without becoming erratic between points, leading to a smoother, more generalizable function.

• Effective Dimension: Larger models may have a lower “effective” dimension than their parameter count suggests, focusing only on the most relevant features for generalization.

Practical Implications

• Rethinking Model Size: It suggests that “bigger is/could be better,” which underpins the success of massive models like Transformers and LLMs.

• Data Noise Sensitivity: The peak at the interpolation threshold is significantly amplified by noisy data or incorrect labels.

• Optimization over Selection: Choosing the right optimization path (like SGD) and training duration is as critical as choosing the model architecture itself.

Recent Developments in LLMs

In LLMs, double descent manifests in scale-up laws: ImageNet-CNNs, ResNets, Transformers show test loss peaking at train-fit threshold, reviving with more parameters and data.

• Epoch-wise Double Descent: For fixed large model, loss decreases, spikes at mid-training (overfitting), then descends longer via SGD dynamics, reversing overfitting.

• Grokking (“delayed generalization”): it is a phenomenon where a model suddenly transitions from overfitting to perfect generalization after training for a long period (many epochs) beyond initial convergence. Grokking is a delayed generalization over time (epoch), which is related to double descent (delayed generalization over model size/complexity). Some researchers are exploring a unified approach via “staggered learning.”

• Research indicates that double descent, which once thought to be unique to supervised learning, also manifests in unsupervised and self-supervised learning (though its appearance is more sensitive to model architecture and data noise).

Further Exploration

1. Bias-Variance Decomposition Derivation

   Suppose the true relationship is y = f(x) + ϵ, where 𝔼[ϵ] = 0 and Var(ϵ) = σ². Let f̂ be our model estimate trained on a dataset D. The expected squared error at a point x (Err(x)) is (with expectations being taken over both the noise ϵ and the choice of training data D)
   $$\begin{aligned} \text{Err}(x) &= \mathbb{E}\left[(y - \hat{f})^2\right] \\ &= \mathbb{E}\left[(f(x) + \epsilon - \hat{f})^2\right] \\ &= \mathbb{E}\left[(f(x) - \hat{f})^2\right] + \mathbb{E}[\epsilon^2] + 2\mathbb{E}[(f(x) - \hat{f}) \, \epsilon]\end{aligned}$$
   Since ϵ is independent of the training data and 𝔼[ϵ] = 0, the cross-term vanishes. Thus: $$ \text{Err}(x) = \mathbb{E}\left[(f(x) - \hat{f})^2\right] + \sigma^2. $$
   Now, we decompose the term 𝔼[(f(x)−f̂)²] by adding and subtracting 𝔼[f̂]:
   $$\begin{aligned} \mathbb{E}\left[(f(x) - \hat{f})^2\right] &= \mathbb{E}\left[ \left(f(x) - \mathbb{E}[\hat{f}] + \mathbb{E}[\hat{f}] - \hat{f}\right)^2 \right] \\ &= \mathbb{E}\left[ (f(x) - \mathbb{E}[\hat{f}])^2 \right] + \mathbb{E}\left[ (\mathbb{E}[\hat{f}] - \hat{f})^2 \right] + 2\mathbb{E}[(f(x) - \mathbb{E}[\hat{f}])(\mathbb{E}[\hat{f}] - \hat{f})]\end{aligned}$$
   Note that:

   This gives the final decomposition (which is ([07a=01])):
   $$\text{Err}(x) = \underbrace{(f(x) - \mathbb{E}[\hat{f}])^2}_{\text{Bias}^2} + \underbrace{\mathbb{E}\left[(\hat{f} - \mathbb{E}[\hat{f}])^2\right]}_{\text{Variance}} + \underbrace{\sigma^2}_{\text{Irreducible Error}}.$$ Note that:

   • (f(x) − 𝔼[f̂])² is deviation relative to the expectation, so 𝔼[(f(x) − 𝔼[f̂])²] = (f(x) − 𝔼[f̂])² (Bias²).

   • 𝔼[(f̂−𝔼[f̂])²] is the Variance of f̂.

   • The cross-term is zero because 𝔼[𝔼[f̂] − f̂] = 𝔼[f̂] − 𝔼[f̂] = 0.

   • This gives the final decomposition
     $$
     \text{Err}(x) = 
     \underbrace{(f(x) - \mathbb{E}[\hat{f}])^2}_{\text{Bias}^2} 
     + \underbrace{\mathbb{E}\left[(\hat{f} - \mathbb{E}[\hat{f}])^2\right]}_{\text{Variance}} 
     + \underbrace{\sigma^2}_{\text{Irreducible Error}}. $$

2. Key papers on double descent:

   • P. Nakkiran et al., “Deep Double Descent: Where Bigger Models and More Data Hurt” (ICLR 2020, arXiv:1912.02292).

   • A. Power et al., “Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets” (arXiv:2201.02177).

   • Y. Huang et al., “Unified View of Grokking, Double Descent and Emergent Abilities” (arXiv:2402.15175).

   • Y. Gu et al., “Unraveling the Enigma of Double Descent” (arXiv:2310.13572).

   • M. Belkin et al., “Reconciling modern machine learning practice and the classical bias-variance tradeoff” (PNAS 2019).