5.1: Chapter 5 Summary

Based on SD Chapter 5.

1. Sect. 5.1. Deep Learning(DL)/Neural Networks (NNs) as Non-linear Function Approximators

   Architecture and Input-Output Relation: Neural Networks (NNs) consist of an input layer, multiple hidden layers, and an output layer. Each layer is composed of neurons (nodes) connected to the next layer. When the NNs consist of multiple layers, we calll them deep neural networks (DNNs).

   Briefly:

   • Each neuron computes a weighted sum of its inputs, then applies a nonlinear activation function to introduce nonlinearity into the model.

   • The (overall) function represented by a DNN is a complex nonlinear mapping from input space to output space, building up through successive layers.

   • The successive layers of nonlinear functions (with parameters) aim to approximate the highly complex relationships between the input data and the output.

   • Here is a common schematic that illustrates the architecture of NNs, see, e.g., Neural Network (Wikipedia).

   Universal Approximation:

   NNs with at least one hidden layer and appropriate nonlinear activation functions are universal approximators; they can approximate any continuous function on compact subsets of the input space to arbitrary precision given sufficient neurons.

   More layers usually mean stronger expressive power, allowing networks to approximate complex functions more efficiently than shallow networks with the same number of parameters.

   The ability to approximate composition of functions exponentially improves error decay rates compared to polynomial rates in shallow models. Universal approximation guarantees do not specify network size but support the theoretical foundation for using DNNs as powerful function approximators in diverse applications.

   See the last section for more details.

2. Sect. 5.2. Gradient Computation and Backpropagation, Optimization Methods, and Current Developments

   Gradient Computation and Backpropagation of Error (History):

   • Backpropagation (BP) is an efficient algorithm for computing gradients of the loss function with respect to network weights using the chain rule of calculus. This enables multi-layer networks to be trained.

   • The concept goes back to the 1960s. E.g., Seppo Linnainmaa discusses “reverse mode automatic differentiation” in his 1970 master’s thesis. See also Paul Werbos’ work in 1974, in which he proposed backpropagation applied to multilayer perceptrons.

   • BP gained prominence after the seminal 1986 paper by Rumelhart, Hinton, and Williams demonstrating practical effectiveness. The method enabled training of deep layered architectures by efficiently propagating error derivatives backward through layers, overcoming limitations of earlier models.

   • Yann LeCun used BP in convolutional neural networks (CNNs) during the late 1980s for digit recognition. This was a key milestone in demonstrating practical industrial use.

   Gradient Descent, variations, and options

   • Batch Gradient Descent (BGD): it involves computing the gradient of the loss function using the entire training dataset before updating the parameters once per iteration. While it provides a stable and accurate estimate of the gradient, the method is computationally expensive and slow for large datasets, making it impractical for modern DL models.

   • Stochastic Gradient Descent (SGD): it updates the weights incrementally using the gradient from a single or a small batch of training samples, enabling frequent updates. This method reduces computation per update and introduces noise in gradient estimates that can help escape local minima. It often accelerates convergence in practice. Variants include mini-batch SGD, which balances stability and speed by using small subsets of data, and momentum-based optimizers that further improve convergence.

   • Other Options: Adaptive optimizers like Adam, RMSprop, and AdaGrad automatically adjust learning rates per parameter, enhancing training stability and speed on complex networks. Techniques such as learning rate scheduling, warm restarts, and gradient clipping improve convergence and prevent issues like exploding or vanishing gradients. Research continues into second-order methods, meta-learning optimizers, and decentralized or federated training techniques to scale training further. Parallelization on GPUs and TPUs combined with these optimization advances drives the ongoing industrial and research revolution in DL applications.

3. Sect. 5.3. Loss Functions

   Loss Functions for Regression:

   • Mean Squared Error (MSE)/L2 Loss:
     $$\text{MSE} = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 .$$
     Here: where y_i is the true target and ŷ_i is the prediction for the i-th sample. This Measures the average of squared differences between predicted and true values. Sensitive to outliers.

   • Mean Absolute Error (MAE)/L1 Loss:
     $$\text{MAE} = \frac{1}{n} \sum_{i=1}^{n} |y_i - \hat{y}_i| .$$
     This measures average absolute differences; less sensitive to outliers than MSE. Provides robustness for data with outliers but with less smooth gradients.

   • Huber Loss: Define
     $$L_\delta(y_i - \hat{y}_i) = \begin{cases} \frac{1}{2} (y_i - \hat{y}_i)^2 & \text{if } |y_i - \hat{y}_i| \leq \delta \\ \delta |y_i - \hat{y}_i| - \frac{1}{2} \delta^2 & \text{if } |y_i - \hat{y}_i| > \delta \end{cases}$$
     and the Huber Loss function is given by $\frac{1}{n} \sum_i^n L_\delta(y_i - \hat{y}_i)$. This combines MSE and MAE by using quadratic penalty for small errors and linear for larger errors. Robust to outliers and differentiable, balancing benefits of MSE and MAE.

   • Log-Cosh Loss:
     $$\text{Log-Cosh Loss} = \frac{1}{n} \sum_{i=1}^n \log \left( \cosh(y_i - \hat{y}_i) \right) .$$
     Similar to MSE but smoother, less affected by outliers.

   Loss Functions for Classification:

   • Binary Cross-Entropy Loss (Log Loss): Measures difference between predicted probabilities and actual binary labels. Suitable for binary classification; encourages probabilistic outputs close to true labels.

   • Categorical Cross-Entropy Loss: Generalization of binary cross-entropy for multi-class classification. Measures discrepancy between predicted class probabilities and true one-hot labels.

   • Hinge Loss: Used mainly in SVMs for maximum-margin classification. Penalizes predictions on the wrong side of the margin; promotes large margin classification.

   • Focal Loss: Variant of cross-entropy that focuses training on hard, misclassified examples. Useful in imbalanced classification scenarios.

   Choice of loss function depends on task nature, noise/outliers in data, and model requirements for probabilistic interpretation or margin maximization.

4. Sect. 5.4. Activation Functions Used in DL

   Common Activation Functions with Pros and Cons:

   • ReLU (Rectified Linear Unit):

     Pros: Computationally efficient, mitigates vanishing gradient problem, enables sparse activation.

     Cons: Can suffer from “dying ReLU” where neurons output zero for all inputs, non-differentiable at zero.

   • Sigmoid:

     Pros: Maps input to (0,1), useful for binary classification output layers.

     Cons: Prone to vanishing gradients in deep networks, slower convergence.

   • Tanh (Hyperbolic Tangent):

     Pros: Output ranges from -1 to 1, zero-centered which helps optimization compared to sigmoid.

     Cons: Also suffers from vanishing gradient for large inputs.

   • Softmax:

     Pros: Converts network outputs into probability distributions; used in multi-class classification tasks.

     Cons: Computationally expensive for many classes, assumes mutually exclusive classes.

   • Leaky ReLU:

     Pros: Fixes dying ReLU problem by allowing small non-zero gradient for negative inputs.

     Cons: Adds hyperparameters; performance gains depend on tuning.

   • ELU (Exponential Linear Unit):

     Pros: Combines benefits of ReLU with smooth output for negative values; helps reduce bias shift.

     Cons: More computationally expensive than ReLU.

   • Swish:

     Pros: Self-gated, smooth nonlinearity that often outperforms ReLU on complex tasks.

     Cons: Slightly increased computational complexity.

   • GELU (Gaussian Error Linear Unit):

     Pros: Used in transformer models; offers smooth activation that improves performance in NLP tasks.

     Cons: More complex calculations than ReLU.

   Choosing activation functions depends on the network architecture, task (classification, regression), and practical trade-offs like training speed and gradient behavior.

5. Sect. 5.5. Convolutional Neural Networks (CNNs)

   CNN Architecture Overview:

   • CNNs consist of input, convolutional layers, activation functions (usually ReLU), pooling layers, fully connected layers, and output layers.

   • They specialize in processing grid-like data such as images by exploiting spatial relationships between pixels.

   • The convolutional layer applies a set of learnable filters (kernels) that slide over the input volume spatially. Each filter is a small matrix that convolves (performs a sliding dot product) with a local region of the input at each position, producing an activation map.

   • This process extracts feature maps—local patterns such as edges or textures—by activating when specific features are detected at different spatial positions.

   • Parameter sharing: this means the same filter weights apply across the entire input spatial dimension, enabling translation invariance.

   • Hyperparameters include the number of filters (depth), filter size (spatial extent), stride (step length of sliding), and padding (zero-padding around input edges).

   • Multiple convolution layers stacked deepen feature extraction, starting from simple edges in early layers to complex shapes and objects in deeper layers.

   • These models can be designed to exploits spatial hierarchy by learning increasingly abstract features layer by layer.

   • Translation invariance helps recognize patterns regardless of location shifts.

   Some more details:

   • Convolution Operation: The core of CNNs is 2D discrete convolution, where a kernel K of size m × n slides over input image I to produce feature map O at position (i, j):
     $$O(i,j) = \sum_{u=0}^{m-1} \sum_{v=0}^{n-1} I(i+u, j+v) \cdot K(u,v) .$$
     This extracts local features like edges via weighted sums.

   • Pooling (Max): Downsamples feature maps by taking max in kernel window, e.g., 2 × 2 max pooling:
     O(i, j) = max_{u, v ∈ [0, 1]}I(2i + u, 2j + v),
     providing translation invariance and reducing parameters.

   • Fully Connected Layer: Flattened features x transform via z = Wx + b, where W is weights, b bias, followed by activation like ReLU: f(z) = max (0, z).

   • For a schematic illustration of a CNN filter: see Convolutional neural network (Wikipedia).

6. Sect. 5.6. Some Advanced Developments in DL

   Transformer models revolutionized deep learning since their introduction in the 2017 paper “Attention Is All You Need,” replacing sequential recurrent networks with parallelizable self-attention mechanisms that capture long-range dependencies efficiently. This shift enabled scalable training on massive datasets, powering breakthroughs in natural language processing (NPL), computer vision, and beyond, leading to LLMs such as GPT.

   See the additional notes in the next section for details.

   Leading proprietary models (as of late 2025)

   • OpenAI GPT-5, with specific versions like GPT-5 Pro and GPT-5.1 Thinking/Instant. They offer high-end reasoning, coding, and multimodal capabilities, making it superior for complex tasks, long-context understanding, and agentic functions compared to its predecessors.

   • Anthropic’s most advanced Claude models are the Claude 4.5 family, with Claude Opus 4.5 being their most intelligent, powerful model for complex coding, agents, and reasoning, while Claude Sonnet 4.5 offers the best balance of speed, cost, and intelligence for most production use cases, and Claude Haiku 4.5 is the fastest for instant responses. Note: “Haiku” signifies speed and efficiency, drawing parallels to the concise, impactful nature of Japanese poetry.

   • Google’s Gemini model is Gemini 3 Pro. An enhanced reasoning mode called Gemini 3 Deep Think is available to premium subscribers.

   • The most advanced xAI Grok models available are from the Grok 4 family (the base Grok 4, the specialized Grok 4 Heavy, and the efficient Grok 4 Fast). A minor update, Grok 4.1, was released in November 2025 with further refinements.

   Leading Open-Source Models:

   • DeepSeek-V3.2 series and the specialized DeepSeek-V3.2-Speciale model (for “long thinking” and complex algorithmic reasoning tasks). Both use a Mixture-of-Experts (MoE) architecture with a DeepSeek Sparse Attention (DSA) mechanism.

   • Meta’s Llama 4 Scout and Llama 4 Maverick.

   • Mistral AI’s Mistral Large 3, part of the new Mistral 3 family of models.

   • We note that Mamba is a new, efficient AI architecture that challenges traditional Transformers (like those in GPT/Gemini) by replacing expensive self-attention with State Space Models (SSMs) for linear scaling with sequence length.

1. Cybenko’s Universal Approximation Theorem: See the next section for more details.

2. A quick overview on transformer models with attention mechanism

   Transformer models are the backbone of almost all LLMs (like GPT). They start with converting input words into dense vector representations called embedding.

   These vectors, combined with positional encodings for tracking word order, are signals that flow through a stack of transformer blocks (each with its own unique set of learned parameters). These blocks progressive build richer contextual understanding.

   After passing through multiple such blocks (e.g., between 10+ to 90+ layers), the final output representation/vectors feeds into a softmax to predict, e.g., probabilities for the next word.

   Below is a simplified view of this process. See

   Vaswani, A. et al. (2017). Attention is all you need. In Advances in neural information processing systems (pp. 5998–6008).

   • Input Embeddings + Positional Encoding: Token embeddings X summed with positional encodings P to retain order: $$ Z_0 = X + P, $$
     where X ∈ ℝ^n × d, P ∈ ℝ^n × d, where n = sequence length, d = embedding/model dimension.

     One possible formulation for P is given by
     $$\label{05z=06} P_{i, k} = \begin{cases} \sin\left(\frac{i}{10000^{k/d }}\right) & \text{if } k \text{ is even} \\ \cos\left(\frac{i}{10000^{(k-1)/d }}\right) & \text{if } k \text{ is odd} \end{cases}$$
     Here, i denotes the position in the input sequence.

   • Multi-Head Self-Attention: Attention is applied across parallel heads, with output concatenated and projected. For a single-head attention:
     $$\label{05z=07} \text{Attention}(Q,K,V) = \text{softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)V,$$
     where $$ Q = Z W^Q, \quad K = Z W^K, \quad V = Z W^V. $$
     Here, Q (Query), K (Key), and V (Value) are matrices created by linearly transforming the input sequence data to determine relevance and extract weighted information.¹ Note: all the W^* matrices are learned parameters of the model.

     Instead of running just one attention, LLMs use multi-head attention. It splits the process into several parallel “heads,” each with its own unique focus (and learned parameters). The goal is to create a richer and more comprehensive understanding of the input data.

     Let head_i = Attention(Q_i, K_i, V_i). Then $$ \text{MultiHead}(Z):=
       \text{Concat}(\text{head}_1, ..., \text{head}_h)W^O. $$
     Here, Concat means stacking their resulting matrices side-by-side (a simple horizontal stacking). The matrix W^O (W-Output): it is a learned matrix for dimensionality management (projecting the high dim. concatenated output into the expected dimension required for further processing).²

   • Add & Norm (Post-Attention): Residual connection with layer normalization: $$ Z_1 = \text{LayerNorm}(Z_0 + \text{MultiHead}(Z_0)). $$
     What for?

     • LayerNorm

       For input vector x = [x₁, x₂, …, x_H]:
       $$y_i = \gamma \cdot \frac{x_i - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta$$
       where

       • \( \mu = \frac{1}{H} \sum_{i=1}^H x_i \) (mean across features)

       • \( \sigma^2 = \frac{1}{H} \sum_{i=1}^H (x_i - \mu)^2 \) (variance across features)

       • ϵ > 0 (small constant, e.g., 10^− 5, for numerical stability)

       • γ, β ∈ ℝ^H (learnable scale and shift parameters per feature)

       In matrix form for sequence embeddings X ∈ ℝ^n × d, LayerNorm applies element-wise after per-token statistics across d-dimensional features.

     • Residual connections. Introduced in ResNet and adopted in transformer models, one adds the input x directly to a layer’s output F(x), yielding H(x) = F(x) + x. This proves to be very helpful in training NNs in general:

       • Enable direct backpropagation of gradients through skip paths, mitigating vanishing gradients in deep networks (e.g., 100+ layers).

       • Allow layers to learn identity mappings (F(x) ≈ 0) when extra depth adds no value. This avoids performance drops as networks deepen—plain deep nets degrade.

   • Feed-Forward Network (FFN): Position-wise application per token: $$ \text{FFN}(Z_1) = \sigma(Z_1 W_1 + b_1) W_2 + b_2, $$
     where σ is ReLU or GELU activation.

   • Add & Norm (Post-FFN): Residual connection and normalization (again): $$ Z_2 = \text{LayerNorm}(Z_1 + \text{FFN}(Z_1)). $$
     Add & Norm (i.e., residual connections and LayerNorm) are used together again. This prevents exploding/vanishing signals through deep stacks, providing numerical stability.

   • Output to Next Block: Processed Z₂ is passed as input to the next transformer block.

3. Tips for running/deploying DL models/NNs at scale.

   • Emphasize Robust Infrastructure and Practical Simplicity

     Before scaling complex architectures, invest deeply in a robust, elastic ML infrastructure capable of handling distributed training, large datasets, and serving at scale. Optimize the end-to-end data and compute pipelines for reproducibility, efficiency, and fault tolerance. Even with cutting-edge architectures (e.g., Transformers, diffusion models), prefer simplicity when it meets product needs. In production, a smaller, well-regularized model with stable latency and controllable behavior often outperforms a massive but fragile alternative.

   • Domain Knowledge and Informed Model Design

     Integrate deep domain expertise throughout model development to guide architecture choices, input representations, and loss design. Feature engineering remains relevant—even for DL—through feature normalization, embedding design, and signal enrichment rooted in domain intuition. Mitigate training–serving skew by aligning preprocessing code, ensuring deterministic pipelines, and validating model behavior across real-time inputs.

   • Evaluation and Objective Alignment

     Connect every modeling objective to a tangible business or user outcome. Use composite evaluation that reflects long-term user satisfaction, not just accuracy, AUC, or perplexity. Where applicable, evaluate model calibration, robustness under distribution shift, and fairness metrics to understand deployment implications. Optimize for downstream task utility, not benchmark performance.

   • Data, Metadata, and Feature Management

     Adopt managed data pipelines and feature stores (e.g., Google services like Vertex AI Feature Store, etc.) for consistent feature generation and lineage tracking. Continuously compute and refresh key features using scheduled batch or streaming jobs to maintain temporal relevance. Maintain strong data versioning, schema evolution tracking, and metadata logging to support reproducibility and quick rollback during anomalies.

   • Automated Experimentation and Optimization Use distributed hyperparameter optimization frameworks (like Google’s Vizier) for efficient exploration of the architecture–learning rate–regularization space. Centralize experiment logging, visualization, and artifact tracking via professional-grade tools. Adopt early stopping, checkpoint management, and resource preemption policies to maximize research throughput without compute waste.

   • Model Serving, Monitoring, and Reliability Deploy models through containerized, autoscaling services with high availability and low prediction latency. Establish continuous monitoring for data drift, concept drift, and resource health. Integrate feature attribution tools (e.g., Integrated Gradients, SHAP) to interpret changes in model reasoning and detect anomalous inputs early. Set up shadow evaluation and canary releases to measure real-world performance before full rollout.

   • MLOps and Workflow Orchestration

     Build modular, pipeline-based workflows (e.g., TFX, Kubeflow, Dagster) for reproducible model training, evaluation, and deployment. Version control everything—data, model weights, architecture definitions, and configurations—to enable full lineage tracking and regulatory compliance. Automate retraining schedules, validation gates, and artifact promotion to streamline continuous delivery and rollback safety.

   • Collaboration and Cross-Disciplinary Transparency

     Foster a culture of openness and peer review where model designs, training anomalies, and empirical findings are shared across teams. Encourage visibility between modeling, data engineering. Coordinate teams for scalability and maintainability. Balance model sophistication, interpretability, and cost to serve, focusing on product-level value rather than academic novelty.

   For more info, see Google’s Deep Learning Tuning Playbook.

---

1. The names are motivated by classic information retrieval systems: we use a query to search for a relevant key within a database, and upon finding a match, the system returns the associated value (content).↩︎

2. How do researchers train the W matrices in multi-head attention so that different heads focus on specific aspects of data (e.g., verbs vs. nouns in text)?

   • Randomize the starting points: make sure the parameters in each head start at different initial values to create diversity.

   • Ensure different sets of matrices receive (slightly) different slices of the input feature space and a different portion of the backpropagated gradient. Their updates during optimization would move them in different directions (creating possibly different focuses).

   • In a recent paper (MoH: Multi-Head Attention as Mixture-of-Head Attention, by Jin et al.), authors reimagine multi-head attention as a Mixture-of-Experts (MoE) system. Attention heads act as “experts” and a learnable router (a NN) dynamically selects the most relevant heads per token.

   • The dropout trick. Randomly zeros out a fraction of the weights or activations during training. This forces the model to learn more robust, non-redundant representations and avoid co-adaption.

   • Head Pruning: Post-training removal of low-contribution heads (e.g., via magnitude or attention patterns).

   ↩︎

---