1.6: A Brief Discussion

NN/DL: The Formulation

• Neural networks (NNs) are a broad family of ML models inspired by the structure and function of the human brain.

• Deep learning (DL) is essentially NNs with many layers, hence the term “deep”.

Defining DL:

• A DL model with d hidden layers consists ofL

  • d weight matrices W₀, W₁, …, W_d − 1

  • A nonlinear activation function σ : ℝ → ℝ applied coordinate-wise. This means: given a input vector z = (z₁, z₂, ⋯), the output would be a vector given by (σ(z₁), σ(z₂), ⋯).

• Common activation functions used in deep learning include:

  • Rectified Linear Unit (ReLU) with bias:
    ReLU_b(x) = max (0, x − b)
    where b is the bias parameter.

  • Sigmoid (Logistic) function:
    $$\label{01e=02} \sigma(x) = \frac{1}{1 + e^{-x}}$$
    which compresses the input to the range (0, 1).

  • Hyperbolic tangent (tanh):
    $$\label{01e=03} \sigma(x) = \tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}$$
    compresses input to the range ( − 1, 1).

  • Leaky ReLU:
    $$\label{01e=04} \sigma(x) = \begin{cases} x - b, & x > b \\ \alpha (x - b), & x \leq b \end{cases}$$
    where α ∈ (0, 1) is a small slope for negative inputs, and b is bias. Design to address the “dying ReLU” problem, where neurons can become inactive during training due to consistently receiving negative inputs, leading to zero output and zero gradients.

  • Parametric ReLU (PReLU): Similar to Leaky ReLU, but the slope α for negative inputs is learned during training.

  • Exponential Linear Unit (ELU):
    $$\label{01e=05} \sigma(x) = \begin{cases} x, & x > 0 \\ \alpha (e^{x} - 1), & x \leq 0 \end{cases}$$
    where α > 0 is a hyperparameter controlling the value for negative inputs. Design to address some of the limitations of ReLU while maintaining its advantages.

  • Swish:
    $$\label{01e=06} \sigma(x) = x \cdot \text{sigmoid}(\beta x) = \frac{x}{1 + e^{-\beta x}}$$
    where β can be a constant or a trainable parameter. Google Brain reseachers designed it to improves the performance, compared to ReLU and sigmoid functions. See Ramachandran, P., Zoph, B., & Le, Q. V. (2017). “Searching for activation functions.” arXiv:1710.05941.

• Define the input vector as y₀ = x₀, then the forward pass for i = 0, …, d − 1 proceeds layer-wise as: $$  \textbf{y}_{i+1} = \sigma(W_i \textbf{y}_i ), $$
  where σ( ⋅ ) is applied coordinate-wise to the input vector. This can also be written as $$ \textbf{y}_d = 
    \sigma \left( W_{d-1}
    \sigma \left( W_{d-2}
    \cdots
    \sigma \left( W_{0}
    \textbf{y}_0
    \right)
    \right)
    \right).  $$   

• The dimensions of y_i and W_i are compatible for matrix-vector multiplication.

• Each coordinate of y_i corresponds to a node (neuron) in the network; each entry in W_i represents an edge (weight) connecting layers.

• The output of the network is given by y_d, so the DL model defines a mapping: $$ f_{W_1, \ldots, W_d} : \mathbb{R}^n \to \mathbb{R}^m, \quad \textbf{y}_d = 
      f(\textbf{x}_0). $$
  Note: The bias terms are absorbed to the weight matrices.

• The total number of parameters equals the sum of all entries in the matrices W_i.

• Training involves solving an optimization problem to find {W_i} minimizing a supervised learning loss function, often via empirical risk minimization:
  $$\label{01e=10} \min_{ \{ W_i \} } \frac{1}{N} \sum_{j=1}^N \mathcal{L}(y_d^{(j)}, t^{(j)})$$
  where t^(j) are target outputs, and ℒ is a loss function like mean squared error or cross-entropy.

• This optimization problem is typically nonconvex.

• Gradient-based methods like stochastic gradient descent or its variants are used for training, with efficient gradient computation via backpropagation. More on these in later chapters.

• Increasing depth often improves the expressiveness and compactness of representations. That said, shallow learning models and small deep learning (DL) models offer advantages in terms of computational efficiency, faster training times, and reduced resource requirements.

Major Architectures

• Convolutional Neural Networks (CNNs)

  • Exploit local connectivity in data (images, signals). Filters are “convolved” with the input data (e.g., an image): sliding small filters (or kernels) across the input, performing element-wise multiplication with the input, and summing the results to produce single output values.

  • Standard for image and spatial data domains

  • AlexNet (Krizhevsky et al., 2012) that popularized deep CNNs in ImageNet classification.

• Recurrent Neural Networks (RNNs)

  • NNs with temporal memory via recurrent connections

  • Natural fit for sequential/time series data modeling

  • Key references:

    Early RNN and LSTM foundational works (Hochreiter & Schmidhuber, 1997)

    Popular sequence modeling applications in NLP and time series (Graves, 2013)

• ResNet

  • It is a DL architecture that revolutionized image recognition by introducing skip connections to address the vanishing gradient problem in very deep NNs. This allows for training networks with hundreds or even thousands of layers, significantly improving performance in tasks like image classification and object detection.

  • Original paper:

    He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770–778.

• Transformers

  • Attention-based architecture for sequence data modeling and completely revolutionized DL/AI, widely used in NLP (e.g., language models). More to come on these amazing developments in later chapters.

  • Orginal papers:

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30, 5998–6008.

• Deep Neural Decision Forests

  • A novel approach that combines the strengths of both deep NNs and decision forests (like Random Forests). The core idea is to enrich classification trees with the power of deep networks to learn data representations while still benefitting from the interpretability and ensemble capabilities of decision forests.

  • It introduces stochastic and differentiable decision trees compatible with backpropagation, enabling joint optimization of split and leaf node parameters globally, unlike conventional decision trees that use greedy local training.

  • Representation Learning Guided by Trees. The decision trees guide the representation learning in the NN layers, reducing routing uncertainty at split nodes, which helps steer feature learning in lower layers of deep networks.

  • Original paper:

    Kontschieder, P., Fiterau, M., Criminisi, A., & Bulo, S. R. (2015). Deep neural decision forests. Proceedings of the IEEE International Conference on Computer Vision (ICCV), 1467–1475.

• Graph Neural Networks (GNNs)

  • Designed to process data represented as graphs, which consist of nodes and edges that represent relationships between data points.

  • GNNs are particularly useful for analyzing complex relationships and patterns within graph-structured data, where traditional neural networks struggle. They achieve this by iteratively updating node representations based on information from their neighbors, effectively capturing the dependencies and structure within the graph.

  • Used for social networks, molecule analysis, and other graph applications

    Key references:

    • Kipf, T., & Welling, M. (2017). Semi-supervised classification with graph convolutional networks. International Conference on Learning Representations (ICLR).

    • Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., & Monfardini, G. (2009). The graph neural network model. IEEE Transactions on Neural Networks, 20(1), 61–80. (early work of GNNs)

    • Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., & Dahl, G. E. (2017). Neural message passing for quantum chemistry. Proceedings of the 34th International Conference on Machine Learning (ICML), 1263–1272.

    • Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., & Bengio, Y. (2018). Graph attention networks. International Conference on Learning Representations (ICLR).

    • Hamilton, W., Ying, Z., & Leskovec, J. (2017). Inductive representation learning on large graphs. Advances in Neural Information Processing Systems (NeurIPS), 1024–1034.

We will discuss some of these models (as well as others not mentioned above) in later chapters.