1.2: A Brief History of Neural Networks and Deep Learning

We provide a brief history of neural networks (NN) and deep learning (DL) up to 2017, the year “Attention Is All You Need” was published (which marks the beginning of large language models (LLMs)). We will have sections on LLMs.

• 1943: The McCulloch-Pitts model

  First mathematical model of neural networks; foundational for the field.

• 1949: Hebb’s Rule

  Introduced Hebbian learning: “neurons that fire together wire together.”

• 1958: The Perceptron model by Rosenblatt

  First trainable artificial neural network (ANN) capable of binary classification.

• 1960: ADALINE/MADALINE (by Widrow & Hoff)

  ADALINE = Adaptive Linear Neuron or Adaptive Linear Element) and MADALINE = Multiple ADALINE. Neural networks using least mean squares learning; early adaptive systems.

• 1960s: Minsky & Papert’s book, Perceptrons

  Began studies on the limitations of perceptrons. 1969: Published Perceptrons (1st ed.), systematically proving the limitations of single-layer perceptrons — especially their inability to solve problems like XOR. A major factor leading to the first AI Winter. Note: An expanded edition was published in 1988 with a prologue and epilogue addressing criticisms and discussing the revival of neural nets. In 2017, the expanded edition was reprinted, including a new foreword discussing the book’s impact from the perspective of modern deep learning.

• 1960s–early 70s: Various developments

  Group Method of Data Handling by Alexey Ivakhnenko and V. G. Lapa (1968). Multilayer perceptron (early deep learning) trained via stochastic gradient descent by Amari (1967–68).

• 1980: Neocognitron (Fukushima)

  Early convolutional neural network for visual pattern recognition. Inspired by the work of Hubel and Wiesel (1950s and 60s), building on Fukushima’s earlier work (including the Cognitron). Recognized as the direct precursor to modern Convolutional Neural Networks (CNNs).

• 1982: Hopfield Network (Hopfield), or Amari-Hopfield Network

  Recurrent network using energy minimization for pattern storage and retrieval. Amari (1972) discovered a network equivalent to the Hopfield network. (Note: Amari is credited as the founder of information geometry.) Hopfield was awarded the 2024 Nobel Prize in Physics (shared with Geoffrey Hinton).

• 1980s: Recurrent Neural Networks (RNNs)

  Designed for processing sequential data where each output depends on previous inputs. Including the Jordan Network (1986) and Elman Network (1990).

• 1985–86: Restricted Boltzmann Machine (RBM; Hinton, Sejnowski, & Smolensky)

  Stochastic generative model with bipartite structure; important for unsupervised learning. Hinton and coworkers trained it with fast learning algorithms, such as Contrastive Divergence. In the mid-2000s, RBMs became the building blocks for Deep Belief Networks (DBNs) and other deep learning architectures.

• 1986: Backpropagation (Rumelhart, Hinton, Williams)

  Popularized practical training for multilayer perceptrons using backpropagation. Inspired by the work by Rosenblatt (1962: “back-propagating error correction”), Dreyfus (62, depending on Bryson and Kelley around the same time), Linnainmaa (1970), and Werbos (1974).

• 1989: LeNet (LeCun)

  Applied CNNs to handwritten digit recognition; deployed in real-world applications. The concepts were developed by LeCun and its team at AT&T in the late 80s. Refined in the early 90s, and LeNet-5 was released in 98 (successful for digit recognition). In the late 90s, it was used in industry (notably for reading handwritten checks and postal codes in banks and ATMs).

• 1997: Long Short-Term Memory (LSTM, by Hochreiter & Schmidhuber)

  LSTM (Long Short-Term Memory) Enabled learning long-term dependencies in sequences; a solution to vanishing gradients in RNNs. Each unit has a memory cell with controlled update mechanisms, and three gates (input, forget, and output gates), which are crucial for long-term information retention and gradient flow in deep networks.

  Note also the Gated Recurrent Units (GRUs). They are RNN-type networks, introduced in 2014 by Cho et. al., to efficiently process sequential data by using two main gates (the reset gate and the update gate), allowing the networks to selectively forget and retain information at each time step, effectively solving the vanishing gradient problem and enabling the learning of both short-term and long-term dependencies in sequences.

• 2011: DanNet (Ciresan et al.)

  First pure deep GPU-based CNN to win major international computer vision contests (IJCNN, ICDAR).

• 2012: AlexNet (Krizhevsky et al.)

  It won ImageNet with a large margin; triggers the DL revolution.

• 2014: Seq2Seq with Attention (Bahdanau et al.)

  Introduced the attention mechanism in sequence modeling; boosts translation and NLP tasks.

• 2015: ResNet (He et al.)

  It uses residual connections (skip connections) to enable the training of very deep networks (100+ layers) by mitigating the vanishing gradient problem.

• 2017: Transformer (Vaswani et al.)

  “Attention is All You Need”: Introduced Transformers, revolutionizing NLP. It marks the dawn of a new era in AI/ML. We will include sections on this and on subsequent developments inspired by it.

---