1.1: Chapter 1 Summary

Building on the idea of understanding the brain through artificial neural networks (ANNs), here are more details about these concepts and architectures.

• Biological Inspiration and Motivation:

  • The goal: Understand and explain the brain’s (fascinating!) capabilities, particularly its ability to learn, adapt, and survive in dynamic environments.

  • ANNs and related systems are designed for parallel, adaptive information processing. Their designs are inspired by biological neurons and networks.

  • This approach helps to model biological systems at a basic level and provides insights into the principles underlying biological intelligence.

• Spiking Neurons and Synaptic Interactions:

  • The physiology of biological systems is highly complex, even at the single neuron level.

  • Electrochemical processes are fundamental to how neurons communicate. Neurons maintain an electrochemical potential difference across their cell membrane.

  • An action potential, also known as a spike, is generated when the membrane potential reaches a specific threshold. This is a brief, localized electrical pulse that propagates along the axon from the cell body (soma).

  • When a spike reaches the end of the axon, it triggers the release of neurotransmitters into the synaptic cleft. These chemical messengers then bind to receptors on the post-synaptic neuron.

  • This binding leads to a change in the post-synaptic neuron’s membrane potential (the post-synaptic potential), which can be either excitatory (increasing the potential and likelihood of spiking) or inhibitory (decreasing it).

  • Synaptic interactions are generally unidirectional, meaning information flows from the pre-synaptic to the post-synaptic neuron. There is also a small but significant synaptic delay (around 0.5 ms) due to the electrochemical processes.

  • Models like the Hodgkin-Huxley model use complex differential equations to describe these dynamics, while simpler models like the Integrate-and-Fire (IaF) model capture essential features for studying large networks.

• Firing Rate Models and Activation Functions:

  • To simplify the complexity of individual spikes, firing rate models represent neural activity as the average number of spikes per unit time.

  • These models assume that the crucial information for neural processing is contained within the mean activity and frequency of spikes.

  • The activity of a neuron i (denoted S_i) is determined by the weighted sum of inputs from other neurons j (S_j) passed through an activation function h(x):
    $$ S = h \left( \sum_j w_{ij} S_j\right). $$

  • Synaptic weights (w_ij) represent the strength of the connection from neuron j to neuron i. A positive w_ij indicates an excitatory connection, while a negative w_ij indicates an inhibitory connection.

  • Sigmoidal activation functions (h(x) or g(x)) are commonly used because they capture key biological characteristics.

  • The McCulloch-Pitts model is an early, simplified model of neurons.

• Hebbian Learning:

  • Beyond fixed functionalities, biological neural networks can learn and adapt to their environment, a process known as synaptic plasticity.

  • Hebbian Learning is a fundamental hypothesis for this learning, famously summarized as “neurons that fire together wire together”.

  • Mathematically, the change in synaptic weight (Δw_AB) is proportional to the product of the pre- and post-synaptic activities (x_A and s_B): $$ \Delta w_{AB} \propto x_A S_B$$

  • This implies that if both neurons are active, the synapse strengthens (excitatory increase), and if their activities are decorrelated, the synapse may weaken or become inhibitory.

  • Hebbian learning forms the basis for many learning algorithms in ANNs.

• This chapter also discusses models like the Hopfield Model and several types of ANNs. See the following subsections for details.

---