i2notes

Neural networks are the technology behind some of the most impressive achievements in modern artificial intelligence. They power the systems that can recognize a face in a crowd, translate speech in real time, generate photorealistic images from a text description, and hold a coherent conversation on almost any topic. If you have ever used a voice assistant, gotten a recommendation from a streaming service, or interacted with an AI chatbot, you have experienced the outputs of a neural network.

Yet for most people, neural networks remain something mysterious and technical — a black box that somehow produces impressive results without anyone being quite sure how. The reality is far more interesting and far more understandable than the mystery suggests. Neural networks are built on a simple and elegant idea, and grasping that idea transforms how you understand the AI technology that increasingly shapes everyday life.

The Inspiration: How the Human Brain Works

To understand neural networks, it helps to start with what inspired them. The human brain is the most sophisticated information processing system known to exist. It contains roughly 86 billion neurons — specialized cells that receive, process, and transmit information through electrical and chemical signals. Each neuron connects to thousands of others through structures called synapses. The strength of these connections — how readily a signal passes from one neuron to another — changes over time based on experience. This changing of connection strengths is the physical basis of learning and memory in biological brains.

When you learn to ride a bicycle, the experience strengthens certain neural connections and weakens others, gradually building a pattern of activity that allows you to balance, steer, and pedal simultaneously without conscious effort. When you recognize a friend’s face, a specific pattern of neural activity fires across a vast network of interconnected neurons in your visual cortex and beyond — a pattern that encodes your memory of that person’s appearance.

Artificial neural networks are inspired by this biological architecture. They do not replicate it with biological fidelity — artificial neurons are mathematical abstractions, not biological cells, and artificial neural networks do not work the same way the brain works in any deep sense. But they borrow the key structural idea: a network of interconnected processing units whose connection strengths can be adjusted through a learning process, enabling the network to learn from experience.

What an Artificial Neuron Actually Is

An artificial neuron is a very simple mathematical object. It takes one or more numerical inputs, multiplies each input by a corresponding weight (a number that represents the strength of that connection), adds all the weighted inputs together, adds a bias term (a number that shifts the result), and then passes the result through an activation function that determines the neuron’s output.

The weights are the parameters of the neuron — the numbers that are adjusted during training to make the neuron produce useful outputs. A high weight on a particular input means the neuron responds strongly to that input. A low or negative weight means the neuron largely ignores that input or is inhibited by it. By adjusting the weights, the learning process shapes how each neuron responds to its inputs.

The activation function introduces nonlinearity into the computation — the ability to represent relationships that are not simply proportional. Without nonlinear activation functions, a neural network of any depth would be mathematically equivalent to a single layer, no matter how many layers it contained. The activation function is what gives neural networks their expressive power — their ability to represent arbitrarily complex functions given sufficient size and depth.

A single artificial neuron is not very powerful. It can only represent a simple linear relationship between its inputs and output. But when many neurons are connected together in a network — when the output of some neurons becomes the input to others — the collective computation becomes dramatically more powerful.

The Architecture of a Neural Network

A neural network is organized into layers of neurons. The most basic architecture — called a feedforward neural network or multilayer perceptron — consists of three types of layers.

The input layer receives the raw data. If the network is processing an image, the input layer might contain one neuron for each pixel in the image, with each neuron receiving the numerical value of that pixel’s brightness or color. If the network is processing text, the input layer might receive a numerical representation of each word or character. The input layer does no computation — it simply passes the raw input data to the next layer.

The hidden layers sit between the input and output layers and do the actual computational work. Each neuron in a hidden layer receives inputs from all the neurons in the previous layer, computes a weighted sum, applies an activation function, and passes its output to all the neurons in the next layer. The word “hidden” refers to the fact that these layers are not directly connected to the outside world — they are internal to the network and their activity is not directly observed.

The output layer produces the network’s final result. For a classification task, the output layer might contain one neuron for each possible category, with each neuron’s output representing the network’s confidence that the input belongs to that category. For a generation task like a language model, the output might represent a probability distribution over all possible next words or tokens.

The depth of a neural network refers to the number of layers it contains. Shallow networks have one or two hidden layers. Deep networks have many hidden layers — sometimes dozens or even hundreds. The term “deep learning” refers specifically to the use of these deep, multi-layered neural networks. The depth is what enables the network to learn hierarchical representations of data, processing it from simple low-level features in the early layers to complex high-level abstractions in the later layers.

How Neural Networks Learn: Training

A freshly initialized neural network is useless. Its weights are set randomly, and its outputs are random guesses. Training is the process that transforms a random network into a capable one — by adjusting all the weights in the network so that the network produces accurate outputs for the training examples.

Training begins by passing a training example through the network from input to output — a process called a forward pass. The network produces an output, which is compared to the correct answer for that example. The difference between the network’s output and the correct answer is measured by a loss function, which produces a single number representing how wrong the network was. A high loss means the network was very wrong. A loss of zero would mean the network was perfectly correct.

The goal of training is to minimize the loss across all the training examples. To do this, the training algorithm needs to know how to adjust each weight in the network to reduce the loss. This is computed using a technique called backpropagation — short for backward propagation of errors. Backpropagation calculates, for each weight in the network, how much the loss would change if that weight were increased or decreased slightly. This information — called the gradient — tells the training algorithm which direction to adjust each weight to reduce the loss.

The training algorithm then adjusts all the weights by a small amount in the direction that reduces the loss — a process called gradient descent. The size of each adjustment is controlled by a parameter called the learning rate. If the learning rate is too large, the adjustments overshoot and the training becomes unstable. If it is too small, training proceeds very slowly. Finding the right learning rate is one of many practical challenges in training neural networks.

This process — forward pass, loss calculation, backpropagation, weight update — is repeated for every training example, typically many times through the entire training dataset. Each pass through the full dataset is called an epoch. Training a large neural network might require many epochs across billions of training examples, with billions of weight updates. The computational demands of this process are why training large neural networks requires specialized hardware and significant energy.

What Neural Networks Learn to Represent

One of the most remarkable things about neural networks is what they learn to represent internally — the features and patterns they extract from raw data through the training process.

In an image recognition network, researchers have found that the early layers typically learn to detect simple visual features — edges, color gradients, and textures. Middle layers detect more complex features — shapes, patterns, and combinations of edges. Later layers detect high-level features — faces, objects, scenes. These representations emerge automatically from the training process without anyone explicitly programming the network to detect edges or faces. The network discovers these features because they are useful for distinguishing between the categories in the training data.

In language models, neural networks learn to represent words and concepts as points in a high-dimensional numerical space — vectors called embeddings. Words with similar meanings end up close together in this space. Relationships between words are encoded as consistent directions in the space. The famous example is that the vector for “king” minus the vector for “man” plus the vector for “woman” ends up close to the vector for “queen” — the network has learned to encode the relationship between gender and royalty purely from the statistical patterns in text.

These internal representations are the source of much of the power of deep learning. By learning to represent complex data in ways that make the underlying structure explicit, neural networks can perform tasks that are impossible for simpler methods.

Different Types of Neural Networks for Different Problems

The basic feedforward architecture is just the starting point. Over decades of research, many specialized neural network architectures have been developed to handle specific types of data and tasks more effectively.

Convolutional neural networks (CNNs) are designed specifically for image data. They use a mathematical operation called convolution that efficiently detects local patterns in images while being invariant to their position — a network that can detect a cat in the center of an image can also detect it in the corner, without needing separate neurons for every possible position. CNNs are the architecture behind most computer vision applications, from smartphone cameras to medical imaging AI to self-driving vehicle perception systems.

Recurrent neural networks (RNNs) were designed for sequential data — data where the order of elements matters, like text, speech, and time series. RNNs process sequences one element at a time, maintaining a hidden state that carries information from previous elements to inform the processing of later ones. This allows them to model dependencies across time and sequence. While RNNs have largely been superseded by transformers for many language tasks, they remain useful for specific sequential modeling problems.

Transformer networks are the architecture behind most of the most capable AI systems of recent years, including large language models and many image generation systems. Transformers use a mechanism called attention to directly model relationships between any pair of elements in a sequence, regardless of their distance from each other. This ability to attend to long-range dependencies — to connect a pronoun at the end of a long paragraph to the noun it refers to at the beginning — is what makes transformers so effective for language tasks. The transformer architecture, introduced in 2017, triggered the wave of AI progress that produced the powerful language models in widespread use today.

The Limits of Neural Networks

Neural networks are powerful, but they have real limitations that are important to understand. They require enormous amounts of training data to perform well. They are computationally expensive to train. They can be difficult to interpret — understanding why a neural network made a particular decision is often not straightforward, which matters in applications where explainability is required.

Neural networks can also fail in unexpected ways on inputs that are different from their training data. A network trained on photographs taken in natural light may perform poorly on photographs taken in unusual lighting conditions. A language model trained primarily on text from one time period or culture may perform poorly on text from a different time period or culture. This brittleness outside the training distribution is a fundamental limitation of the learning-from-data approach.

Neural networks are also susceptible to adversarial examples — inputs that have been specifically crafted to fool the network. Small, carefully designed perturbations to an image that are invisible to human eyes can cause a neural network to completely misclassify it. This vulnerability has significant implications for the use of neural networks in security-sensitive applications.

Understanding these limitations is not a reason to dismiss neural networks. They are genuinely transformative technology. But using them wisely requires honest engagement with what they can and cannot do, and maintaining appropriate human oversight in applications where errors could have serious consequences.

Frequently Asked Questions

Is a neural network the same as a brain?

No. Artificial neural networks are loosely inspired by the structure of biological brains but do not replicate how brains actually work. Biological neurons are complex electrochemical systems with dynamics that artificial neurons do not model. The brain involves processes — neurotransmitter dynamics, hormonal modulation, continuous adaptation — that have no equivalent in artificial neural networks. The analogy to the brain is useful as a conceptual starting point but should not be taken to imply that artificial neural networks work the same way or have the same properties as biological brains.

How many neurons does a typical neural network have?

It varies enormously. Simple neural networks used for basic classification tasks might have hundreds or thousands of neurons. Large language models like those powering modern AI assistants are described in terms of parameters rather than neurons — they can have hundreds of billions or even trillions of parameters, spread across billions of artificial neurons arranged in hundreds of layers. The scale of these systems is genuinely staggering and is one reason why training them requires massive computing infrastructure.

Can a neural network learn anything?

In theory, neural networks are universal function approximators — given sufficient size and the right training, they can approximate any mathematical function to arbitrary precision. In practice, this theoretical result has important limitations. The network needs enough data, enough computing power, and a well-designed training process. There are also tasks — particularly those requiring genuine causal reasoning or common sense grounded in physical experience — that current neural network architectures struggle with despite their theoretical generality.

Why are deep neural networks better than shallow ones?

Deep networks can learn hierarchical representations of data — simple features in early layers that combine into complex features in later layers. This hierarchical processing is particularly well-suited to the structure of natural data like images and language, where meaning is built up from combinations of simpler elements. Shallow networks can theoretically represent the same functions as deep ones, but they typically require exponentially more neurons to do so. Depth provides a more efficient way to represent complex functions, which is why deep networks have proven so much more powerful in practice.

What happens inside a neural network that we cannot see?

This is one of the most important and challenging questions in AI research. The internal representations learned by neural networks — the patterns of activation across millions or billions of neurons — are difficult to interpret directly. Researchers have made progress in understanding what individual neurons and groups of neurons respond to, but a complete picture of how complex behaviors emerge from the collective activity of a large network remains elusive. This opacity is the central challenge of AI interpretability research — developing methods to understand not just what neural networks do but why they do it.