← AI Terminology
Neural Network
A neural network is a computational model loosely inspired by biological neurons — organised as layers of interconnected nodes (neurons) with learnable weights, where each neuron applies a weighted sum followed by a nonlinear activation function, enabling the network to learn complex patterns from data.
Neural networks are the foundational architecture of all modern deep learning.
Neural networks are the foundational architecture of all modern deep learning.
Why It Matters in AI
Neural networks are universal function approximators: given enough depth and width, they can learn any mapping from input to output from data alone — no hand-engineered features required. This property enabled the deep learning revolution: replacing handcrafted computer vision pipelines with CNNs (2012), rule-based NLP with transformers (2017), and expert systems with LLMs (2020+). Every modern AI system — from face recognition to ChatGPT — is built on neural network architectures.
Key Points
| Aspect | Description |
|---|---|
| Layers | Input, hidden (one or more), output — "deep" = multiple hidden layers |
| Neuron | Weighted sum of inputs + bias → nonlinear activation: output = f(w·x + b) |
| Learning | Backpropagation computes gradients; gradient descent updates weights to minimise loss |
| Activation | ReLU, GELU, Sigmoid, Tanh — introduces nonlinearity essential for learning complex patterns |
| Architectures | MLP (fully connected), CNN (convolutional), RNN/LSTM (recurrent), Transformer (attention) |
| Universal approx. | Hornik 1989: a single hidden layer with enough neurons approximates any continuous function |
Simple Analogy
A chain of voting committees: the first committee (input layer) looks at raw features and votes on patterns; each subsequent committee (hidden layer) combines and refines those votes; the final committee (output layer) makes the decision. Each committee member learns how to weight the inputs from training data — initially random, progressively better.
Common Usage Examples
nn.Sequential(nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 10))— simple PyTorch networkmodel.forward(x)ormodel(x)— forward pass through the network- Training:
loss.backward(); optimizer.step()— backprop + weight update - Keras:
model = Sequential([Dense(128, activation='relu'), Dense(10, activation='softmax')]) - TensorBoard: visualise layer activations, weight distributions, and loss curves during training
Summary
In short: A neural network is a multi-layer computational graph of weighted, nonlinear operations — the universal learning architecture that learns any input-output mapping from data, forming the foundation of all modern AI.