← AI Terminology
Feedforward Network
A feedforward network (also called a multilayer perceptron or fully connected network) is a neural network where information flows in one direction — from input layer through hidden layers to output — with no cycles or feedback connections.
It is the simplest and foundational neural network architecture.
It is the simplest and foundational neural network architecture.
Why It Matters in AI
Feedforward networks are the building block that all modern architectures extend. Transformer FFN (feed-forward) blocks are two-layer feedforward networks applied after each attention layer. Fully connected classification heads are feedforward networks. Understanding feedforward networks — how they represent functions, where expressiveness comes from, why depth helps — is foundational for understanding any architecture in deep learning.
Key Points
| Aspect | Description |
|---|---|
| Layers | Input → hidden (×N) → output; each layer applies linear transformation + activation function |
| Parameters | Each pair of adjacent layers: input_dim × output_dim weights + output_dim biases |
| Distinction | Feedforward = no loops. Recurrent = loops (LSTM, RNN). Convolutional = weight sharing. |
| Depth benefit | Depth is exponentially more efficient than width for representing many functions |
| In Transformers | Each Transformer block contains a 2-layer FFN: Linear → GELU → Linear — expands then contracts |
| Universal approx | A single hidden layer with enough neurons can approximate any continuous function (UAT) |
Simple Analogy
Water flowing through a pipe system: enters at the top (input), flows through a series of chambers (hidden layers) where it's processed and redirected (weights + activations), exits at the bottom (output). It only ever flows forward — no backflows, no circular pipes. Backpropagation flows backwards, but that's only for training, not inference.
Common Usage Examples
nn.Sequential(nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 10))— MNIST classifier- Transformer FFN:
x = x + FFN(LayerNorm(x))whereFFN = Linear(d, 4d) → GELU → Linear(4d, d) - Classification head: a feedforward network attached to the output of a pre-trained backbone
- MLP mixer: architecture that replaces attention entirely with feedforward mixing operations
sklearn.neural_network.MLPClassifier(hidden_layer_sizes=(256, 128))— scikit-learn FFN
Summary
In short: A feedforward network is the simplest neural architecture — data flows in, through hidden layers, and out — and it's the universal building block that every more complex architecture is built from or extends.