← AI Terminology
Weight
A weight in a neural network is a learnable scalar parameter that multiplies an input signal — the numerical values adjusted during training via gradient descent to encode the model's knowledge about the task.
Weights, along with biases, are the learned components of a model; everything else is architecture.
Weights, along with biases, are the learned components of a model; everything else is architecture.
Why It Matters in AI
A neural network's "intelligence" lives entirely in its weights — randomly initialised scalars that, after training on data, encode statistical patterns, world knowledge, and task-specific skills. Model size is measured in weights: GPT-3 has 175B, Llama 3 70B. Weights are what get saved in checkpoints, distributed for inference, quantised to reduce memory, and fine-tuned for new tasks. Understanding weights — their magnitude, distribution, and gradient flow — is essential for diagnosing training problems.
Key Points
| Aspect | Description |
|---|---|
| Norm | L2 weight norm ‖W‖ — regularised by weight decay to prevent overfitting |
| Update | w ← w − lr · ∂L/∂w — gradient descent rule; Adam uses adaptive per-weight learning rates |
| Sharing | Transformers reuse embedding weights in the output head (tied embeddings) — saves parameters |
| Weight matrix | Linear layer: y = Wx + b — W is the weight matrix, b is the bias vector |
| Frozen weights | Transfer learning: freeze early layer weights, only train task-specific head |
| Initialisation | Random init (Xavier, He) — scale matters to prevent vanishing/exploding gradients at start |
Simple Analogy
Volume knobs on a mixing board: each knob controls how much a particular input signal contributes to the output. Training is the sound engineer turning all 175 billion knobs simultaneously, guided by audience feedback (gradients), until the mix sounds right. The final knob positions are the model's weights — the distilled result of all that tuning.
Common Usage Examples
model.state_dict()— dictionary of all weight tensors, used to save/load checkpointstorch.save(model.state_dict(), "checkpoint.pt")/model.load_state_dict(torch.load(...))- Inspect:
sum(p.numel() for p in model.parameters())— total parameter count - Freeze:
for p in model.base_model.parameters(): p.requires_grad = False model.named_parameters()— iterate over (name, weight_tensor) pairs for inspection
Summary
In short: Weights are the learnable numbers that define what a neural network knows — initialised randomly, updated by gradient descent, and ultimately encoding all patterns the model has learned from data.