← AI Terminology
Parameter
A parameter in machine learning is a learnable numerical value within a model — primarily weights and biases — that is adjusted during training via gradient descent to minimise the loss function.
Model size is most commonly described by parameter count: "a 7B parameter model" has 7 billion such values.
Model size is most commonly described by parameter count: "a 7B parameter model" has 7 billion such values.
Why It Matters in AI
Parameter count is the primary measure of model capacity and, to a significant extent, capability: scaling laws show that more parameters (with proportional data and compute) consistently produce better models. GPT-3 has 175B parameters; Llama 3 70B has 70B; a typical MLP classifier might have 100K. Parameters represent all the knowledge a model has learned — they are what gets saved in a checkpoint, what requires GPU memory to run, and what determines inference cost.
Key Points
| Aspect | Description |
|---|---|
| Biases | Scalar offsets added after the weighted sum — one per neuron |
| Memory | FP32: 4 bytes/param; FP16: 2 bytes/param — 7B params ≈ 14GB in FP16 |
| Weights | Matrix values that multiply inputs — the majority of parameters in any neural network |
| Hyperparameter | Different from parameters: hyperparameters (LR, batch size) are set by humans, not learned |
| Active vs total | MoE models: total params >> active params per token — e.g. 141B total / 22B active (Mixtral 8x22B) |
| Trainable vs frozen | Fine-tuning freezes base parameters; LoRA adds a small trainable set on top |
Simple Analogy
A musician's technique settings: the tuning of every string, the tension of every drumhead, the breath pressure habits — each a learned, adjustable value that shapes performance. Training is the practice that adjusts all these settings (parameters) to improve the output. The total number of adjustable settings is the model's "parameter count."
Common Usage Examples
sum(p.numel() for p in model.parameters())— count total parameters in a PyTorch modelsum(p.numel() for p in model.parameters() if p.requires_grad)— count trainable parameters onlymodel.state_dict()— dictionary of all parameter tensors, keyed by layer name- LoRA fine-tuning: "training 0.1% of parameters" — ~7M of 7B frozen base parameters
torch.nn.utils.parameters_to_vector(model.parameters())— flatten all params to a single 1D tensor
Summary
In short: Parameters are the learnable numerical values inside a model — the billions of weights and biases adjusted during training that encode everything the model knows, and whose count determines both capacity and inference cost.