← AI Terminology
Causal Masking
Causal masking (look-ahead masking) prevents attention from seeing future tokens: position i may attend only to positions ≤ i.
It is what makes decoder-only transformers valid autoregressive language models.
It is what makes decoder-only transformers valid autoregressive language models.
Why It Matters in AI
Without a causal mask, the model could cheat by reading answer tokens during training. Causal masking enforces left-to-right generation and is why GPT-style models produce text one token at a time.
Key Points
| Aspect | Description |
|---|---|
| Shape | Lower-triangular mask (allow past, block future) |
| Related | Prefix-LM masks (bidirectional prefix + causal suffix) |
| Training | Enables parallel teacher-forced next-token training |
| Inference | Natural: only past tokens exist in the context |
| Implementation | Additive −inf mask before softmax, or flash-attn causal flag |
| Vs bidirectional | BERT uses full attention for encoding |
Simple Analogy
Writing a story where you may only look at what you have already written — never the blank pages ahead.
Common Usage Examples
is_causal=Trueinscaled_dot_product_attention- FlashAttention causal mode
- GPT training: triangular mask on the full sequence
- Decoder self-attention in encoder–decoder models
Summary
In short: Causal masking blocks future tokens in attention so the model predicts only from the past — the rule that makes autoregressive LLMs valid.