← AI Terminology
LSTM - Long Short-Term Memory
LSTM (Long Short-Term Memory) is a type of recurrent neural network (RNN) architecture that uses gated mechanisms — forget gate, input gate, output gate — to selectively retain or discard information over long sequences, overcoming the vanishing gradient problem of vanilla RNNs.
Introduced by Hochreiter & Schmidhuber in 1997; dominated sequence modelling until transformers (2017).
Introduced by Hochreiter & Schmidhuber in 1997; dominated sequence modelling until transformers (2017).
Why It Matters in AI
Standard RNNs fail to propagate information across long sequences because gradients vanish during backpropagation through time. LSTMs introduced the cell state — a "conveyor belt" that carries information with minimal transformation, controlled by multiplicative gates. This enabled sequence tasks previously impossible: machine translation, speech recognition, time series forecasting. While transformers have largely superseded LSTMs for NLP, LSTMs remain widely used for time series, streaming data, and on-device applications where transformers are too heavy.
Key Points
| Aspect | Description |
|---|---|
| Cell state | Long-term memory: a vector that runs through the sequence with only additive/gating updates |
| Input gate | Controls what new information is written to the cell state each time step |
| Forget gate | Sigmoid layer: decides what fraction of the cell state to erase (0 = forget, 1 = keep) |
| Output gate | Controls what portion of the cell state is exposed as the hidden state (short-term memory) |
| Bidirectional | BiLSTM: two LSTMs reading forward and backward — used in NER, POS tagging |
| vs Transformer | Transformers are better at long-range dependencies and parallelise; LSTMs are better on-device |
Simple Analogy
A person taking notes in a meeting with a sticky note (cell state) and a whiteboard (hidden state): the forget gate decides which sticky notes to peel off, the input gate decides what to add, and the output gate decides which notes to read aloud. Key information persists on sticky notes even if not mentioned for many turns.
Common Usage Examples
nn.LSTM(input_size=128, hidden_size=256, num_layers=2, batch_first=True)— PyTorch LSTM- Time series forecasting: LSTM reads historical prices → predicts next N steps
- Speech recognition: BiLSTM encoder in early end-to-end ASR systems (DeepSpeech 2)
keras.layers.LSTM(128, return_sequences=True)— returns output at every time step- Hybrid: LSTM combined with attention mechanism —
LSTMWithAttentionfor document classification
Summary
In short: LSTMs solved the vanishing gradient problem in sequence modelling via gated cell states — dominating NLP and time series for a decade, and still the preferred choice for on-device or streaming sequence tasks where transformers are too expensive.