← AI Terminology
RNN - Recurrent Neural Network
A recurrent neural network is a neural network architecture designed for sequential data — processing inputs one step at a time and maintaining a hidden state that carries information across time steps, enabling the network to model temporal dependencies.
It was the dominant sequence model before transformers (2017) and remains used for streaming and on-device applications.
It was the dominant sequence model before transformers (2017) and remains used for streaming and on-device applications.
Why It Matters in AI
Before transformers, all sequence modelling — speech, language, time series — was done with RNNs. The recurrent connection allows the network to "remember" previous inputs: the hidden state is a compressed summary of all prior context. RNNs enabled early breakthroughs in machine translation (sequence-to-sequence with attention), speech recognition (DeepSpeech), and handwriting generation (LSTM). While transformers have largely superseded RNNs for NLP, RNNs are still used in streaming applications where the full sequence is not available simultaneously.
Key Points
| Aspect | Description |
|---|---|
| LSTM/GRU | Gated variants that solve vanishing gradient — hidden state replaced by gated cell state |
| Hidden state | hₜ = f(Wₓ xₜ + Wₕ hₜ₋₁ + b) — state at time t depends on input and previous state |
| RWKV / Mamba | Modern linear RNN architectures combining RNN efficiency with transformer-level performance |
| Bidirectional | Process sequence forward and backward; concatenate — BiLSTM for NER, POS tagging |
| Sequential compute | RNNs process tokens one at a time — cannot parallelise over sequence length (unlike transformers) |
| Vanishing gradient | Gradients shrink exponentially over long sequences — RNNs fail to learn long-range dependencies |
Simple Analogy
Reading a book one word at a time and keeping a mental note of everything important so far: each new word updates your notes (hidden state) based on the word and your current notes. The limitation is that very early words get buried as the notes keep being updated — the vanishing gradient problem.
Common Usage Examples
nn.RNN(input_size=128, hidden_size=256, batch_first=True)— basic PyTorch RNNnn.LSTM(128, 256, num_layers=2, bidirectional=True)— BiLSTM for sequence labelling- Sequence-to-sequence:
encoder_rnn → decoder_rnn + attention— early MT architecture - Time series: RNN reads historical stock prices → predicts next-day price
- RWKV:
pip install rwkv— transformer-quality LLM with RNN inference (constant memory per step)
Summary
In short: RNNs process sequential data by maintaining a hidden state across time steps — the dominant sequence architecture before transformers, still used for streaming applications and on-device scenarios where constant-memory inference is essential.