← AI Terminology
Semantic Segmentation
Semantic segmentation is a computer vision task that assigns a class label to every pixel in an image — painting each region with a category colour — without distinguishing between individual instances of the same class.
It provides a dense, pixel-level understanding of the scene.
It provides a dense, pixel-level understanding of the scene.
Why It Matters in AI
Bounding boxes are imprecise for scene understanding — they include background and overlapping objects. Semantic segmentation provides exact pixel-level classification: every pixel is labelled "road", "sky", "person", or "tree". This precision is required for autonomous driving (drive on road, not on pedestrian), medical imaging (outline tumour vs. healthy tissue), and satellite image analysis (map land use per pixel). ViT-based models (SegFormer, Mask2Former) have largely replaced traditional FCN approaches.
Key Points
| Aspect | Description |
|---|---|
| Evaluation | Mean IoU (mIoU) per class, averaged over all classes — standard metric on Cityscapes, ADE20K |
| Panoptic seg | Unified: semantic seg for background ("stuff") + instance seg for objects ("things") |
| Encoder-decoder | FCN, U-Net, SegFormer — encoder extracts features; decoder upsamples to original resolution |
| vs Instance seg | Semantic seg: all cars = "car"; Instance seg: car 1 gets unique mask, car 2 gets another |
| Transformer-based | SegFormer, Mask2Former, Segmenter — ViT-based encoders with segmentation heads |
| vs Object detection | Detection: bounding boxes; Semantic seg: pixel-level class labels for the whole image |
Simple Analogy
Colouring a colouring book where every pixel must be coloured with the correct category colour — green for grass, blue for sky, grey for road, red for cars. No outlines needed; just classify every pixel. Unlike instance segmentation, every car gets the same shade of red — categories matter, individual objects don't.
Common Usage Examples
- SegFormer:
from transformers import SegformerForSemanticSegmentation; model.from_pretrained("nvidia/segformer-b0-finetuned-cityscapes-512-1024") mmseg.apis.inference_segmentor(model, image)— MMSegmentation framework- U-Net for medical:
UNet(in_channels=1, out_channels=2)— binary tumour vs. background segmentation torchvision.models.segmentation.deeplabv3_resnet101(pretrained=True)— DeepLabV3 in PyTorch- Cityscapes: 19-class driving scene dataset — standard semantic segmentation benchmark
Summary
In short: Semantic segmentation classifies every pixel in an image with a category label — providing dense scene understanding for autonomous driving, medical imaging, and satellite analysis, where per-pixel precision matters more than object-level bounding boxes.