Semantic segmentation is the task of assigning a category label to every single pixel in an image. Where object detection is a coarse, bounding-box-level localization task, segmentation performs fine-grained, pixel-level region division. As of 2026, two currents run in parallel: refinement via transformer-based methods, and a shift toward foundation models that can carve out an arbitrary target from a prompt.

New to the fundamentals? See the "Object Detection & Semantic Segmentation Primer" first.

Meta AI (developer of Segment Anything)Meta AI

Image: Meta AI logo, Wikimedia Commons

Evaluation Metric: mIoU

Segmentation accuracy is conventionally evaluated with mIoU (mean Intersection over Union), which extends object detection's IoU to a per-class basis. For a given class c, the IoU is computed from the overlap between the ground-truth region G_c and the predicted region P_c, then averaged across all C classes.

\text{mIoU} = \frac{1}{|C|} \sum_{c \in C} \frac{|G_c \cap P_c|}{|G_c \cup P_c|}

When training this as a per-pixel classification problem, the loss function commonly used is per-pixel cross-entropy, averaged and summed across all pixels.

\mathcal{L} = -\frac{1}{N} \sum_{n=1}^{N} \sum_{c \in C} y_{n,c} \log \hat{y}_{n,c}

Here y_{n,c} is the ground-truth label — 1 if pixel n actually belongs to class c, 0 otherwise — and \hat{y}_{n,c} is the model's predicted probability that pixel n belongs to class c.

The Endpoint of Transformer-Based Methods: SegFormer and Mask2Former

What became mainstream in place of CNN-based methods (FCN, U-Net, and the like) is methods built around a transformer. SegFormer combines a CNN's hierarchical structure with a transformer's global modeling ability, using a hierarchical transformer encoder paired with a lightweight MLP decoder. It's regarded as a reliable, high-accuracy method across both fisheye and standard imagery.

Mask2Former goes a step further, formulating segmentation as a query-based mask-classification problem. It pairs a pixel decoder that preserves high-resolution spatial information with a transformer decoder that learns a fixed number of queries, where each query attends to a relevant region to predict a mask and its category. Built on a Swin-L backbone, it achieves state-of-the-art performance across all three segmentation types — panoptic, instance, and semantic — and the introduction of masked attention gets it converging 8x faster than Mask R-CNN.

Why Mask2Former's "Masked Attention" Is Fast

The technical core behind Mask2Former pairing fast convergence with high accuracy is masked attention: where a standard transformer decoder computes cross-attention against the entire image, Mask2Former restricts each query's attention to only the foreground region of the mask predicted at the previous layer. Rather than re-examining the whole image every time, narrowing down to only the "locally relevant region" implied by the immediately preceding prediction cuts wasted computation while extracting local features more accurately.

The overall architecture consists of a backbone feature extractor, a pixel decoder that preserves high-resolution spatial information, and a 9-layer transformer decoder (holding 100 learnable queries). To handle small objects, it also uses a multi-scale strategy, feeding the feature pyramid the pixel decoder produces (three resolution levels — 1/8, 1/16, 1/32) into successive transformer-decoder layers in round-robin fashion. This design achieves strong performance across all three segmentation types — panoptic, instance, and semantic — and converges 8x faster than Mask R-CNN.

The Prompt-Based Shift: The Segment Anything Lineage

Another major current is "prompt-based" segmentation, unconstrained by any predetermined fixed category list. The Segment Anything Model (SAM), released by Meta in 2023, takes a variety of prompts as input — a point, a bounding box, a rough mask — and generates a high-quality segmentation mask from them. This shift — carving out an arbitrary target based on a prompt, rather than predicting from a fixed class list — gave a major push toward the foundation-modeling of computer vision.

Its successor, SAM 2, replaced the single-scale ViT with Hiera, a multi-scale hierarchical Vision Transformer pretrained via masked autoencoding, and added support for video segmentation. It runs at 130 FPS across 37 zero-shot datasets while holding high accuracy (mIoU 58.9/81.7).

2026 then brought SAM 3 from Meta, which can detect, segment, and track a "visual concept" specified by a text prompt or an example image. It's the next step along the same direction — prompt-based, general-purpose segmentation — pushed one level further, up to specifying a concept in text.

SAM 3 Reaching Real-World Deployment: Results from Video Segmentation Competitions

Since the start of 2026, applied research built on SAM 3 has been appearing rapidly across a range of fields. SAM 3's defining trait — specifying a concept via text prompt — has repeatedly been carried over into specialized domains: maritime surveillance (MariSat, a maritime dataset that builds SAM 3 into a semi-automatic annotation pipeline) and communication-tower inspection (carving components out of cluttered UAV aerial imagery), among others.

Results from the MOSEv2 track of the 8th LSVOS Challenge, held at ECCV 2026, offer a useful gauge of how far video segmentation has come. SAM3Dual, a training-free method built on SAM 3 that combines two temporal memory branches — one for recent frames, one for historical context — placed third with a J&F score of 64.37. Competitive Memory Readout, which took second place at the same competition, explicitly folds in evidence about same-class "competitor" objects when retrieving target information from memory, achieving a primary metric score of 66.20. Both results underscore that preserving object identity across time in video (continuing to track the same object as the same target throughout a sequence) remains the core challenge determining how well SAM-family models actually perform.

Concrete Examples of Lightweighting and Few-Shot Adaptation

Efforts to push SegFormer/Mask2Former-family efficiency further are also continuing. DPNeXt (July 2026), a lightweight decoder for ViT-based dense prediction (segmentation plus multi-task work like depth estimation), cuts trainable parameters by 78.6% versus a standard DPT (Dense Prediction Transformer) while holding state-of-the-art-level performance on both the Cityscapes and NYUv2 benchmarks. TraceCLIP (July 2026), a fully training-free method that recovers local semantic information by isolating each patch's individual contribution to CLIP's CLS-token attention, reports mIoU improvements of 1.3 to 4.5 points over the strongest prior methods across eight zero-shot semantic segmentation benchmarks.

For a concrete example of few-shot adaptation, there's HASTE (July 2026), a platform for rapidly assessing post-disaster building damage from satellite imagery. By leveraging foundation-model embeddings, it matches the accuracy of a fully-supervised ResNet-50 baseline (one trained on labels for the entire dataset) while using only a twentieth as many labels, and it has supported more than 30 real-world disaster responses — earthquakes, hurricanes, wildfires — since 2023. It's a concrete embodiment of what segmentation is aiming for in the foundation-model era: production-grade accuracy at a fraction of the labeling cost.

When to Use Which Current

The practical breakdown at this point is: for a fixed task where the categories are already known ahead of time (cars, pedestrians, signs in a road scene, say), transformer-based methods in the SegFormer/Mask2Former family have the edge on both accuracy and efficiency; for situations that call for handling unknown targets or few-shot cases, SAM-family prompt-based foundation models show their strength — and that division of labor keeps solidifying.

References

#Semantic Segmentation #Segment Anything