A Thorough Guide to SegFormer — Why a Transformer Without Positional Encoding Works for Segmentation

Attempts to introduce Transformers into semantic segmentation existed before SegFormer, but many depended on positional encodings and complicated decoders. They therefore faced two problems: accuracy could fall when the test resolution differed from the training resolution, and computation could become expensive. SegFormer (Xie, Wang, Yu, Anandkumar, Alvarez, and Luo, NeurIPS 2021) addresses both problems with a remarkably simple design: a hierarchical encoder without positional encoding and an All-MLP decoder that uses no convolution at all. This article examines, step by step, why that design works, based on the original paper (arXiv:2105.15203).

This article is based on the original paper “SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers” (Xie et al., NeurIPS 2021, arXiv:2105.15203).

0. What you will learn

• The problems that SegFormer set out to solve in earlier Transformer-based segmentation models
• How the hierarchical MiT (Mix Transformer) encoder works without positional encoding
• Why a decoder made only of MLPs, with no convolution at all, can still achieve high accuracy
• Measured mIoU, parameter counts, and computational cost for B0–B5 on Cityscapes and ADE20K
• How to choose a model in practice, including robustness to image corruption

1. What is SegFormer?

SegFormer is a semantic segmentation model that combines a hierarchical Transformer encoder (MiT) without positional encoding with a lightweight All-MLP decoder that uses no convolution at all. Its central idea is to redesign the division of labor between encoder and decoder: once the encoder has produced features at multiple resolutions, the decoder can be greatly simplified.

2. Why was this design necessary?

Using a Vision Transformer (ViT) directly as an encoder for semantic segmentation creates two problems. First, the features produced by ViT have only a single resolution. Objects can have many different sizes within one image, and conventional CNN-based segmentation models such as FCN and U-Net address this by combining features at multiple resolutions. A ViT that produces only one resolution loses this multi-scale representational capacity.

The second problem is dependence on positional encoding. ViT divides an image into patches and feeds them to a Transformer, but the Transformer itself has no information about where each patch is located in the image. A fixed or learned positional encoding must therefore be added separately. Because this encoding is built for the input resolution used during training, an image with a different resolution at inference time requires interpolation of the positional encoding, which can reduce accuracy. Segmentation often requires higher-resolution inputs than detection, making this resolution dependence particularly troublesome.

SegFormer addresses both problems at their root by using a hierarchical encoder and eliminating positional encoding altogether.

3. What is the input?

As with other segmentation models, the input is an RGB image x \in \mathbb{R}^{H \times W \times 3}. SegFormer is evaluated on high-resolution images (1024×2048) for Cityscapes and images around 512×512 for ADE20K. Whereas ViT downsamples with coarse 16×16 patches, SegFormer's encoder starts with patches as small as 4×4, preserving more detail at the beginning of feature extraction.

4. What does it predict?

The output is a segmentation map at 1/4 of the input resolution, with the class of each pixel represented as \hat{y} \in \mathbb{R}^{\frac{H}{4} \times \frac{W}{4} \times N_{cls}} (N_{cls} is the number of classes). It is then returned to the original resolution using nearest-neighbor interpolation or a similar method. The core idea is that if the encoder creates sufficiently rich multi-scale features, the decoder only needs a combination of simple linear layers to integrate them and convert them into an accurate mask.

5. Basic architecture

SegFormer consists of two major blocks: a MiT (Mix Transformer) encoder that extracts features at four resolutions, and an All-MLP decoder that combines those features and outputs a segmentation mask.

Figure 1 — The MiT encoder produces features at four resolutions (1/4, 1/8, 1/16, and 1/32), and the All-MLP decoder combines them using only linear layers to create a segmentation mask. The only convolution is in the encoder's Mix-FFN; the decoder is literally made entirely of linear layers.

Because the encoder builds features at four resolutions in advance, the decoder can obtain an accurate mask with almost minimal processing: align the features and combine them. This division of labor is the main reason SegFormer keeps its total parameter count low.

6. Technical details of the components

Overlapped Patch Merging — a four-stage hierarchy

Rather than dividing the image into patches only once as ViT does, the MiT encoder gradually reduces resolution across four stages. The output resolutions are 1/4, 1/8, 1/16, and 1/32 of the input, reproducing the hierarchical structure used by CNN backbones such as ResNet.

The patch partitioning method also differs from ViT's non-overlapping patches: adjacent patches overlap when they are merged. The first stage uses kernel size K=7, stride S=4, and padding P=3; later stages use K=3, S=2, P=1. The overlap preserves local continuity across patch boundaries—the information shared by neighboring patches—while the feature map is reduced.

Efficient Self-Attention

Ordinary multi-head self-attention requires O(N^2) computation for a sequence of length N. Since N becomes very large for high-resolution segmentation, this is a serious bottleneck. SegFormer introduces Sequence Reduction, which compresses the Key and Value sequences by a reduction ratio R before computing attention.

The K, V sequences are reduced from length N to N/R before they are used. The reduction ratio R for Stages 1–4 is set to [64, 16, 4, 1]: shallow stages have high resolution and long sequences, so they compress aggressively; deep stages have lower resolution and shorter sequences, so they do not compress at all. This reduces computational cost from O(N^2) to O(N^2/R) and makes self-attention practical even for high-resolution inputs.

Mix-FFN — how positional encoding is eliminated

SegFormer can avoid positional encoding altogether because it mixes a 3×3 convolution into the feed-forward network (FFN), creating Mix-FFN.

The 3×3 convolution uses zero padding at the image boundary. This padded boundary indirectly leaks a clue about “where in the image I am” into the convolution, so positional information is woven into the features simply by passing through Mix-FFN, without an explicit positional encoding. As a result, testing at a resolution different from the training resolution does not require interpolation of a positional encoding, which could otherwise reduce accuracy.

All-MLP decoder — why convolution is unnecessary

The decoder consists of the following four steps, literally using only linear layers (MLPs).

1. Unify channels: map each stage feature F_i to the same channel count C with an independent linear layer.
2. Upsample and concatenate: upsample every feature to 1/4 resolution and concatenate along the channel dimension.
3. Fuse: map the concatenated 4C-dimensional feature to C dimensions with a linear layer.
4. Predict: output a segmentation mask with one channel for each class using a final linear layer.

The original paper supports this design with an analysis of the effective receptive field (ERF). CNN-based decoders such as DeepLabv3+ need complicated mechanisms—dilated convolutions and multi-scale pooling—to obtain a broad receptive field and achieve high accuracy. In contrast, the MiT encoder naturally acquires local ERFs in shallow stages and non-local ERFs that cover the whole image in deep stages through self-attention, without additional mechanisms. Because the encoder itself already builds features containing information from local to global scales, the decoder does not need a complex receptive-field expansion mechanism. This is the theoretical reason a simple combination of linear layers is sufficient.

Training recipe

The original paper uses a simple training setup without special tricks. The MiT encoder is pretrained on ImageNet-1K, while the decoder is trained from random initialization. AdamW is used for optimization, with the learning rate decaying from an initial value of 0.00006 under a poly schedule (power-law decay with coefficient 1.0). The number of iterations is 160K on ADE20K and Cityscapes and 80K on COCO-Stuff. Training augmentation is limited to random resizing with a ratio of 0.5–2.0, random horizontal flipping, and random cropping (512×512 for ADE20K, 1024×1024 for Cityscapes, and 640×640 for B5 on ADE20K). The paper explicitly states that it does not use widely adopted tricks such as OHEM (Online Hard Example Mining), auxiliary losses, or class-balanced losses. The goal is to show that the MiT encoder and All-MLP decoder can reach high mIoU without those additions.

7. Comparing model variants

SegFormer is available in six sizes, from MiT-B0 to MiT-B5. The following are the measured Cityscapes and ADE20K results reported in Table 2 of the original paper.

Source: Table 2 of the original paper (Xie et al., NeurIPS 2021). Cityscapes mIoU is measured with multi-scale and left-right flip (MS) testing. Parameter counts include the full encoder and decoder; even for B5, the decoder is only about 4% of the total. For reference, B5 reaches 82.4 mIoU with single-scale (SS) testing and 84.0 with MS testing.

The paper compares these results with the state of the art at the time and reports that SegFormer-B4 reaches 50.3% mIoU on ADE20K with 64.1M parameters, exceeding the best method of that period by 2.2 points with a model nearly one-fifth the size. B0 maintains a practical Cityscapes mIoU of 76.2 with only 3.8M parameters, and the numbers support its edge-oriented design.

8. What it struggles with

Most of SegFormer's weaknesses are shared by Transformer-based models in general. First, larger models (B3 and above) have high training and inference costs. Cityscapes FLOPs grow from 125.5G for B0 to 1460.4G for B5, an increase of more than ten times. For high-resolution segmentation, this growth can make real-time operation difficult.

Dependence on large-scale pretraining data also matters. The MiT encoder is pretrained on ImageNet-1K, so the quality and quantity of fine-tuning data strongly affect transfer to domains that differ substantially from the pretraining data, such as medical or satellite imagery.

Very small objects and thin structures—for example, wires and signposts—are not fundamentally easy for SegFormer either. Overlapped Patch Merging downsamples by 4×4 even in the first stage, so structures only a few pixels wide can lose information before reaching deeper stages.

At the same time, the robustness evaluation on Cityscapes-C reported by the original paper is a clear strength. For corrupted images with added Gaussian noise, SegFormer-B5 records a maximum relative mIoU improvement of 588% over DeepLabv3+ with an Xception-71 backbone; for snow-like corruption, the maximum relative improvement is 295%. These measurements show greater tolerance to image corruption than ordinary CNN-based models. This robustness is thought to arise because self-attention is less easily dominated by local noise.

9. How to choose a model in practice

For edge devices and real-time processing, SegFormer-B0 or B1 is a realistic choice. B0 balances accuracy and compactness with 3.8M parameters and 76.2 mIoU on Cityscapes, making deployment on embedded devices and drones plausible.

For applications such as drivable-area recognition in autonomous driving, where high-resolution images and consistent accuracy are required, B2 or B3 is often a reasonable compromise. B4 and B5 have the highest accuracy, but their Cityscapes FLOPs exceed 1200G, so real-time inference on an in-vehicle GPU generally requires optimization such as quantization or TensorRT conversion.

For offline precision analysis of medical or satellite imagery, accuracy takes priority over real-time performance. B4 or B5 is appropriate, with fine-tuning on domain-specific data as needed.

For outdoor environments with frequent bad weather or lighting changes, such as agricultural robots and outdoor surveillance cameras, the robustness measured on Cityscapes-C is a practical advantage. Compared with CNN-based models of similar accuracy, SegFormer can be expected to lose less accuracy under image corruption.

To learn Object Detection and Semantic Segmentation from the fundamentals, see “An Introduction to Object Detection and Semantic Segmentation”. For recent trends across segmentation, see “Technology Trends in Semantic Segmentation”.

10. Three-line recap

• SegFormer combines a hierarchical encoder (MiT) that produces multi-scale features without positional encoding with an All-MLP decoder that uses no convolution.
• The 3×3 convolution in Mix-FFN implicitly leaks positional information, removing the need for positional encoding, while the Transformer's broad effective receptive field makes a simple MLP decoder possible.
• The family ranges from B0 (3.8M parameters, 76.2 mIoU on Cityscapes) to B5 (84.7M, 84.0 mIoU), and measured results also support its robustness to image corruption.

References

• SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers (NeurIPS 2021, arXiv:2105.15203)
• SegFormer paper on ar5iv (full text)
• Official SegFormer implementation (NVlabs/SegFormer, GitHub)
• NeurIPS 2021 Proceedings: SegFormer