Count patch tokens
Calculate tokens for 224x224 images using 16x16 patches. Explain whether your count includes a class token.
Remember ViT
Full name: Vision Transformer
Memory aid: Vision is processed by a Transformer.
Patch construction, pretraining, and evaluation affect visual performance.
Study area: AI & machine learning · Batch 9
Relevant study for: Azure AI Fundamentals AI-901 · Azure AI Apps and Agents Developer AI-103 · Azure AI Fundamentals (retired) AI-900 · Azure AI Engineer (retired) AI-102
Practice with exam flashcards →Data / Knowledge library
Vision Transformer (ViT)
image patches processed with transformer attention
Why learn Vision Transformer (ViT)?
ViT embeds image patches as tokens, adds positional information, and processes them with Transformer blocks. Self-attention relates patches across the image. CNNs provide a useful comparison because their local filters encode different architectural assumptions. Neither approach is universally superior: compare accuracy, pretraining data, compute, and latency on the same task.
The problem it solves
Provides an attention-based way to learn image representations. Standard global self-attention models relationships between all patch tokens; its pairwise attention computation grows quadratically with sequence length. Smaller patches or larger images increase the token count.
Where it came from
Dosovitskiy and colleagues introduced ViT in an October 2020 preprint, subsequently published at ICLR 2021. The work applied a Transformer to sequences of image patches and demonstrated strong image-recognition results after large-scale pretraining. ViT is an architecture family, not a product with one current release.
Build your understanding
Work through these exercises in order. Try the task before opening a reference.
- Split a 224x224 image into non-overlapping 16x16 patches and count the tokens.
- Trace shapes from flattened RGB patches through a learned projection.
- Run a pretrained ViT using the preprocessing specified for its weights.
- Compare a CNN and ViT on the same held-out image split.
- Measure the accuracy, memory, and latency impact of changing image resolution.
More ways to practice
Trace patch embeddings
Calculate flattened RGB patch dimensions and the output shape after projection to an embedding width of 192.
Pretrained inference
Load a pretrained ViT, use its documented preprocessing, and record predictions on a small labeled image set.
Compare architectures
Compare a CNN and ViT with the same data split; disclose different pretraining datasets and compute budgets.
Resolution tradeoffs
Measure accuracy, peak memory, and latency at supported resolutions; document any positional-embedding adaptation.
Documentation and sources
- Original ViT paper (arxiv.org)
- Google Research: ICLR publication (research.google)
- Torchvision: ViT models (docs.pytorch.org)
- Keras: ViT image classifier (keras.io)
Catalog research notes
- Reference snapshot: September 9, 2026. Architecture history and implementation references, not a release changelog.
- Torchvision provides ViT builders with optional pretrained weights. Keras provides an image-classification implementation using patch projection and Transformer blocks.
Connect the ideas
Alternatives
Explore next
Real World Example
Automated Quality Control for Precision Manufacturing
High-precision electronics manufacturers requiring automated visual inspection of circuit boards.
The business problem
Traditional convolutional neural networks struggle with long-range dependencies across complex board layouts, leading to high false-rejection rates for subtle defects.
How this topic solves it
Vision Transformer (ViT) processes the image as a sequence of patches, allowing the self-attention mechanism to model global relationships between distant components on the board, which improves defect detection accuracy.
- Divide high-resolution circuit board images into fixed-size square patches.
- Flatten patches into a sequence and prepend a learnable classification token.
- Apply positional embeddings to preserve spatial information of the patches.
- Pass the sequence through multiple transformer encoder layers to compute global attention.
- Use the final classification token to output the defect status of the board.
The flow from raw image input to final classification output.
Raw circuit board capture
Splitting image into 16x16 segments
Self-attention processing
Defect vs Pass decision
- Input Image → segmentation → Patching
- Patching → sequence input → Transformer Encoder
- Transformer Encoder → token output → Classification
Expected business outcome
Improved consistency in defect identification across diverse board designs without requiring manual feature engineering.
Limitations and checks
ViT models require large datasets for pre-training to achieve high performance; check for overfitting by monitoring validation loss on smaller, domain-specific datasets.