Computer vision

Overview

Computer vision is the part of machine learning that takes pixels in: classifying an image, finding objects in it, labelling every pixel, matching it to text, and, since 2022, generating images and video from a description. The field moved from hand-designed features to convolutional networks in 2012, added vision transformers in 2020, and now mostly fine-tunes pretrained backbones. This hub is for engineers who need to pick a model and a training recipe for a visual task and want to know what is going on inside.

It matters because vision tasks have sharp practical constraints that text tasks do not: images are large, labels are expensive to draw, and inference often has to run on a camera, a phone or an embedded board. Choosing the right backbone and the right amount of fine-tuning decides whether the project fits its budget.

Key ideas

Convolutions and receptive fields

A convolutional layer slides a small filter, typically 3 by 3, across the image and computes the same weighted sum at every position. Weight sharing means a feature detector learned in one corner works everywhere, and locality means early layers see edges and textures while later layers, whose inputs are the outputs of earlier ones, see larger and larger patches of the original image. That patch is the receptive field, and it grows by two pixels per 3 by 3 layer and multiplies with every stride or pooling step. Residual connections made networks of 50 to 150 layers trainable, and a ResNet-50 (25 M parameters, 2015) pretrained on ImageNet is still a reasonable default backbone for a new classification task with a few thousand labelled images.

Vision transformers and pretrained backbones

A vision transformer cuts the image into patches, usually 16 by 16 pixels, embeds each patch as a token and runs the same attention stack as a language model, see transformers. ViT-B/16 (86 M parameters, 2020, Apache-2.0 weights) has no built-in locality, so it needs far more data than a CNN to train from scratch, but given that data it scales better. In practice nobody trains either from scratch for an applied task. Self-supervised backbones such as DINOv2 (ViT-g at 1.1 B parameters, Apache-2.0, April 2023) give features that work for classification, retrieval and depth with a linear head and no labels, and CLIP (2021) pairs an image encoder with a text encoder so that a class can be described in words at inference time, which is zero-shot classification. timm and torchvision ship hundreds of pretrained backbones with a licence listed per model.

Detection and segmentation

Detection returns boxes with class labels. Two-stage detectors such as Faster R-CNN propose regions and then classify them, one-stage detectors such as the YOLO family predict boxes on a grid in a single pass and dominate real-time use, and DETR (2020) treats detection as set prediction with a transformer and no hand-designed anchors or non-maximum suppression. Segmentation labels pixels: semantic (one class per pixel), instance (one mask per object) or panoptic (both). U-Net remains the shape of most medical and industrial segmenters, and the Segment Anything Model (636 M parameters in the ViT-H variant, Apache-2.0, April 2023) segments any object from a click or box prompt with no task-specific training, with SAM 2 (July 2024) extending it to video. The metric for both tasks is average precision over intersection-over-union thresholds, and mAP at a single threshold of 0.5 hides most of the quality differences. Note that the widely used Ultralytics YOLO implementations are AGPL-3.0, which matters for a closed product.

Diffusion for images and video

A diffusion model learns to remove noise. Training adds Gaussian noise to an image at a random level and trains a network to predict the noise, and generation starts from pure noise and applies the network a few dozen times, each step removing a little. Latent diffusion runs this in the compressed space of a variational autoencoder, which made it affordable: Stable Diffusion 1.5 (860 M parameter U-Net, CreativeML OpenRAIL-M licence, October 2022) generates a 512 by 512 image on a consumer GPU. Text conditioning comes from a frozen text encoder, and classifier-free guidance trades diversity for prompt adherence. Newer models replace the U-Net with a transformer: FLUX.1 [dev] (12 B parameters, August 2024) is open weights under a non-commercial licence. Video diffusion treats frames as a third axis and adds temporal attention, which multiplies memory and compute by the frame count and is why video models lag images by a few years at every quality level. The generative models hub covers the mathematics and the audio and 3D variants.

Where to start

Begin with Your first neural network in PyTorch, then build the MNIST CNN project, which trains the network in the diagram above on 28 by 28 images and shows what each filter learned. Embeddings and similarity search carries over directly to image retrieval with a pretrained backbone. The deep learning hub explains the normalisation and augmentation choices every vision recipe depends on.

Tools and platforms

Common mistakes

• Training a backbone from scratch on 2,000 images. Fine-tune a pretrained one, and start with a frozen backbone and a linear head.
• Normalising with one mean and standard deviation in training and another at inference, or swapping RGB and BGR between OpenCV and PIL.
• Resizing without preserving aspect ratio, so that the model sees squashed objects it never met in training.
• Evaluating on augmented validation images. Augmentation is for training only.
• Reporting mAP at IoU 0.5 alone. Report the 0.5 to 0.95 average as well, since box precision is what downstream code depends on.
• Shipping an AGPL-3.0 detector inside a closed product without reading the licence.

Going deeper

• He et al., Deep Residual Learning for Image Recognition, 2015.
• Dosovitskiy et al., An Image is Worth 16x16 Words, 2020, the ViT paper, and Oquab et al., DINOv2, 2023.
• Carion et al., End-to-End Object Detection with Transformers, 2020, and Kirillov et al., Segment Anything, 2023.
• Ho, Jain and Abbeel, Denoising Diffusion Probabilistic Models, 2020, and Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models, 2021.
• Radford et al., Learning Transferable Visual Models From Natural Language Supervision, 2021, the CLIP paper.
• The torchvision models documentation, which lists accuracy, parameters and licence for each pretrained weight.