C25 · Learned image methods
Vision tasks and their outputs
One scene, different questions: category, box, instance, depth or motion.
Theory
Choose the question before the output type
Classification gives an image-level category. Localization adds one object's position; detection returns boxes and categories for multiple objects. Semantic segmentation assigns a class to each point; instance segmentation separates objects of the same class. Panoptic annotation combines object instances with regions such as sky and road.
Two red shapes may have the same semantic class but different instance IDs. An ID is categorical: averaging instances 1 and 2 does not create object 1.5. Each task needs matching annotations and evaluation.
A class or an individual object
One unchanged scene answers different questions. Compare the image class, two boxes, semantic mask and individual instances. Threshold mode shows the actual red channel on the left: select a pixel and check R against the threshold.
Exact values are in the point inspector
RGB; display range 0…1
Exact values are in the point inspector
Color denotes a category, not magnitude.
Inspect the selected point
What the plot shows
A class does not locate objects. A box does not follow their exact shape. A semantic mask does not identify individuals of one class. A threshold tests a number, not meaning.
- Objects
- 2
- Object classes
- 1
Parameters and details
One input, different output formats. A class is one name; boxes give locations; masks label individual pixels; instances additionally identify individual objects. Only threshold mode computes a prediction using a declared rule on visible R. Annotation and rule output are kept distinct.
Select an image point. Arrows move it; Home and End reach row edges. Viewing does not change the calculation.
Changing the example requires JavaScript and Canvas. Computed images remain available above.
Boxes: overlap, duplicates and misses
IoU divides box-intersection area by union area. Detector evaluation matches predictions to annotations under specified rules and counts false detections and missed objects. A high score alone does not imply an accurate box.
A proposal-based detector first proposes regions; a dense head predicts on a grid. Anchors define reference boxes, while anchor-free variants do not use them in the same way. NMS removes some overlapping predictions using score and IoU, but can suppress nearby true objects. DETR treats outputs as a set and trains with one-to-one matching, so it should not be presented as necessarily requiring NMS.
How well boxes overlap
Shift the candidate relative to the reference. The intersection category grows or shrinks with the exact IoU.
Exact values are in the point inspector
Color denotes a category, not magnitude.
Exact values are in the point inspector
Color denotes a category, not magnitude.
Inspect the selected point
What the plot shows
IoU=1 means identical boxes; IoU=0 means no overlap. Classifier confidence and localization quality are different quantities.
- IoU
- 0.5556
Parameters and details
Boxes share coordinates: compute exact intersection and union, including box area outside the image. IoU measures geometric overlap, not class probability. NMS compares predictions with each other, not with ground truth.
Select an image point. Arrows move it; Home and End reach row edges. Viewing does not change the calculation.
Changing the example requires JavaScript and Canvas. Computed images remain available above.
Every dense output has meaning and units
Reference and further methods
Keypoints are coordinates of joints or other landmarks. Depth is distance or inverse depth in a declared system; one ordinary photograph does not determine absolute scene scale without further assumptions. Optical flow is a two-dimensional displacement between frames, not velocity in meters per second without calibration and timing.
For a mask, class and boundaries matter; for a box, location; for restoration, detail and consistency with measurements. Cross-entropy trains categorical outputs; regression losses train numerical values, while task metrics evaluate their consequences. A spectral–spatial model need not accept three RGB channels: bands have real wavelengths, and training/test splits must avoid leaking nearby portions of the same scene.
Practice
Test the idea under another condition. Follow the task above the images; point inspection helps explain the result. This exploration is not scored.
Separate objects with one visible channel
At threshold 0.7 some real object pixels are missing. Find a threshold retaining both objects and removing background. Inspect a dark object cell and a bright background cell: between which values must the threshold lie? Compare with instance labels: a good binary mask has not assigned individual object identities.
Exact values are in the point inspector
Grayscale 0…1: black to white.
- Compare R with the threshold
Exact values are in the point inspector
Color denotes a category, not magnitude.
Inspect the selected point
What the plot shows
A class does not locate objects. A box does not follow their exact shape. A semantic mask does not identify individuals of one class. A threshold tests a number, not meaning.
- Objects
- 2
- Object classes
- 1
Parameters and details
One input, different output formats. A class is one name; boxes give locations; masks label individual pixels; instances additionally identify individual objects. Only threshold mode computes a prediction using a declared rule on visible R. Annotation and rule output are kept distinct.
Select an image point. Arrows move it; Home and End reach row edges. Viewing does not change the calculation.
Changing the example requires JavaScript and Canvas. Computed images remain available above.
Games
Test your explanation
Question 1 of 5
Two red objects belong to one class. Which output directly distinguishes individual objects?
Exact values are in the point inspector
RGB; display range 0…1
Exact values are in the point inspector
Color denotes a category, not magnitude.
Exact values are in the point inspector
Grayscale 0…1: black to white.
Exact values are in the point inspector
Color denotes a category, not magnitude.
Exact values are in the point inspector
RGB; display range 0…1
Exact values are in the point inspector
Color denotes a category, not magnitude.
Choose one answer. It is checked immediately.
Limitations and references
Limits of this example
All masks and boxes here are authored or computed by simple declared rules. There is no trained detector, depth or flow inference on arbitrary photographs.