C26 · Learned image methods
Patches, attention and representations
Trace actual weights and contributions of sixteen patches without downloading a transformer.
Background if needed
Theory
A query determines what to mix
Split the image into patches. Each can produce a vector, a token. A query is compared with each token's key through a dot product. Division by accounts for key dimension; softmax turns numbers into nonnegative weights summing to 1. These weights mix value vectors, not necessarily pixels.
In this demonstration a patch mean represents content and projections are hand-authored. Multi-head attention performs several mixtures with different projections and combines their results. A large weight shows contribution to this operation but is not proof of a causal explanation of a model's final answer.
Who contributed how much to the answer
Select a patch in the input: the outline pins query Q. Then select another patch on the weight map: this is the inspected key K. Follow three visible stages: similarities, normalized weights, one response. Each stage exposes its actual arithmetic in the center. Lower temperature T concentrates weight on similar keys.
Exact values are in the point inspector
Grayscale 0…1: black to white.
- Q · K
- Scale similarity
- Exponential after a common shift
- Divide by the sum of 16 exponentials
Exact values are in the point inspector
Grayscale 0…0.25: black to white.
Inspect the selected patch
What the plot shows
Weights are nonnegative and sum to 1. Lower temperature concentrates them. Temperature is an educational extension, not a required setting of every ViT.
- Tokens
- 16
- Query–key pairs in a full layer
- 256
- Selected-query response
- 0.555
Parameters and details
Exact teaching attention: Q=K=[2·(mean−0.5), x position, y position], V=patch mean. Dark and bright patches have opposite signs in their first component, so similarity distinguishes them. Projections are not learned. Weights use a fixed 0…0.25 display scale: values above the white threshold remain unclipped in calculations and available numerically. The response is one weighted sum, not a class or reconstructed image.
Select an image point. Arrows move it; Home and End reach row edges. Viewing does not change the calculation.
Changing the example requires JavaScript and Canvas. Computed images remain available above.
Where a patch is and who can communicate
Content alone cannot distinguish identical patches at different positions. Positional representations convey order or geometry. Without them, self-attention treats a token permutation with the corresponding output permutation; pixel layout is not supplied automatically.
Global attention has N² query–key pairs. Sixteen tokens give 256; halving patch side at fixed image size gives four times as many tokens and sixteen times as many pairs. ViT uses patches; Swin restricts communication to windows and shifts them between layers. Locality saves computation, but distant regions then communicate through multiple steps.
Similar content at different positions
Pin the top-left dark query. At coordinate contribution 0, compare its weight with the equally dark patch two columns right: identical content gives identical weights. Restore coordinate contribution to 1 without changing the query. Position now distinguishes these locations while the image stays unchanged.
Exact values are in the point inspector
Grayscale 0…1: black to white.
- Q · K
- Scale similarity
- Exponential after a common shift
- Divide by the sum of 16 exponentials
Exact values are in the point inspector
Grayscale 0…0.25: black to white.
Inspect the selected patch
What the plot shows
Without coordinates, equal brightness gives equal keys. Coordinates let this example distinguish location; this is a simple authored encoding, not a learned positional table.
- Tokens
- 16
- Query–key pairs in a full layer
- 256
- Selected-query response
- 0.4052
Parameters and details
Exact teaching attention: Q=K=[2·(mean−0.5), x position, y position], V=patch mean. Dark and bright patches have opposite signs in their first component, so similarity distinguishes them. Projections are not learned. Weights use a fixed 0…0.25 display scale: values above the white threshold remain unclipped in calculations and available numerically. The response is one weighted sum, not a class or reconstructed image.
Select an image point. Arrows move it; Home and End reach row edges. Viewing does not change the calculation.
Changing the example requires JavaScript and Canvas. Computed images remain available above.
One representation can learn different objectives
Reference and further methods
Classification learning separates specified classes. Contrastive learning brings related examples together and separates others; CLIP pairs images with text. Masked autoencoding reconstructs hidden image portions. These are different training objectives rather than interchangeable architecture names.
Closeness in embedding space depends on data, normalization and similarity; it does not guarantee the same physical object. SAM passes an image and user prompt to a mask decoder; a point or box refines the request but does not promise correct recognition of every class. Actual embeddings and results need a model name, version, preprocessing and source data; hand-authored numbers here are not passed off as such results.
Practice
Test the idea under another condition. Follow the task above the images; point inspection helps explain the result. This exploration is not scored.
Separate content and position effects
The corner query is pinned, temperature is 0.3 and coordinates are off. Inspect a distant key on the weight map and note its contribution. Enable coordinates without changing the query. Does proximity matter under this declared rule? Repeat with another query; do not interpret this map as a semantic explanation.
Exact values are in the point inspector
Grayscale 0…1: black to white.
- Q · K
- Scale similarity
- Exponential after a common shift
- Divide by the sum of 16 exponentials
Exact values are in the point inspector
Grayscale 0…0.25: black to white.
Inspect the selected patch
What the plot shows
Without coordinates, equal brightness gives equal keys. Coordinates let this example distinguish location; this is a simple authored encoding, not a learned positional table.
- Tokens
- 16
- Query–key pairs in a full layer
- 256
- Selected-query response
- 0.3193
Parameters and details
Exact teaching attention: Q=K=[2·(mean−0.5), x position, y position], V=patch mean. Dark and bright patches have opposite signs in their first component, so similarity distinguishes them. Projections are not learned. Weights use a fixed 0…0.25 display scale: values above the white threshold remain unclipped in calculations and available numerically. The response is one weighted sum, not a class or reconstructed image.
Select an image point. Arrows move it; Home and End reach row edges. Viewing does not change the calculation.
Changing the example requires JavaScript and Canvas. Computed images remain available above.
Games
Test your explanation
Question 1 of 5
What is the sum of softmax weights over available keys?
Exact values are in the point inspector
Grayscale 0…1: black to white.
Exact values are in the point inspector
Grayscale 0…0.25: black to white.
Exact values are in the point inspector
Grayscale 0…1: black to white.
Exact values are in the point inspector
Grayscale 0…1: black to white.
Choose one answer. It is checked immediately.
Limitations and references
Limits of this example
This executes one small attention operation with declared manual projections, not ViT, CLIP or SAM. There are no semantic embeddings, text prompts or large-model inference.