C27 · Learned image methods
Generation and learned reconstruction
Where does a pattern come from when no original is known? Separate the allowed image family, adding noise, prediction and a step toward a cleaner state.
Theory
A decoder defines a family of possible images
Constructing a pattern requires specifying which patterns are allowed. Our decoder takes two numbers, a and b, and combines horizontal and vertical patterns. It emits 256 values but offers only two independent directions of change. Try making a circle: no a and b can produce it, because that form is absent from this family.
A compact controlling code is called a latent representation. In a learned system the image mapping is fitted to data and is usually nonlinear. An autoencoder first encodes a known image, then reconstructs it through a decoder. A generator can start from a chosen or random code without a known original. Compression, reconstruction and generation remain different tasks even when their decoders look similar.
Two numbers instead of an entire pattern
Change a and b: the decoder combines two authored patterns with these coefficients. Try horizontal-only, vertical-only and mixed stripes.
Exact values are in the point inspector
Blue − · white 0 · red +; scale -2…2
Exact values are in the point inspector
Blue − · white 0 · red +; scale -2…2
Inspect the selected point
What the plot shows
A small code restricts the output family. This decoder cannot recover details absent from its basis.
- Latent coordinates
- 2
- Image values
- 256
Parameters and details
Two numbers control 256 values, but only inside the linear plane of two authored patterns. This is an authored linear decoder, not a VAE/GAN/diffusion model. It cannot generate arbitrary photographs.
Select an image point. Arrows move it; Home and End reach row edges. Viewing does not change the calculation.
Changing the example requires JavaScript and Canvas. Computed images remain available above.
Noising is specified; reverse prediction must be learned
The forward diffusion process gradually adds Gaussian noise. In a standard formulation controls new-noise variance and is the product of . A marginal sample can be computed directly from known and ε. Square-root coefficients make the variances consistent.
In this first example the clean pattern is already known: the slider only mixes it with noise. The graph answers one question: which coefficients currently weight signal and noise? Markers at t connect its curves to the image. Moving the slider backward simply recomputes a known mixture; this is not generation or recovery of an unknown original.
Forward path: add noise
Select step t. At a point, inspect how much known image remains and how much fixed Gaussian noise contributes.
Exact values are in the point inspector
Blue − · white 0 · red +; scale -4…4
Exact values are in the point inspector
Blue − · white 0 · red +; scale -4…4
Inspect the selected point
What the plot shows
Markers at the selected step t give the exact multipliers currently applied to the known image and noise. This is a noising schedule, not restoration quality.
- Signal
- Noise
- Signal now: 6; 0.8982
- Noise now: 6; 0.4396
At step zero xₜ=x₀. The noise contribution then grows. There is no predicted noise or reverse generation here: only a specified forward noising calculation.
- Step t
- 6
- ᾱₜ
- 0.8068
Parameters and details
β increases linearly from 0.01 to 0.12 over 12 steps; ᾱ is the product of (1−β). One fixed Gaussian sample compares t: each frame has the required conditional marginal, but this is not a stepwise Markov trajectory. There is no learned reverse model here.
Select an image point. Arrows move it; Home and End reach row edges. Viewing does not change the calculation.
Changing the example requires JavaScript and Canvas. Computed images remain available above.
How to go backward without a known original
Generation does not know which pattern is hidden in noise: there is no predetermined correct original. A predictor uses learned regularities to estimate clean signal or noise. A separate sampler uses that estimate for one step. Each new state needs another prediction. Subtracting a previously known noise field once cannot replace this procedure.
Here a fully declared two-pattern dictionary replaces a large network. Bayes' rule measures how compatible each is with the current state; their weighted mixture estimates the clean pattern. A cosine schedule starts at pure noise and reduces its contribution over 12 deterministic steps. This is a separate schedule, not a continuation of the short forward mixture above. Each estimate is recomputed from the current state, with no hidden correct output in the code.
From noise: prediction and a reverse step
Increase completed updates. The left view is the state before the last update; the right is its result. Three visible stages expose the predicted clean pattern, estimated noise and their new mixture. Select a cell: the two center contributions must produce its right-hand value. Initial noise remains available separately and stays fixed for one variant.
Exact values are in the point inspector
Blue − · white 0 · red +; scale -4…4
- Estimate residual noise from the current value
Exact values are in the point inspector
Blue − · white 0 · red +; scale -4…4
Inspect the selected point
What the plot shows
Early estimates can be uncertain mixtures. As noise decreases, one candidate can dominate. Changing initial noise can change the result. The predictor knows its dictionary, not a hidden correct original: this generates within a declared small family rather than recovering a real photograph.
- Completed steps
- 4
- Remaining t
- 8
- Pattern A weight
- 0
- Pattern B weight
- 1
Parameters and details
Only a two-pattern dictionary is known, not a hidden correct original. Bayes' rule weights its candidates for the current noisy state and estimates a clean pattern. A deterministic step transfers estimated signal and residual noise to a lower t. This explicit two-pattern teaching model uses a cosine schedule; it is neither a trained network nor photo generation. Twelve finite steps do not establish exact sampling of the specified distribution.
Select an image point. Arrows move it; Home and End reach row edges. Viewing does not change the calculation.
Changing the example requires JavaScript and Canvas. Computed images remain available above.
How this connects to actual generators
Reference and further methods
Latent diffusion noises a compressed representation and later decodes it. U-Net and DiT are different prediction backbones, not two names for a noise schedule: DiT processes latent patches with a transformer. An autoregressive model instead predicts the next element or token from earlier ones. These families should not be merged into one 'magic filter'.
Conditioning, such as text, a class or a known image region, changes which possible outputs are preferred. Guidance strengthens conditioning but can affect diversity and artifacts. A VAE learns a latent-code distribution; a GAN trains a generator using discriminator feedback. These families are further-reading directions: the examples above show a restricted family and sampling steps, not simulated training of these models.
Restoration must remain consistent with measurements
Restoration, unlike unconstrained generation, has an observation the answer must match. Yet different detailed originals can share one blurred or reduced measurement. A model assumption, or prior, prefers plausible candidates. Neither that term nor high sharpness makes unverified detail factual: below, different checker patterns give exactly the same block averages.
Two different originals, one measurement
The camera kept only each 2×2 block average: the measurement on the left. Move the detail slider from −1 to +1. The pattern on the right changes while each measured block average stays fixed. Inspecting a selected block lets you check this numerically.
Exact values are in the point inspector
Grayscale 0…1: black to white.
Exact values are in the point inspector
Grayscale 0…1: black to white.
Inspect the selected point
What the plot shows
Equal measurement consistency does not make a detailed result uniquely correct. When data cannot distinguish candidates, a model assumption chooses the detail. Verification needs an independent more detailed observation, not another sharpness score.
Parameters and details
Every detailed candidate gives the same 8×8 measurement after 2×2 averaging. The slider changes only unverified fine detail; it acquires no new data. Neither apparent sharpness nor agreement with this measurement establishes which candidate was the actual original.
Select an image point. Arrows move it; Home and End reach row edges. Viewing does not change the calculation.
Changing the example requires JavaScript and Canvas. Computed images remain available above.
How denoisers are trained and connected
Reference and further methods
Noisy→clean supervision compares a corrupted observation with a known clean reference. Noise2Noise substitutes a different corrupted version: under squared loss the argument needs suitable conditional means and noise independence. Shared noise or bias breaks that argument. Blind-spot methods hide the center value from the predictor: another training task, with its own noise and structural assumptions.
Deep Image Prior fits network parameters to one image; early stopping can separate structure from later noise fitting but gives no universal guarantee. Unrolling turns a known algorithm's updates into tunable stages. Plug-and-play alternates measurement consistency and denoising; an arbitrary denoiser need not minimize one energy or guarantee convergence. These are further-study directions: the explicit example above establishes measurement ambiguity without implementing these methods.
Practice
Test the idea under another condition. Follow the task above the images; point inspection helps explain the result. This exploration is not scored.
What changed: initial noise or the predictor?
This shows a completed path from a different noise sample. Return to zero updates and increase one at a time. Compare the state before the last update with its result; prediction and residual-noise stages explain each transition. Change only the noise variant: when does the stripe pattern change or stay the same? Two equal outputs do not mean that a hidden original was known.
Exact values are in the point inspector
Blue − · white 0 · red +; scale -4…4
- Estimate residual noise from the current value
Exact values are in the point inspector
Blue − · white 0 · red +; scale -4…4
Inspect the selected point
What the plot shows
Early estimates can be uncertain mixtures. As noise decreases, one candidate can dominate. Changing initial noise can change the result. The predictor knows its dictionary, not a hidden correct original: this generates within a declared small family rather than recovering a real photograph.
- Completed steps
- 12
- Remaining t
- 0
- Pattern A weight
- 0
- Pattern B weight
- 1
Parameters and details
Only a two-pattern dictionary is known, not a hidden correct original. Bayes' rule weights its candidates for the current noisy state and estimates a clean pattern. A deterministic step transfers estimated signal and residual noise to a lower t. This explicit two-pattern teaching model uses a cosine schedule; it is neither a trained network nor photo generation. Twelve finite steps do not establish exact sampling of the specified distribution.
Select an image point. Arrows move it; Home and End reach row edges. Viewing does not change the calculation.
Changing the example requires JavaScript and Canvas. Computed images remain available above.
Games
Test your explanation
Question 1 of 5
The example knows x₀ and ε. What does the t slider compute?
Exact values are in the point inspector
Grayscale 0…1: black to white.
Exact values are in the point inspector
Grayscale 0…1: black to white.
Exact values are in the point inspector
Grayscale 0…1: black to white.
Exact values are in the point inspector
Blue − · white 0 · red +; scale -4…4
Exact values are in the point inspector
Blue − · white 0 · red +; scale -4…4
Exact values are in the point inspector
Blue − · white 0 · red +; scale -4…4
Exact values are in the point inspector
Blue − · white 0 · red +; scale -4…4
Choose one answer. It is checked immediately.
Limitations and references
Limits of this example
This runs a linear decoder, forward noising and a 12-step deterministic sampler with an exact Bayesian predictor for two original patterns. It is not a trained denoiser, GAN, VAE or photo generator; exact distribution sampling is not claimed. The measurement counterexample shows ambiguity, not recovery of an actual scene.
Mathematical reference
Sources and further reading
- Kingma and Welling: Auto-Encoding Variational Bayes
- Ho et al.: Denoising Diffusion Probabilistic Models
- Song et al.: Denoising Diffusion Implicit Models
- Rombach et al.: High-Resolution Image Synthesis with Latent Diffusion Models
- Peebles and Xie: DiT
- Lehtinen et al.: Noise2Noise
- Ulyanov et al.: Deep Image Prior