C02 · From light to pixels

Sensor measurements and ISP

One channel is measured and two estimated. Before comparing colours, establish what sensor codes mean.

Background if needed

Light, optics and the camera

Theory

From electrons to codes

An absorbed photon can create an electron. Readout adds noise and offset; ADC maps signal to finite codes. Shot noise depends on mean signal. Black and white levels are needed before comparing codes across modes; black level is not scene light.

Subtracting is not dividing

A dark frame estimates offset; a flat field estimates relative sensitivity. Subtract offset before correcting response. A defective photosite needs another estimate; flat field cannot repair a missing measurement. Here parameters are exact and noiseless, so full correction is reversible.

Compare three modes on one channel. Without correction the code contains an offset; subtraction leaves response scaling; division restores the original magnitude. Calibration precedes channel comparison and mixing. For clarity this experiment displays three aligned planes; real RAW correction often acts on individual mosaic sites before reconstruction.

Offset and shading

Compare no correction, black subtraction and full correction. Inspect centre and corner.

Correction

NoneBlack subtractionBlack and flat field

Values below the imagesRGB

Offset and shading16 × 16

Exact values are in the point inspector

RGB linear → sRGB; display range 0…1

Selected channel: subtract offset
Working range × response
Divide

0.267

Selected correction16 × 16

Exact values are in the point inspector

RGB linear → sRGB; display range 0…1

Inspect the selected point

ViewFit×2×4

Full correction restores original values in a noiseless model. Real division by weak response also amplifies noise.

Parameters and details

Black level Black level0.08

Corner attenuation Corner attenuation0.6

The artificial black level is known exactly; response falls toward the corners. Offset subtraction and flat-field division solve different problems. Full correction is reversible without noise; dividing by a weak response would also amplify noise.

corrected = measurement − black(1 − black) · response

Full correction restores original values in a noiseless model. Real division by weak response also amplifies noise.

Select an image point. Arrows move it; Home and End reach row edges. Viewing does not change the calculation.

Changing the example requires JavaScript and Canvas. Computed images remain available above.

Measured or estimated?

A typical Bayer 2 × 2 block contains red, two green and blue filters. This distributes measurements spatially; it does not provide three complete RGB values per site. Real filters have broad, overlapping spectral responses.

Demosaicing estimates missing values: nearest neighbour is simple, bilinear averaging smooths and a directional estimate may preserve an edge better. None knows the hidden scene; fine patterns can become false colour. This example has known truth, exposing the error.

One measured channel, three output channels

Switch R, G, B at one point. Identify the measured channel, then compare estimates on straight and diagonal edges.

Values below the imagesRGB

Measured mosaic: one channel per site16 × 16

G measured

Exact values are in the point inspector

RGB linear → sRGB; display range 0…1. Color shows the measured channel; other channels are unknown. Pattern: no measurements.

Weights-0.250+0.25

Samples

0.650.30.650.65

⊙

Weights

0.250.250.250.25

Sum0.5625 ≈ 0.5625

0.5625

Reconstructed three channels16 × 16

Exact values are in the point inspector

RGB linear → sRGB; display range 0…1

Inspect the selected point

This channel was estimated

Calculation tables · up to 4 values

Values
0.65	0.3	0.65	0.65

Weights
0.25	0.25	0.25	0.25

Products
0.1625	0.075	0.1625	0.1625

Measured: 1; estimated: 0: 0
Ground truth (synthetic scene): 0.65
Estimation error: -0.0875

The site measures R. Only genuinely measured neighbours of the selected channel are listed below.

ViewFit×2×4

The measured channel stays unchanged. Two others are estimated from genuine neighbouring measurements, not unknown zeros.

Parameters and details

CFA phase

RGGBGRBGGBRGBGGR

RGGB is one CFA layout. Missing mosaic components are marked unmeasured, not as measured zeros. The bilinear variant averages the nearest available samples of the requested colour; borders use only existing samples. The directional variant changes only green estimation by choosing the lower difference; this is a teaching heuristic, not an industrial ISP.

Select an image point. Arrows move it; Home and End reach row edges. Viewing does not change the calculation.

Changing the example requires JavaScript and Canvas. Computed images remain available above.

White balance does not finish processing

Channel gains may neutralize one reference, but camera sensitivities differ from standard coordinates. Colour transformation, tone treatment and display encoding are also needed. Teaching sequence: measurement → correction → reconstruction → white balance and colour → tone and encoding.

Real ISPs combine or reorder some stages. RAW is not always Bayer, and JPEG no longer contains the original mosaic. Finished RGB cannot honestly be relabelled as its original photosite measurements.

White balance and a colour matrix

Choose None and change illumination. White balance removes the cast but the sensor channels remain mixed. Add the matrix and compare with the reference. Select a colour and switch channels to follow the three contributions.

Processing

NoneWhite balanceBalance and matrix

Warm / cool light Warm / cool light0.5

Values below the imagesRGB

After white balance12 × 12

Exact values are in the point inspector

RGB linear → sRGB; display range 0…1

Weights-0.9990+0.999

0.55950.999

0.8109-0.0999

0.84860.01

0.5589 + -0.081 + 0.0085 ≈ 0.4864

0.4864

Selected processing12 × 12

Exact values are in the point inspector

RGB linear → sRGB; display range 0…1

Inspect the selected point

One pixel's channels

Calculation tables · up to 3 values

Values
0.5595	0.8109	0.8486

Reference channel: 0.4864
Error: 0

White balance scales each channel separately. A matrix can mix all three inputs into one output; negative weights are valid.

ViewFit×2×4

Per-channel scaling and mixing three channels solve different problems. Together they restore colour in this exactly known model; a real camera requires measured coefficients.

Parameters and details

Authored linear model: illumination scales channels differently and the sensor mixes neighbouring components. White balance reverses the scaling; a separate matrix reverses mixing. These known coefficients cannot calibrate a real camera. No sRGB encoding has been applied yet.

Select an image point. Arrows move it; Home and End reach row edges. Viewing does not change the calculation.

Changing the example requires JavaScript and Canvas. Computed images remain available above.

Practice

Test the idea under another condition. Follow the task above the images; point inspection helps explain the result. This exploration is not scored.

Do not confuse estimates with measurements

A missing green channel is selected at a diagonal edge with another CFA phase. Predict whether copying one neighbour or averaging will have less error. Change method and compare against known truth. Then select a measured channel: it must stay unchanged.

Values below the imagesRGB

Measured mosaic: one channel per site16 × 16

G measured

Exact values are in the point inspector

RGB linear → sRGB; display range 0…1. Color shows the measured channel; other channels are unknown. Pattern: no measurements.

Weights-10+1

Samples

0.3

⊙

Weights

Sum0.3 ≈ 0.3

0.3

Reconstructed three channels16 × 16

Exact values are in the point inspector

RGB linear → sRGB; display range 0…1

Inspect the selected point

This channel was estimated

Calculation tables · up to 1 values

Values
0.3

Weights
1

Products
0.3

Measured: 1; estimated: 0: 0
Ground truth (synthetic scene): 0.65
Estimation error: -0.35

The site measures R. Only genuinely measured neighbours of the selected channel are listed below.

ViewFit×2×4

The measured channel stays unchanged. Two others are estimated from genuine neighbouring measurements, not unknown zeros.

Parameters and details

CFA phase

RGGBGRBGGBRGBGGR

Select an image point. Arrows move it; Home and End reach row edges. Viewing does not change the calculation.

Changing the example requires JavaScript and Canvas. Computed images remain available above.

Games

Do not confuse estimates with measurements

Which estimation rule produces this result at the edge?

Channel estimate

Nearest sampleBilinearDirectional G estimate

Values below the imagesRGB

Measured mosaic: one channel per site16 × 16

G measured

Exact values are in the point inspector

RGB linear → sRGB; display range 0…1. Color shows the measured channel; other channels are unknown. Pattern: no measurements.

Weights-10+1

Samples

0.3

⊙

Weights

Sum0.3 ≈ 0.3

0.3

Reconstructed three channels16 × 16

Exact values are in the point inspector

RGB linear → sRGB; display range 0…1

Reference16 × 16

Exact values are in the point inspector

RGB linear → sRGB; display range 0…1

Compare your result with the reference.

Inspect the selected point

This channel was estimated

Calculation tables · up to 1 values

Values
0.3

Weights
1

Products
0.3

Measured: 1; estimated: 0: 0
Ground truth (synthetic scene): 0.65
Estimation error: -0.35

The site measures R. Only genuinely measured neighbours of the selected channel are listed below.

ViewFit×2×4

The measured channel stays unchanged. Two others are estimated from genuine neighbouring measurements, not unknown zeros.

Parameters and details

Select an image point. Arrows move it; Home and End reach row edges. Viewing does not change the calculation.

Changing the example requires JavaScript and Canvas. Computed images remain available above.

Test your explanation

Question 1 of 5

Measurement is 0.30, black level 0.10, working range × response 0.50. Corrected result?

Choose one answer. It is checked immediately.

Without JavaScript, use the questions below. Open each answer after making your choice.

Measurement is 0.30, black level 0.10, working range × response 0.50. Corrected result?

0.40
0.50

Answer and explanation

0.40. First 0.30 − 0.10 = 0.20, then 0.20 / 0.50 = 0.40.

White balance removed the illumination cast, but the sensor mixes channels. What is needed next?

A colour matrix
Only a common gain

Answer and explanation

A colour matrix. One output can depend on several input channels. Independent gains are insufficient.

How many components does one Bayer photosite measure?

One
Three
None

Answer and explanation

One. Yes. Two additional output components are estimated.

Is unmeasured red at a green site zero?

Yes, it is absent
It is unknown before estimation

Answer and explanation

It is unknown before estimation. Correct. Unknown differs from measured zero.

Black was subtracted but corners remain dark. What should be checked?

Response and shading
Subtract the same black level again

Answer and explanation

Response and shading. Yes. They multiply signal and need another correction.

Limitations and references

Limits of this example

Synthetic mosaic. Directional estimation is a local green heuristic, not Malvar or an industrial ISP. No RAW import or real-camera calibration.

Mathematical reference

Probability and statistical models

From electrons to codes

Subtracting is not dividing

Offset and shading

Measured or estimated?

One measured channel, three output channels

White balance does not finish processing

White balance and a colour matrix

Do not confuse estimates with measurements

Do not confuse estimates with measurements

Test your explanation

Limits of this example

Mathematical reference

Sources and further reading