C24 · Learned image methods
Why CNN architectures differ
Compare information paths, shapes and block cost, rather than network names alone.
Background if needed
Theory
Sequence, branches and a shortcut
Start with a question: along which path can an input cell influence an answer? In a sequence each layer receives only the previous output. Two 3×3 filters can depend on a 5×5 input region; three on 7×7. Here branch F really averages the image repeatedly: an explicit test of paths and support, not a trained network.
A shortcut adds original x to F(x), supplying an unchanged contribution. Toggle it with layers and inspected cell fixed. Addition requires equal shapes: two 16×16×1 maps give one such map. Channel concatenation keeps both values separately and gives 16×16×2. These are different operations; a brighter sum alone proves neither better quality nor easier training.
Plain path and a shortcut
The diagram traces x through branch F of 3×3 averages. Enable the shortcut to send the original x to the sum around F. Select stages to compare fields on one scale.
- Input x16×16×1 · one map
- Pass x unchangedInput x → Sum x + F(x)
Exact values are in the point inspector
Grayscale 0…2: black to white.
Exact values are in the point inspector
Grayscale 0…2: black to white.
Inspect the selected point
What the plot shows
Each layer expands the dependency region. A shortcut preserves a direct x path, but does not promise a prettier output or values restricted to 0…1.
- 3×3 layers
- 2
- Preserved side: 16×16
- 16
- Receptive-field side
- 5
- Branch multiplications
- 4608
- Shortcut additions
- 256
Parameters and details
Here F is a specified chain of 3×3 averages, not a trained ResNet. Addition requires matching sizes and channels. Addition is not channel concatenation.
Select an image point. Arrows move it; Home and End reach row edges. Viewing does not change the calculation.
Changing the example requires JavaScript and Canvas. Computed images remain available above.
Fewer operations and more spatial detail
A k×k convolution from C input to O output channels has weights. Depthwise applies a separate spatial kernel to each channel; pointwise 1×1 mixes channels: . For k=3, C=16, O=32 this is 4608 versus 656 weights, excluding biases. It changes the block's expressiveness rather than providing a free identical replacement.
On a 16×16 map each weight is applied at 256 positions: here that is 1,179,648 versus 167,936 multiplications per pass, excluding biases and other operations. Operation count is useful, but not a browser/GPU speed measurement: memory, implementation and shapes matter. Halving both sides quarters the position count but discards fine detail. The next example shows why a separate detail path is useful.
Two resolutions and two channels
Averaging each 2×2 block reduces a 16×16 map to 8×8: different fine patterns can have identical averages. This is the information loss seen in C14's scales and pyramids. Convolution on the small grid covers a larger input region. Repeating cells restores the 16×16 size, but not the lost pattern.
Retain the input map alongside the upsampled map. Concatenation joins along the channel axis: each point holds [u,x], giving 16×16×2 rather than a sum. A 1×1 convolution multiplies this pair by the weight row [1−α, α]. This is M02's matrix channel mixing, without spatial displacement. The diagram's concatenation node is static: its two channels are already available separately in the input and upsampled fields.
U-Net combines high-resolution features with the decoder's upsampled output. Our one-level block illustrates only that principle: averaging, fixed convolution, cell repetition and an adjustable mixture. It is not a full U-Net: there is no training, hierarchy of levels or class prediction. At the original returns through a direct copy, not because a decoder learned to recover detail.
Coarse context and a direct detail path
Follow the diagram down to 8×8 and back to 16×16. The comparison task is to preserve original fine detail. At α=0 inspect lost detail and error; at α=1 the retained copy returns the input. Compare multiplication counts on 8×8 and 16×16 grids: coarse-path savings do not provide free detail recovery.
- Input x16×16×1 · direct-path channel
- Concatenate channels16×16×2 · [upsampled, input]
- Keep the original map xInput x → Concatenate channels
Exact values are in the point inspector
Grayscale 0…1: black to white.
Exact values are in the point inspector
Grayscale 0…1: black to white.
Inspect the selected point
What the plot shows
Upsampling does not recover lost detail. A separate direct path retains it. Concatenation keeps both channels; the 1×1 convolution decides how much to use from each. Here you choose its weights; there is no training.
- Mean squared input error
- 0.0007
- 3×3 multiplies on the 8×8 grid
- 576
- 3×3 multiplies on the 16×16 grid
- 2304
- Channels after concatenation
- 2
- Coarse-path weight
- 0.5
- Direct-path weight
- 0.5
Parameters and details
One-level authored encoder–decoder with fixed weights, not U-Net or segmentation. Convolution repeats edges. Each coarse cell averages exactly one 2×2 block; upsampling repeats its position without shifting the grid. Concatenation has two scalar channels, viewed separately as the upsampled coarse path and input map, not as RGB. The final 1×1 layer mixes them with weights [1−α, α].
Select an image point. Arrows move it; Home and End reach row edges. Viewing does not change the calculation.
Changing the example requires JavaScript and Canvas. Computed images remain available above.
Upsampling does not reverse information loss
A transposed convolution is the transpose matrix of a specified linear transform. It adds input-element footprints onto a larger grid. It is not the inverse matrix: after downsampling, different originals may have produced the same result.
Resize-then-convolution first interpolates a map, then filters it. The coverage comparison isolates one source of checkerboard artifacts. Actual appearance also depends on weights, boundaries and training. A larger output alone does not establish trustworthy new detail.
Where checkerboard patterns come from
Select a cell on the right map: outlines show every footprint covering it. Each adds exactly 1. Change footprint size at the same stride 2 and compare outline count with the cell's brightness. At an edge, a footprint can extend beyond this crop of an infinite grid.
Exact values are in the point inspector
Grayscale 0…4: black to white.
Exact values are in the point inspector
Grayscale 0…4: black to white.
Inspect the selected point
What the plot shows
A size-3 kernel at stride 2 gives alternating 1, 2 and 4 contributions. A size divisible by stride gives even interior coverage, though learned weights may still create artifacts.
- Minimum coverage
- 1
- Maximum coverage
- 4
Parameters and details
Outlines on the right are all footprints covering the selected cell: each contributes exactly 1. Their count equals the coverage-map value. The unit grid is infinite with stride 2; edge outlines may continue outside the shown crop. A 3×3 kernel overlaps unevenly; 2×2 and 4×4 evenly. This is a coverage map, not a generative-model run.
Select an image point. Arrows move it; Home and End reach row edges. Viewing does not change the calculation.
Changing the example requires JavaScript and Canvas. Computed images remain available above.
Where these connections appear
Reference and further methods
LeNet, AlexNet and VGG use recognizable sequences of local filters and map reductions. Inception concatenates parallel branch channels. ResNet uses shortcut addition; DenseNet concatenates earlier maps along channels. A network name does not replace checking shapes: addition and concatenation have different requirements.
MobileNet uses the separate spatial and channel operations counted above. EfficientNet jointly scales depth, width and input resolution. FCN and U-Net produce spatial outputs; U-Net retains high-detail decoder skips. A feature pyramid supplies several resolutions to task heads. This chapter compares mechanisms and operation counts, not trained outputs or an architecture ranking.
Practice
Test the idea under another condition. Follow the task above the images; point inspection helps explain the result. This exploration is not scored.
Which path brought the detail back?
Start at α=0: the input and coarse output differ. Select a small detail in the image. Increase α gradually and compare that cell's contributions. At α=1 the detail returns through the direct path: explain why this does not show that upsampling reconstructed it.
- Input x16×16×1 · direct-path channel
- Concatenate channels16×16×2 · [upsampled, input]
- Keep the original map xInput x → Concatenate channels
Exact values are in the point inspector
Grayscale 0…1: black to white.
Exact values are in the point inspector
Grayscale 0…1: black to white.
Inspect the selected point
What the plot shows
Upsampling does not recover lost detail. A separate direct path retains it. Concatenation keeps both channels; the 1×1 convolution decides how much to use from each. Here you choose its weights; there is no training.
- Mean squared input error
- 0.0029
- 3×3 multiplies on the 8×8 grid
- 576
- 3×3 multiplies on the 16×16 grid
- 2304
- Channels after concatenation
- 2
- Coarse-path weight
- 1
- Direct-path weight
- 0
Parameters and details
One-level authored encoder–decoder with fixed weights, not U-Net or segmentation. Convolution repeats edges. Each coarse cell averages exactly one 2×2 block; upsampling repeats its position without shifting the grid. Concatenation has two scalar channels, viewed separately as the upsampled coarse path and input map, not as RGB. The final 1×1 layer mixes them with weights [1−α, α].
Select an image point. Arrows move it; Home and End reach row edges. Viewing does not change the calculation.
Changing the example requires JavaScript and Canvas. Computed images remain available above.
Games
Test your explanation
Question 1 of 5
Can 16×16×8 and 16×16×16 be added without alignment?
Exact values are in the point inspector
Grayscale 0…4: black to white.
Exact values are in the point inspector
Grayscale 0…1: black to white.
Exact values are in the point inspector
Grayscale 0…1: black to white.
Choose one answer. It is checked immediately.
Limitations and references
Limits of this example
These are exact teaching blocks, not inference of the named architectures. Parameter count alone does not measure device speed, memory or quality.