M05 · Mathematics alongside
Optimization and inverse problems
One slider changes the tradeoff between matching measurements and preferring regularity; it does not reveal hidden truth without assumptions.
Mathematical support for the textbook chapters: start with a small calculation, then an interactive example and precise definitions.
Work through a small example
An observation of 0.8 competes with a preference for values near zero. Where is the equal-weight compromise?
- Cost of a chosen x
- The slope is zero at the minimum
- Two cost terms
A regularizer changes the solution, not merely computation speed. It can suppress noise and introduce bias. In an image, data agreement and structural preference act on many connected pixels.
Declare the measurement and objective first
Observation connects unknown original x with measurement operator A. Blur, missing samples and projections give different A. If several x produce the same y, data fit alone is insufficient. Regularization selects a preference among plausible alternatives; it does not add a measurement.
In , the first term penalizes mismatch and the second an undesired property. λ depends on data scale and noise. A constraint 0≤x≤1 reflects a declared model, but clipping after arbitrary steps does not solve every constrained problem.
Keep the corrupted measurement fixed and vary regularization. Weak regularization tries to recover detail while amplifying interference; strong regularization favors smoother output. This illustrates an inverse-problem tradeoff. The next section's derivative and optimizer update are a separate solution mechanism.
The restoration trade-off
The observation uses a known horizontal blur. Lower λ and see which frequencies get amplified. Add interference: the multiplier cannot distinguish it from useful signal. Compare restoration with the known reference.
Exact values are in the point inspector
Grayscale 0…1: black to white.
Exact values are in the point inspector
Grayscale 0…1: black to white.
Inspect the selected point
What the plot shows
Where blur nearly erased signal, division also amplifies interference. λ bounds amplification; H=0 means absent information for every λ.
- Blur transfer H
- Gain H/(H²+λ)
Weak regularization amplifies unstable frequencies; strong regularization suppresses detail. A zero transfer frequency remains unrecoverable even without interference.
- MSE
- 0.0005
- RMSE
- 0.023
- PSNR (range 0–1)
- 32.757 dB
Parameters and details
Periodic horizontal blur [1,2,1]/4; frequencies use H/(H²+λ). The zero transfer frequency at Nyquist cannot be recovered. This is Tikhonov, not blind deconvolution or Richardson–Lucy.
Select an image point. Arrows move it; Home and End reach row edges. Viewing does not change the calculation.
Changing the example requires JavaScript and Canvas. Computed images remain available above.
A gradient gives a local direction
Definitions and details
A gradient step opposes local increase. For quadratic the gradient is 2(2w−1) and the minimum w=0.5. Excessive steps can increase loss. For smooth convex F with L-Lipschitz gradient, a standard step gives familiar descent and objective-convergence guarantees when a minimum exists; rates depend on further properties.
A deep network usually defines a nonconvex objective. Training-loss decrease guarantees neither a global minimum, transfer to new data nor physically accurate restoration. These require separate evaluation.
A nonsmooth objective needs no pretend gradient
Definitions and details
Absolute value and TV are nonsmooth at zero. A proximal step solves . For proper closed convex R and η>0, this Euclidean problem has a unique minimum. Projection onto a nonempty closed convex set is a special case; a nonconvex set can have several nearest points.
Forward–backward splitting separates a smooth term and a proximal term. Step-size, convexity and subproblem-accuracy conditions belong to a specific algorithm. Replacing prox with an arbitrary learned denoiser defines another procedure to which the old guarantee does not automatically apply.
Check a result through the observation model
Definitions and details
After restoration, compute A x̂ and compare with y: this is a measurement residual, not error against an unknown original. A small residual can coexist with an incorrect x̂ when A loses information. Large λ can favor excessive smoothness; small λ can fit noise.
Objective value, residual, iterate change and stopping rule convey different things. Small change between steps can mean convergence or merely a tiny step size. Keep parameters and actual iteration count with the result.