Trust, limitations and model inspection

A right answer for the wrong reason

If training images of a class often contain a hospital mark, studio background or file-format cue, a model may use it instead of the intended feature. This is a shortcut. Changing camera, lighting or data source creates domain shift: the earlier statistical association may disappear.

The declared rule below depends on object mean and corner-cue brightness. It is not trained: its complete mechanism is deliberately visible. Holding the object fixed and changing the cue gives a controlled dependency test rather than an impression from a colorful heatmap.

Intervention is useful but limited

Occlusion covers a patch and recomputes the result. A large score change makes that patch important under this intervention. However, the fill creates a new image, and a model may react to the artificial patch itself. Gradient saliency measures local sensitivity; feature visualization seeks an input that increases a feature: these answer different questions.

A structural receptive field says where information may arrive from. Attention describes one mixture. Neither alone guarantees causal interpretation or class correctness. Several consistent tests, counterexamples and baseline comparisons are needed.

A new set tests transfer; temperature changes confidence

One counterexample reveals dependence; evaluation needs a series of independently labeled examples. Here 40 pairs use object brightness above 0.5 for class 1 and below 0.5 for class 0. A corner mark is irrelevant information. It helps on the first set and hurts on the second. The evaluation set changes while the selected rule stays fixed.

Accuracy is the fraction of correct class choices. Confidence is the number attached to each choice. Calibration groups similarly confident predictions and compares confidence with the group's correctness frequency. The diagonal means agreement. Temperature divides the logit before sigmoid and preserves its sign, so it cannot repair wrong classes.

Confidence, data and reproducibility

Softmax or sigmoid returns a number in 0…1, but this is not yet a validated probability. Calibration compares stated confidence with correctness frequencies on independent data. Data ambiguity and model uncertainty are different sources. An unknown original does not permit an honest reference PSNR against that original.

Split by independent people, scenes or time when that matches deployment. Duplicates and nearly identical neighboring frames across train and test create leakage. Record weight version, preprocessing, thresholds, data source and subgroup metrics. An ablation changes one component to test its contribution; a high aggregate score does not remove local failures.

Quantization reduces numerical precision and can reduce size/cost, but needs renewed quality tests and actual memory/latency measurements. Local processing avoids server upload only if the code actually sends no file. Neither watermarking nor synthetic-image detection universally proves the origin of every image.