Active Learning Needs Orthogonal Axes
Most active-learning systems start by asking which samples to label next. The question sounds practical, but it quietly assumes that every model failure is a labeling problem. It isn't.
Some failures indicate bad labels. Some indicate missing coverage. Some reveal decision-boundary uncertainty, or domain shift, or a model, threshold, resolution, or postprocessing limitation. When all of these signals are compressed into one global score, a high score becomes impossible to act on. It might mean “review this label,” “collect more data like this,” “tune the detector,” “investigate a new domain,” or “ignore this irrelevant out-of-distribution sample.” These are different actions, with different costs and different success criteria.
The problem with one global score
A common active-sampling score blends many useful signals — model uncertainty, similarity to known failures, domain gap, class rarity, slice rarity, duplicate avoidance, expected labeling value. This produces a reasonable ranking, but it conflates decisions that should remain separate.
A high-confidence unmatched prediction on a labeled image may look like a false positive — but it may be a missing ground-truth label. The right action there is not to collect new data; it is to audit the annotation. A rare target slice with high failure rates is a coverage problem. A well-covered slice that still fails because resolution is too low or NMS is suppressing the right boxes is not a data-acquisition problem at all. And a very novel image that is irrelevant to the product domain only wastes budget if collected. So the core separation is:
selection policy — how much budget to assign to each intervention
diversity constraints — how to avoid redundant selections within an intervention
These three things should not be collapsed into one number.
Three candidate universes
Before defining the axes, separate the universes a candidate can come from. Mixing them is one of the biggest causes of wasted annotation effort.
| Universe | Possible actions | It is about |
|---|---|---|
| Labeled pool existing train / val annotations |
audit labels, correct classes, fix boxes, add missing labels, remove invalid annotations | label integrity |
| Unlabeled pool images / frames without trusted labels |
select for annotation, prioritize for review, search for a target slice | data acquisition |
| Model or run thresholds, NMS, architecture, resolution, tiling, calibration, loss |
run an experiment, tune postprocessing, change training config, evaluate architecture limits | model & pipeline improvement |
Axis A — Label integrity
Is an existing annotation wrong or incomplete? (labeled pool)
This axis is not asking whether the model failed. It asks whether the dataset itself contains an annotation problem. Typical signs:
- high-confidence prediction with no matching ground truth;
- high-overlap prediction with a different class than the annotation;
- systematic localization disagreement;
- duplicate annotations, or missing labels in dense scenes;
- disagreement between annotation sources.
Model disagreement is only a suspicion, not proof. A detector can be wrong; a label can be wrong; sometimes both are partially wrong. The output should be a label-audit queue, not an automatic mutation of the dataset.
Success: reviewer acceptance rate, corrected annotations per review hour, inter-reviewer agreement, reduction in annotation-induced errors after correction.
For how this works in practice — turning TIDE error types into a ranked label-audit queue — see the deep dive, When the Detector Disagrees With the Label.
Axis B — Target-coverage scarcity
Is an important target-domain slice underrepresented in the labeled training pool?
This is not class imbalance. A dataset may hold many examples of a class yet poorly cover the conditions that matter: tiny or distant objects, unusual viewpoints, thermal low-contrast targets, crowded scenes, specific backgrounds, rare sensors, night or haze, high-mounted camera views.
A failure becomes a coverage problem only when the failing slice has too few independent examples. Ten thousand adjacent frames from one camera are less useful than a hundred examples from different sessions, locations, or conditions. The axis must distinguish underrepresented-and-failing from well-represented-but-failing, from rare-but-handled-correctly, from irrelevant outliers — only the first is a clean acquisition target.
Success: coverage increase on the target slice, and improved performance on frozen target evaluation slices.
Axis C — Decision-boundary information
Which unlabeled samples would most reduce uncertainty in the decision boundary?
The classic active-learning axis — handled carefully. Low confidence does not mean useful: a sample may be empty, corrupt, irrelevant, out of domain, or impossible to label consistently. Useful boundary samples expose distinctions the model can realistically learn: confused look-alike classes, borderline object/background cases, hard negatives, uncertain scale or truncation, samples where checkpoints or ensembles disagree. Separate three situations:
- Learnable uncertainty — more labels can improve the boundary;
- Aleatoric ambiguity — the image lacks the information to decide;
- Ontology ambiguity — the class definition or guideline is unclear.
Only the first is a clean acquisition target.
Success: improved calibration, reduced disagreement, performance gain per annotation.
Axis D — Distribution shift and novelty
Is the candidate meaningfully different from the labeled pool — and is that difference relevant to the target domain?
Novelty by itself is not valuable. A detector for aerial surveillance may find underwater scenes highly novel, but they should not be labeled. This axis combines novelty + target relevance. The useful cases are target-relevant shifts: a new camera source, sensor modality, geographic environment, weather or illumination, background type, capture altitude, or operational domain.
The action depends on relevance: if relevant, create a new domain slice and collect more; if unknown, investigate product relevance first; if irrelevant, reject it from acquisition.
Success: better target-domain coverage, improved performance on newly established slices, accurate rejection of irrelevant out-of-distribution data.
Axis E — Model and pipeline limitation
Does the evidence suggest that changing labels or acquiring more data is unlikely to solve the problem?
This is the routing axis that prevents every detector failure from becoming an annotation request. Some failures are the model or pipeline: threshold too high, poor calibration, NMS suppressing valid detections, duplicate detections, resolution too low for small objects, tiling missing tiny targets, a localization loss that does not match evaluation, an architecture unsuited to the object scale, temporal instability, or a regression from a new model version.
If many correct objects are detected at low confidence, suspect calibration or thresholding. If tiny objects are missed despite dense coverage, suspect resolution, tiling, or feature stride. If duplicates dominate, suspect NMS or postprocessing. The output is an experiment recommendation, not a collection brief.
Success: improvement on frozen slices, no unacceptable regression elsewhere, acceptable cost and latency impact.
Why the axes must stay independent
The same observation can activate several axes. A high-confidence unmatched prediction could mean a missing label, a false positive, an ontology ambiguity, a domain-shift issue, or a calibration issue. A failure slice with few similar training examples may be coverage scarcity — but if it is also novel and target-relevant, it activates the novelty axis; if it is well represented yet still failing, it routes to model limitation. This is expected. The axes are not mutually exclusive labels; they are independent questions, each with its own evidence, actions, and success metric.
A latent cluster is not an acquisition instruction
“Cluster 17 is failing” is useful for computation, not for collection. A collection team or dataset owner needs a semantic description that is predictive and collectable — what objects are involved, what visual and capture conditions make the slice different, what failure pattern appears, and how it differs from nearby successful examples.
| Slice description | |
|---|---|
| Usable | Tiny vehicles at long range on low-contrast thermal backgrounds, mostly from elevated static cameras, with frequent missed detections below the confidence threshold. |
| Not usable | Cluster 17 has high failure similarity and low train density. |
The semantic layer does not replace the quantitative axes; it translates their evidence into a human-usable description. “Hard images,” “unusual scenes,” or “bad examples” are not enough.
From diagnosis to budget allocation
Once the axes are separated, selection becomes two stages. First, each axis produces its own queue — label-audit, coverage-acquisition, uncertainty-annotation, novelty-investigation, model-experiment — each with its own evidence, eligibility gates, and success metrics. Then a planner allocates budget across them explicitly. For example:
| Queue | Example budget |
|---|---|
| Label audit | 30% |
| Target-coverage acquisition | 30% |
| Decision-boundary annotation | 20% |
| Novelty investigation | 10% |
| Model / pipeline experiments | 10% |
The exact split depends on product priorities, cost, expected gain, and previous results. The point is that the budget decision is explicit — not hidden inside a global weighted score.
Diversity is a constraint, not an axis
Diversity and duplicate avoidance are essential, but they are not active-learning objectives by themselves. They are constraints applied within every selected batch: avoid too many adjacent frames, cap samples per video or session, limit examples from the same camera, suppress near duplicates, prefer independent evidence, balance classes and sources. Diversity makes each queue efficient, but it is not the reason a sample was selected.
Human validation stays central
Automated systems can propose; they should not silently rewrite the truth. A detector disagreement is not proof of a label error, and neither is a vision-language model description — both are evidence that helps prioritize review. Every selected sample should carry its provenance: which axis selected it, what evidence supported it, what action was recommended, whether a human accepted or rejected it, and whether the action improved the relevant slice later. Without that, data improvement cannot be audited.
The core idea
Ask first: “What kind of problem are we trying to solve?”
A detection failure may need label repair, or new data, or boundary examples, or domain investigation, or model tuning — or no action at all. Orthogonal axes make that decision explicit, turning active learning from a single ranking problem into a structured data-improvement loop: observe the failure, separate possible causes, route to the right intervention, allocate budget explicitly, validate the outcome, and track improvement on frozen slices.
That is the difference between collecting more data and improving the dataset — and in real-world detection, that difference is most of the work.