Automatic Labels Need an Abstain Button

The appealing version of auto-labeling is simple: run a strong model, keep its boxes, and replace weeks of annotation with a GPU job. The dangerous version is almost identical. It replaces weeks of annotation with thousands of plausible, systematic mistakes.

I built Lynceus to explore that distinction. It is a local, offline object-annotation pipeline with versioned ontologies, immutable evidence, explicit uncertainty and evaluation that refuses to turn a small benchmark into a qualification claim. This is what mattered, how the system works, and what the experiments actually showed.

Why auto-labeling matters

Data is still the slow interface in a computer-vision loop. Training an experiment may take hours; acquiring, boxing, naming and auditing the next dataset can take weeks. Dense scenes and tiny objects make the gap worse: a single aerial frame can contain hundreds of instances, and a small box shifted by one pixel may cross the evaluation threshold.

Automation changes the economics. A machine proposal gives a reviewer something to accept, correct or reject instead of asking them to search every pixel from scratch. It also makes labeling reproducible: the same ontology, thresholds and model bundle can be rerun, compared and audited. That matters for active learning, dataset refreshes and data that must remain offline.

But annotation errors are not ordinary inference errors. They become training data. A missed class becomes a false negative for the next model; a confidently wrong fine-grained name teaches the confusion; an oversized box changes localization behavior. Human mistakes are often local. Automated mistakes repeat at scale.

How Lynceus approaches it

Discovery proposes. Refinement changes geometry. Verification adds evidence. Policy—not one model score—decides what can be accepted.

1. Search at the scale of the object

OWLv2 runs on the full image and overlapping source-resolution tiles. Shrinking an aerial frame can turn a twelve-pixel person into a few pixels, so tiling changes what reaches the detector. The grid follows the image aspect ratio instead of wasting almost half a 16:9 tile on square padding.

Repeated views create repeated boxes. IoU merging misses boxes cut by a tile edge; intersection-over-smaller merges those, but also collapses neighbors in crowds. Lynceus uses the permissive rule only when an observation touches a view boundary. Losing observations remain attached as alternatives instead of disappearing.

2. Prefer a defensible broad name to a brittle specific one

Classes come from a versioned ontology. If two vehicle labels compete, the system can emit their supported parent, vehicle, instead of guessing. Specificity can be added during review; a confident wrong class quietly poisons training.

3. Separate finding, geometry and evidence

OWLv2 finds candidates. SAM 3 receives each retained box as a prompt and tightens it against source pixels. A separate SAM 3 detection pass acts as verifier. An unconfirmed box is not deleted; it receives an explicit reason and stays available for review. “Another model did not reproduce it” is evidence, not proof of absence.

4. Make uncertainty part of the output contract

Existence, class, boundary, count and granularity are assessed separately. A proposal becomes accepted only when every required check passed. Runs record the ontology, bundles, candidate evidence, coverage, thresholds and hashes. Resume creates a new immutable run rather than editing history.

What the measurements showed

The main development work used two frozen ten-image VisDrone panels. That can compare mechanisms; it cannot certify an annotator. The useful result was learning which stages earned their cost.

One image, seven operating points

Here is one held-out frame. The OWLv2 and verifier panels use the same geometry and candidate set; only their selection rule changes. Opus 5.5 is a separate detector using the whole frame plus six upscaled tiles, so it is a comparison rather than an ablation. Green boxes match an eligible VisDrone reference at IoU 0.5. Red boxes are unmatched: some may be false positives, but some are real objects outside VisDrone's finite label scope. Click any panel for the full-resolution render.

Frame 0000346 from the held-out VisDrone panel. Verifier geometry is fixed; Opus is an independent size-routed detection pass. Numbers are descriptive and unqualified.

The recommended configuration combines one tiling level, a task-specific co-occurrence ontology, the baseline prompt vocabulary, SAM 3 boundary refinement and optional verification. On held-out, refinement plus the ontology produced 374 IoU 0.75 matches versus 358 for the shipped baseline, while eliminating 166 unknown labels. Verification retained 97.9% of known positives while nearly doubling the measured precision bound.

A verifier threshold targeting 90% sensitivity was fitted on development, then frozen. It transferred to held-out at 90.2%, with 0.785 recall and 0.568 measured precision, then ran unchanged on new datasets:

The flood result exposed a convention mismatch: its boxes include splash, wake and shadow, while Lynceus boxes the person. Under a containment rule suited to that convention, recall is 0.668 and precision 0.707. A metric can measure labeling policy as much as model quality.

Construction equipment exposed the naming trade-off. Conservative naming made no wrong class claims among matched boxes, but fell back to construction_equipment 97% of the time. That is safe and not specific—useful for proposals, and evidence that fine-grained labels still need review or a stronger domain classifier.

Update: what is a model trained on these labels worth?

Every precision number above is a lower bound, because an unmatched box is either a mistake or a real object VisDrone never labeled, and four verifiers could not tell those apart. A model trained on the labels does not need to. A real-but-unlabeled object helps it; a false box hurts it. So the most direct test of an auto-labeler is to train on its output.

I froze the protocol before producing a single label: 200 VisDrone training images, with every sequence that also appears in validation excluded. Lynceus labeled them. The same small detector (YOLO11n) was then trained once on each label set and once on the human labels for the same images, and every model was scored on all 548 validation images against human labels. Two seeds each, so a difference smaller than the seed spread counts as no difference.

The best automatic labels reach 91% of the human-label model, in line with the 90–95% the auto-labeling vendors report on simpler data. Three details matter more than the headline:

Verification pays for itself downstream. Keeping only SAM 3-confirmed boxes moves the trained model from 84% to 90% of human, about fifteen times the seed spread. The earlier precision gain was a bound; this one is not.

Stricter is not better. Raising the verifier threshold from 0.392 to 0.461 improves the label-precision bound and makes the trained model worse. Discarding borderline boxes also discards real objects the model needed to see. A threshold should be chosen by what it does to the consumer, not by how clean the label file looks.

The abstain button needs a destination. This is the uncomfortable result for a post with this title. Conservative naming emits vehicle for most cars it cannot pin down. If those boxes are simply removed from the training set, the student never learns cars and falls to 44% of human. Abstention is cheap in a report and ruinous in a training set. An uncertain label must be resolved, reviewed or masked—never silently turned into background.

How it is resolved matters too. Calling every vehicle a car helps cars and wrecks trucks and buses. Using SAM 3's own label keeps that distinction. Best of all was retrieval: compare the crop with a bank of human-labeled crops from other training sequences and take the class of its nearest neighbors. Against the exact class it looks reckless, wrong half the time because it never abstains. But most of those “errors” are car versus van, which the downstream schema merges anyway. In the classes the consumer actually receives, it resolves 55% of the ambiguous boxes correctly versus SAM 3's 36%. A naming metric and a consumer metric can disagree, and only the second one trains a model.

The limits are the same as before. This is 200 images, one architecture and two seeds on aerial data. The student is scored against VisDrone's three-class convention, which ignores bicycles and motorbikes, so it measures usefulness to that consumer rather than truth. It shows a direction, not a qualification.

What did not become true

Model agreement did not prove correctness. SAM 3, Grounding DINO and a class-agnostic WeDetect verifier looked architecturally different, yet their confirmed known positives were almost subsets of SAM 3's. More models added cost and stricter operating points, not independent truth.

No search schedule can certify that an arbitrary image has no missed object. VisDrone annotates road users, not every tree, building or piece of street furniture, so unmatched predictions are not known negatives. Precision is a lower bound of unknown tightness. Training a model on the labels prices them for one consumer, but it does not certify them. An exhaustive, independently annotated panel is still required for qualification.

The practical lesson

Auto-labeling becomes valuable before it becomes autonomous. It does not need to replace the annotator to save time; it needs to move attention from exhaustive search toward ambiguous cases while keeping its own uncertainty inspectable.

Optimize the whole loop, not only the detector: search coverage, ontology, boundaries, verification, evidence and review. Measure each proposed trick on a frozen panel. Keep negative results. And make abstention first-class, because the most expensive label is not the one a model declined to make—it is the wrong one nobody knew to check.

Code, experiment scripts, frozen mappings and the full delivery record are in the Lynceus repository. The reported panels are small and mostly aerial; the repository deliberately describes the system as unqualified pending an exhaustive independent reference panel.