Taking Apart a Robot Benchmark

The task

Turn a manipulation video into a list of what happened and when.

WGO-Bench asks a model to produce segments. Each segment has a start time, an end time, and an event label such as "twist open the pitcher lid."

The payoff is scalable training signal: Macrodata estimates that one minute of video takes more than ten minutes to annotate.

Macrodata hand-annotated 100 episodes from three camera families: 25 HomER episodes recorded from a head-mounted camera in home settings, 50 DROID robot-arm episodes recorded by an external camera, and 25 Galaxea humanoid episodes recorded from the robot's head. The 100 episodes contain 743 human-labeled events. This post calls that reference timeline the gold.

Three annotation rules define what counts as one event:

1. One event is one completed manipulation, not a combined pick-move-place sequence.

2. A boundary sits at an object-state change: an object becomes held, is released, or reaches a new location.

3. A label names the object and target precisely enough to stand alone.

A model can describe the overall task correctly and still lose credit if it joins two completed manipulations or divides one at the wrong point.

The evaluated pipeline samples one frame every half second, tiles the frames into timestamped contact sheets, and sends those sheets to the evaluated model with a tuned prompt; the published baseline uses Gemini 3.5 Flash. This is one-pass annotation: the model generates segment intervals and labels together. It returns arbitrary timestamps, including times between sampled frames, and the parser turns that reply into the segment list shown alongside.

We evaluate the pipeline at three levels. Segmentation measures whether predicted intervals align with the gold timeline. Label accuracy measures whether temporally matched events are named correctly. Semantic F1 requires both to be correct and represents end-to-end annotation quality. We then supply the gold labels in a separate localization diagnostic to investigate why events fail to match in time.

The first three rows form a funnel: predicted intervals become temporal matches, correctly labeled temporal matches become semantic matches, and semantic matches are the usable end-to-end annotations. Localization is a controlled diagnostic rather than another stage of that pipeline.

Segment precision, recall, and F1 use outer-snapped intervals and one-to-one temporal matches at IoU ≥ 0.75. Outer snapping moves only the first predicted start and last predicted end to the episode edges; section 3 explains the full scorer. We first attempt to reproduce the published segment F1 over the full benchmark (0.306; 1.0 is perfect) and measure end-to-end annotation quality. We then audit how snapping, thresholding, and aggregation shape that result and derive three measurement interventions. Finally, we use a controlled localization diagnostic to investigate the most credible training target.

Reproducing end-to-end performance

We evaluate three models across three fresh stochastic runs on the full 100-episode, 743-event benchmark, using the same frame sampling, prompt, parser, and scorer. For each model, separate cache directories force fresh generations instead of replaying earlier responses.

The published result reports segment F1. We attempt to reproduce segment F1, then follow the reporting format used by Perceptron's separately reported egocentric-video system to add label and end-to-end semantic metrics. An LLM label judge sees each temporally matched gold and predicted label, plus the episode instruction, and decides whether they describe the same manipulation event. This label judge evaluates annotations; it does not generate or relabel them. A correct-label temporal match is a semantic true positive. Label accuracy is the share of temporal matches that become semantic true positives; semantic precision divides them by all predictions, and semantic recall divides them by all gold events.

Each result is pooled over the full benchmark within a run, then summarized across three fresh stochastic runs. Values are mean ± sample SD; with n=3, SD is descriptive rather than a confidence interval. Label-accuracy SD is in percentage points. Cost is estimated per video-hour from mean token use at the standard list prices for Gemini 3.5 Flash, GPT-5.5, and GPT-5.6 Sol, without prompt-cache discounts or label-judge overhead.

The interactive charts default to the event-pooled Overall result used by Perceptron's report; the exact-value tables show the family breakdown. Raw variants repeat the thresholded evaluation without changing the outer boundaries. Continuous variants keep the outer snap but replace the hard IoU cutoff with overlap-weighted credit, so partial boundary improvements receive partial credit. Section 3 gives the exact procedures and motivates both. The family dropdown exposes DROID, Galaxea, and HomER separately because HomER contributes 470 of 743 gold events, compared with 150 for DROID and 123 for Galaxea. The question to watch is whether end-to-end performance is lost mainly during temporal matching or label judging.

Segmentation metrics by family

Semantic metrics by family

Estimated annotation API cost

The published 0.306 segment F1 falls within the observed Flash run range of 0.2615–0.3105, although the three-run mean is lower. Each model used identical settings across its three runs, yet stochastic generation returned different answers. Sol has the highest observed mean across the Overall thresholded segment and semantic metrics; this three-run descriptive comparison does not establish a statistically significant ranking.

The individual events move across the three Flash runs too. A gold event is "matched" here when the scorer pairs it with a prediction at IoU ≥ 0.75; labels are ignored. Across those runs:

The larger result is consistent across models. Label accuracy reaches 68–76% once an event is temporally matched, while semantic recall is about 17–20% because most gold events never become temporal matches.

RQ1: The published segment F1 falls within the observed Flash run range, but our three-run mean is lower, the result varies across runs, and end-to-end semantic F1 reaches only 0.191 for Flash, 0.201 for GPT-5.5, and 0.225 for GPT-5.6 Sol. Across the Flash runs, 190 of 743 gold events match inconsistently and 455 never match. Section 3 audits how the scorer turns those interval changes into credit.

How end-to-end scoring works

The results form a two-stage funnel. The scorer first decides which predictions match gold events in time. The label judge then decides which temporal matches also describe the correct event. Macrodata published a prose specification but no scorer code, so the segment-matching procedure below is our reconstruction.

1. Snap the outer boundaries. Snap the first predicted start and the last predicted end to the gold episode edges. Macrodata's rationale is that outer boundaries are often ambiguous. Interior boundaries do not move.

2. Compute IoU, filter, sort, and pair one-to-one. For every gold-prediction pair, compute temporal intersection over union, or IoU: shared time divided by the total span covered by either interval. For segment F1, keep only pairs at IoU ≥ 0.75, sort them from highest to lowest, then lock pairs greedily. Each gold interval and prediction can be used at most once. Macrodata's prose does not specify greedy or optimal assignment; our reconstruction uses greedy pairing by IoU.

3. Create temporal matches. Every locked pair is a segment true positive. Every prediction and gold event outside those matches contributes a false positive or false negative, respectively. Pool those counts across episodes to compute segment precision, recall, and F1.

4. Judge labels on temporal matches. The label judge receives the gold label, predicted label, and episode instruction. A temporal match becomes a semantic true positive only when the labels describe the same manipulation event. Temporal misses never reach the label judge.

5. Compute semantic metrics. Label accuracy is the share of segment true positives whose labels are correct. Semantic precision uses all predictions as its denominator; semantic recall uses all gold events. Within each pooled run, semantic precision, recall, and F1 therefore equal their segment counterparts multiplied by label accuracy. This identity holds within a run; separately averaged table entries need not multiply exactly.

Tiny example. Three gold events, two predictions, and the same one-to-one pairs:

The example also separates label accuracy from end-to-end quality. The only temporal match is labeled correctly, so label accuracy is 100%, yet semantic F1 remains 0.40 because two gold events were never matched in time.

Three measurement interventions

1. Use continuous semantic F1 as a less threshold-sensitive end-to-end diagnostic. Segment F1 turns small boundary changes near the 0.75 IoU threshold into match-or-miss flips. Continuous segment F1 instead greedily locks every positive-IoU pair after outer snapping. Let S be the sum of those locked IoUs, Ng the number of gold events, and Np the number of predictions; continuous segment F1 is 2S / (Ng + Np). Continuous semantic F1 uses the same pairs but adds a pair's IoU to S only when its label is correct. For a fixed pairing, moving a correctly labeled boundary closer receives additional credit before it crosses the benchmark threshold.

Continuous segment F1 is less threshold-sensitive in the Overall results but compresses the three model means to 0.502–0.507, making it less informative for model selection. Continuous semantic F1 retains more separation: its means span 0.283–0.339, compared with 0.191–0.225 for semantic F1. Its average run SD across models also falls modestly from 0.0168 to 0.0150, although Sol is a counterexample at 0.021 versus 0.020. Descriptively, the span between model means relative to average SD rises from 2.0 to 3.7. Raw semantic F1 moves the other way: its means compress to 0.128–0.135.

These are three-run descriptive results, not a general variance proof. Consider continuous semantic F1 as the less threshold-sensitive end-to-end development diagnostic with modestly lower average run variance here; report continuous segment F1 alongside it to show how much of the signal comes from temporal overlap. Keep the thresholded segment and semantic metrics for benchmark comparison.

2. Report raw variants beside snapped variants. Raw segment F1 applies the same IoU ≥ 0.75 matching without outer snapping. Raw semantic F1 judges the labels on those raw temporal matches. The snap changes only two timestamps, so it can affect a large share of DROID's two- or three-event episodes and a much smaller share of dense HomER episodes.

Section 2 reports both raw metrics directly beside their snapped counterparts for every model and family. In the Overall result, Flash segment F1 is 0.283 ± 0.025, versus raw segment F1 of 0.211 ± 0.023. DROID changes from 0.306 ± 0.040 to 0.118 ± 0.038, while HomER changes from 0.226 ± 0.036 to 0.221 ± 0.039. Raw semantic F1 is useful for exposing snap dependence, but its Overall model means are too compressed to recommend as the primary development signal. Report raw segment and semantic F1 beside their snapped versions.

3. Report the Overall result alongside every family. Section 2 follows Perceptron's event-pooled aggregation for the Overall result. HomER contributes 470 of 743 gold events, or 63% of the gold-event denominator, making it the largest contributor to that denominator and a strong influence on the pooled result. The family dropdown makes distribution-specific differences visible without introducing a second competing overall score.

RQ2: End-to-end scoring first creates temporal matches, then tests their labels. The scorer audit yields three interventions for a more interpretable report: consider continuous semantic F1 as a less threshold-sensitive development diagnostic with modestly lower average run variance here, report raw variants beside snapped variants, and pair the Overall result with family diagnostics. Section 4 removes label generation to test whether localization explains the segmentation weakness exposed by those metrics.

Isolating localization from labeling

Label accuracy reaches 68–76% after an event is temporally matched, while semantic recall is about 17–20% because most gold events never become temporal matches. Open-ended segmentation predicts the set of intervals; the pipeline generates labels jointly, but segment metrics ignore them. The localization diagnostic instead supplies each gold label and tests only whether the model can localize it.

Gold-label localization

The diagnostic supplies timestamped contact sheets and every gold-label occurrence in the full 100-episode benchmark, while hiding the gold intervals. The model is asked to return one interval for each supplied occurrence. Within each exact-label group, predictions are assigned one-to-one to gold occurrences to maximize summed IoU.

Localization hit rate is the share of supplied occurrences assigned an interval at IoU ≥ 0.75. Mean per-event IoU averages assigned IoU over every supplied occurrence. No outer snap is applied; unfilled or unparseable occurrences receive zero, and extra predictions are ignored.

Each model ran the same 743-occurrence diagnostic once. GPT-5.5 and GPT-5.6 Sol ran through Codex server; Flash used its standard API runner.

All three single diagnostic runs show weak localization: localization hit rate ranges from 10.0% to 13.5%, and mean per-event IoU ranges from 0.29 to 0.34.

Does count information help?

Because supplying one label per occurrence also reveals the event count, we separately test whether count information alone improves open-ended segmentation. Over the full 100-episode benchmark, three gold-count prompt ablation runs score 0.273 ± 0.016 in segment F1, compared with 0.283 ± 0.025 across three no-count runs (appendix A). The model follows the requested count in 297 of 300 episode-runs, but segment F1 does not improve in these three-run descriptive results.

RQ3: Supplying gold labels still leaves localization at 10.0–13.5% at IoU ≥ 0.75, making localization the clearest observed weakness in this contact-sheet pipeline. Separately, the gold-count prompt ablation does not improve open-ended segmentation. Part 5 considers how to train localization directly.

Where future work should focus

The measurement takeaway comes first: use the three interventions to make the evaluation more interpretable, then use that clearer diagnosis to choose the training target.

Report the full funnel

Future evaluations should report segment precision, recall, and F1; label accuracy; semantic precision, recall, and F1; and cost per video-hour. Localization hit rate and mean per-event IoU diagnose localization directly. Continuous semantic F1 preserves more model separation with modestly lower average run SD in these Overall results, while continuous segment F1 alone compresses the model means. Raw variants expose snap dependence, and family diagnostics expose distribution-specific exceptions.

Together, the three interventions serve different roles: continuous semantic F1 reduces threshold sensitivity, raw variants reveal dependence on outer snapping, and family results reveal what the Overall result can hide.

Train localization directly

Localization is the clearest capability gap isolated in this contact-sheet pipeline. When supplied every gold label, the three models localize only 10.0–13.5% of occurrences at IoU ≥ 0.75. This direct evidence motivates localization-focused training. The separate gold-count prompt ablation adds narrower evidence that supplying event count alone does not repair open-ended segmentation.

Supporting evidence: Perceptron

Perceptron's reported WGO-Bench result reports higher segment metrics for a system built for egocentric video. Perceptron describes its system and WGO's one-pass variant as one-pass annotation. WGO's separate seeded relabeling variant adds a second, per-segment Gemini labeling pass; it is not included in this table. The table compares Perceptron with the three one-pass baselines evaluated in this post. Our rows report mean ± sample SD across three fresh stochastic runs; Perceptron is an external reported result, so no run-to-run SD is available. We did not independently reproduce its pipeline or verify scorer, label judge, input, or cost-accounting equivalence.

This external comparison is limited to the metrics Perceptron reports. Its family results and raw or continuous variants are unavailable, so this table is not a substitute for the intervention-compliant report in section 2.

Perceptron reports higher segment and semantic precision, recall, and F1 than all three reproduced baselines. Its 75.7% label accuracy falls inside the reproduced range of model means, 67.6–76.2%, and is nearly identical to Sol's 76.2%, while its segment F1 reaches 0.370 instead of 0.283–0.295 across the reproduced model means. Numerically, most of the reported difference appears before the label judge, although unverified pipeline and scoring equivalence prevents causal attribution.

Localization supervision should require models to assign an interval to each supplied gold-label occurrence. Training targets can reward accurate starts, ends, and repeated occurrences directly. The released localization dataset provides a starting point for building that supervision.

Preserve temporal information

Contact sheets compress motion into sparse still frames. Native video, temporal encoders, and denser frame representations may expose the object-state changes that define event boundaries. Their value should be measured through localization on the same gold timeline.

Use the three measurement interventions to make evaluation more interpretable, and prioritize localization training because the gold-label diagnostic isolates localization as the clearest observed weakness. The gold-count prompt ablation provides narrower evidence that event count alone does not repair open-ended segmentation.

Leave questions or comments at haoliyin2002@gmail.com.

Appendix: evidence and limits

Additional evidence and the limits on the localization conclusion.

Family exception. DROID has the weakest localization under all three evaluated models even though its segment F1 benefited from the outer snap. On its 150 occurrences, Flash reaches a 1.3% localization hit rate and mean per-event IoU 0.174; GPT-5.5 reaches 2.0% and 0.208; GPT-5.6 Sol reaches 8.0% and 0.317.

Interactive inspector. Explore synchronized gold and predicted timelines in the WGO-Bench inspector.

A. Supplying the gold count: the model followed the requested count in 297 of 300 episode-runs, but segment F1 did not improve in the three-run descriptive results.

We added one prompt sentence: "This episode contains exactly N events." The gold-count and no-count conditions each ran three times over the full 100-episode benchmark. Values are mean ± sample SD in segment F1.

B. Supporting temporal diagnostics

For each of section 2's 455 never-matched events, we take its best IoU with any outer-snapped prediction across the three Flash runs, ignoring pairing and labels. Of these, 217 reach [0.50, 0.75) and 238 reach [0, 0.50). This best-over-runs IoU distribution shows related boundary failures without identifying their cause.

In a descriptive two-line fit split at ten gold events, predicted count has slope 1.01 through ten and 0.59 above. This count compression is consistent with missing internal boundaries in dense episodes, but does not prove the cause.

C. What this analysis does not establish

No human ceiling. We did not measure inter-annotator agreement. Every granularity claim is scoped to this gold, not a uniquely correct gold timeline.

No conclusion about native video. Every model run used contact sheets. Native-video input may localize better.

D. Methods and reproducibility

Segment scoring. Section 3 defines the scorer reconstructed from Macrodata's prose specification. Their implementation was unavailable. Predicted outer boundaries are snapped to episode edges; pairs at IoU ≥ 0.75 are then locked greedily to create temporal matches. Continuous metrics instead lock all positive-IoU pairs greedily after outer snapping. Raw metrics use the thresholded procedure without outer snapping.

Label judging. Semantic F1 judges thresholded outer-snapped temporal matches; raw semantic F1 judges thresholded raw matches; continuous semantic F1 judges all positive-IoU continuous pairs. The judge receives the gold label, predicted label, and episode instruction. It accepts synonymous descriptions of the same manipulation event and rejects a wrong action, object, source, destination, direction, or event. Timing is excluded from this judgment because temporal overlap supplies the separate temporal credit.

Localization assignment. The localization diagnostic covers all 100 episodes and 743 gold occurrences using raw intervals without outer snapping. Gold and predicted intervals compete only within exact-label groups. Repeated occurrences are assigned one-to-one to maximize summed IoU. Every unfilled gold occurrence receives zero; extra predictions are ignored because the diagnostic measures placement of requested events rather than open-ended segmentation. GPT-5.5 and GPT-5.6 Sol use structured outputs through Codex server; Flash uses the standard API runner. All three are scored with the same assignment code.

Aggregation and cost. Main reproduction metrics pool counts across all episodes within each run, then report the mean and sample SD across three runs. API cost per video-hour uses mean token consumption at standard list prices without prompt-cache discounts. The label judge cost is evaluation overhead and is excluded from annotation cost.

Data and replay. Videos and annotations come from Macrodata Labs' WGO-Bench under CC-BY-NC-SA-4.0. Released derivatives use the same license. Model requests and responses are cached, so existing analyses can run without new API calls.