RoboRACE

Byzantine-robust multi-agent robotics: when some of your robots, cameras or AI agents are wrong on purpose. Two public benchmarks (ERQA, PlanBench) and five RoboRACE tracks, run on a CPU with real VLMs, real LLMs, two simulated robot arms and a maze swarm. Every number below is generated from cached agent outputs in the dataset repository.

Dataset · full workGitHub Leaderboard Protocol Interactive playgroundRACE-Net-V model RACE-Net-M model
SO-101 arm (MuJoCo Menagerie) rollouts judged by five Cosmos-Reason2-2B camera agents; then 3 of 5 cameras lie. Majority vote flips; RACE-S learns negative weights for the liars without labels and stays correct.
ERQA reproduction
33.8%
Cosmos-Reason2-2B on CPU · NVIDIA reports 37.75% · inside our 95% CI (29.3–38.5)
Track A · SO-101 camera swarm
83.4%
race_net (new, learned) · rank 1 · liar-majority 80.0% vs majority vote 46.7%
Track B · Byzantine rendezvous
94.3%
RACE-C + W-MSR · rank 1 · W-MSR 82.1%
Track C · LLM planners
98.2%
engineered (spec + validator + repair) · zero-shot 88.1%
Track A-Panda · new robot, frozen methods
87.0%
race_s_online (new) · majority vote 62.2%
Track D · maze swarm · RACE-Net-M
70.2
SPL×100 · rank 1 of 10 fusion rules · hand-written RACE map 66.2 · best literature baseline 64.1 · oracle 71.3
RACE-Net-V · our learned verification model
83.4%
0.2 M params, trained on synthetic swarms only · zero-shot on SO-101 · Panda 84.2% · on the hold-out 79.0 / 84.3, behind our EM methods
Hold-out · new episodes, nothing tuned on it
82.6 / 86.7%
SO-101: RACE-S online rank 1 · Panda: RACE-Pool rank 1 · majority vote 62.3 / 64.2%

The idea

Majority vote and classical truth inference (Dawid–Skene, GLAD, MACE) fail once liars are the majority, because the latent-truth model becomes symmetric. RACE (from the Liars Are Information program) anchors on the receiver's own honesty and gives every peer a signed channel weight: trust, invert or discard. This work adds RACE-S (score level), online variants against sleepers, RACE-C + W-MSR for robot consensus, confidence-gated anchoring (reported as a compromise, not a free lunch), and an engineered / LLM-Modulo planning layer.

Best method per family on the three RoboRACE tracks.
Best method per family on the three RoboRACE tracks.

Architecture

Cosmos-Reason2-2B (Qwen3-VL) as the success verifier — numbers from the real checkpoint (2.44 B params).
Cosmos-Reason2-2B (Qwen3-VL) as the success verifier — numbers from the real checkpoint (2.44 B params).
RACE-S: anchored EM over rank-Gaussianized scores with signed weights.
RACE-S: anchored EM over rank-Gaussianized scores with signed weights.
The three agent swarms.
The three agent swarms.
PlotNeuralNet view of the verifier: one 448×448 view → 196 image tokens → P(yes).
PlotNeuralNet view of the verifier: one 448×448 view → 196 image tokens → P(yes).

Public benchmark · ERQA

Cosmos-Reason2-2B reproduced on 4 CPU cores: 33.8% (95% CI 29.3–38.5, n = 400), reported 37.75%. At ≤768² px: 35.5% (19 of 400 answers change). Swarm of 4 VLM agents on 283 shared questions:

systemaccuracy95% CIreweighted
⊕ Dawid-Skene (crowd-kit)54.848.9–60.554.5
Qwen3.8-27B @Cerebras54.148.2–59.853.7
⊕ RACE · receiver = Gemini-3.1-flash-lite53.047.2–58.752.9
⊕ RACE · receiver = Gemini-3.5-flash-lite53.047.2–58.752.9
⊕ RACE · receiver = Qwen3.8-27B @Cerebras53.047.2–58.752.9
⊕ RACE sign-only anchor · receiver = Gemini-3.1-flash-lite53.047.2–58.752.9
⊕ RACE sign-only anchor · receiver = Gemini-3.5-flash-lite53.047.2–58.752.9
⊕ RACE sign-only anchor · receiver = Qwen3.8-27B @Cerebras53.047.2–58.752.9
⊕ RACE sign-only anchor · receiver = Cosmos-Reason2-2B (CPU, ours)52.346.5–58.052.1
Gemini-3.5-flash-lite51.946.1–57.751.4
⊕ plurality51.245.4–57.050.8
Gemini-3.1-flash-lite50.244.4–56.050.0
⊕ RACE CGA · receiver = Cosmos-Reason2-2B (CPU, ours)43.838.2–49.643.3
Cosmos-Reason2-2B (CPU, ours)37.131.7–42.936.5
⊕ RACE · receiver = Cosmos-Reason2-2B (CPU, ours)37.131.7–42.936.5
Reproduction by category (left) and the multi-agent swarm without attackers (right).
Reproduction by category (left) and the multi-agent swarm without attackers (right).
Robustness vs clean accuracy with the robot's own CPU model as receiver.
Robustness vs clean accuracy with the robot's own CPU model as receiver.
Cosmos-Reason2 as a binary success verifier: calibration matters (56% → 83%).
Cosmos-Reason2 as a binary success verifier: calibration matters (56% → 83%).

Public benchmark · PlanBench Blocksworld

An exact simulator (accepts all 500 official ground-truth plans) turns into a repair signal: LLM-Modulo with ≤3 repairs.

92 instances answered by all 4 agents.
92 instances answered by all 4 agents.
A real cached LLM plan: the validator rejects step 11, the repaired plan reaches the goal.
agent / strategysuccess95% CILLM calls / task
qwen3.8-27b@cerebras · LLM-Modulo (≤3 repairs)100.096.0–100.01.02
swarm cascade-Modulo (cheap → expensive)100.096.0–100.01.02
gpt-oss-120b@groq · LLM-Modulo (≤3 repairs)100.096.0–100.01.10
swarm verify-select (one-shot, 4 agents)100.096.0–100.04.00
qwen3.8-27b@cerebras · one-shot97.892.4–99.41.00
gemini-3.1-flash-lite · LLM-Modulo (≤3 repairs)95.789.3–98.31.47
gpt-oss-120b@groq · one-shot91.383.8–95.51.00
swarm plurality (one-shot, 4 agents)90.282.4–94.84.00
qwen3.8-27b@groq · LLM-Modulo (≤3 repairs)72.863.0–80.92.26
gemini-3.1-flash-lite · one-shot69.659.5–78.01.00
qwen3.8-27b@groq · one-shot43.533.8–53.71.00

RoboRACE · Track A — SO-101 camera swarm

220 physics rollouts × 5 cameras; ground truth = final physics state.
220 physics rollouts × 5 cameras; ground truth = final physics state.
rankmethodmeancleanliars < 50%liars ≥ 50%worst cell
1 (1–1)race_net (new, learned)83.488.285.180.053.6
2 (2–2)race_s_online (new)82.787.384.279.854.5
3 (3–3)race_pool (new)82.086.283.878.754.5
4 (4–4)race_online (new)77.783.679.773.949.1
5 (5–6)race CGA (new)77.183.679.772.139.1
6 (5–7)race_s (new)76.385.178.671.949.1
7 (6–8)race_online sign-only anchor (ablation)76.083.679.669.116.4
8 (7–8)race75.083.378.069.344.5
9 (9–10)receiver_only71.371.371.371.355.5
refREF stack_lr (labelled)82.692.784.978.148.2
Accuracy as liars take over the camera swarm.
Accuracy as liars take over the camera swarm.
RACE-S signed weights learned without labels: liars get negative weight.
RACE-S signed weights learned without labels: liars get negative weight.
Per-camera quality of the five agents.
Per-camera quality of the five agents.

RoboRACE · Track A-Panda — replication on a different robot

All methods and hyper-parameters were frozen before this data existed. Same tasks, cameras and physics labels, new arm (Franka Panda).

Franka Panda rollouts with the camera swarm's verdicts.
rankmethodmeancleanliars < 50%liars ≥ 50%worst cell
1 (1–1)race_s_online (new)87.091.888.783.653.6
2 (2–2)race_pool (new)85.990.587.782.453.6
3 (3–3)race_net (new, learned)84.286.085.681.554.5
4 (4–4)race_online (new)83.290.085.978.049.1
5 (5–6)race CGA (new)82.790.085.976.634.5
6 (6–8)race_online sign-only anchor (ablation)81.890.085.974.25.5
7 (5–7)race81.794.286.871.95.5
8 (7–8)race_s (new)80.491.382.975.350.0
9 (9–9)receiver_only74.574.574.574.556.4
refREF stack_lr (labelled)84.996.487.080.750.0
Every method's mean accuracy on SO-101 vs Panda.
Every method's mean accuracy on SO-101 vs Panda.
Panda episodes seen by the five cameras.
Panda episodes seen by the five cameras.

Our two models: one design

Both models answer the same question in two domains: a robot receives reports from peers, some of whom lie in a structured way; what is true? They share one design from RACE and AIP: anchor on the receiver's own evidence, estimate every peer's channel in context, and invert consistent liars instead of discarding them. Peers and items are sets, and both models are trained on simulated swarms only. (design note)

RACE-Net-V (pick-and-place): axial set transformer over agents × episodes, 0.2 M parameters.
RACE-Net-V (pick-and-place): axial set transformer over agents × episodes, 0.2 M parameters.
RACE-Net-M (maze): anchored channel encoder, peer transformer, signed read-out, 3 refinement rounds, 45 K parameters.
RACE-Net-M (maze): anchored channel encoder, peer transformer, signed read-out, 3 refinement rounds, 45 K parameters.

RACE-Net-V — the learned verification model

A 0.2 M-parameter axial set transformer that has learned receiver-anchored truth inference from synthetic swarms and is applied unchanged to the real camera swarms. RACE-Pool is the AIP-style pool of RACE-Net-V, RACE-S and RACE. Weights: Nabidnur/RoboRACE-Net.

Training curve and zero-shot transfer to real VLM camera swarms.
Training curve and zero-shot transfer to real VLM camera swarms.
Scaling study on an independent synthetic generator.
Scaling study on an independent synthetic generator.

Hold-out: a test nothing was tuned on

New episodes of both arms (new seeds). Their views were scored by the frozen VLM pipeline, and no fusion method was run on them until a commit had frozen every model, rule and threshold (results/holdout/FREEZE.json).

SO-101 hold-out

rankmethodmeancleanliars < 50%liars ≥ 50%worst cell
1 (1–1)race_s_online (new)82.686.784.179.753.6
2 (2–2)race_pool (new)81.284.982.978.052.7
3 (3–3)race_online (new)79.986.582.175.652.7
4 (4–5)race CGA (new)79.486.582.174.337.3
5 (4–6)race_net (new, learned)79.077.180.077.255.5
6 (5–6)race_online sign-only anchor (ablation)78.486.582.271.311.8
7 (7–7)race74.380.076.869.733.6
refREF stack_lr (labelled)82.693.685.377.545.5

Franka Panda hold-out

rankmethodmeancleanliars < 50%liars ≥ 50%worst cell
1 (1–1)race_pool (new)86.790.988.783.156.4
2 (2–3)race_online (new)85.992.789.080.151.8
3 (2–4)race_s_online (new)85.489.187.082.558.2
4 (3–5)race CGA (new)84.892.788.977.135.5
5 (4–6)race_net (new, learned)84.387.585.781.654.5
6 (5–7)race_online sign-only anchor (ablation)83.492.788.973.06.4
7 (6–7)race82.189.585.376.148.2
refREF stack_lr (labelled)82.693.684.778.550.0

RoboRACE · Track B — Byzantine rendezvous

12 robots, 4 colluding Byzantine robots: DeGroot vs W-MSR vs RACE-C + W-MSR.
rankmethodmeancleanliars < 50%liars ≥ 50%worst cell
1 (1–1)race_c+wmsr94.3100.094.1–0.0
2 (2–3)geomedian87.1100.086.7–0.0
3 (3–5)median84.3100.083.8–0.0
4 (3–5)trimmed83.7100.083.2–0.0
5 (3–6)wmsr82.1100.081.5–12.5
6 (2–6)race_c72.982.572.6–0.0
7 (7–7)mean3.2100.00.0–0.0
Mission success vs number of Byzantine robots.
Mission success vs number of Byzantine robots.

RoboRACE · Track C — LLM planners

Same models, better plumbing: zero-shot vs engineered.
Same models, better plumbing: zero-shot vs engineered.
rankmethodmeancleanliars < 50%liars ≥ 50%worst cell
1 (1–1)receiver_only97.797.797.797.790.9
2 (2–3)race93.297.797.090.09.1
3 (2–3)plurality91.7100.0100.084.80.0
4 (4–4)race_unanchored (DS-style)85.5100.0100.073.40.0
refREF best_single (labelled)86.2100.091.780.90.0

RoboRACE · Track D — maze exploration with lying robots · RACE-Net-M

Six mobile robots explore an unknown maze and share what they sense; up to four corrupt their broadcasts. RACE-Net-M (45 K parameters, trained on simulated mazes with random liars; the benchmark's attack names never appear in training) verifies each peer against cells the robot sensed itself and inverts consistent liars. It is compared with six rules from the literature and with our hand-written RACE map. Score = SPL of honest robots. Weights: Nabidnur/RoboRACE-Net-M.

Three of six robots invert their reports: majority map (false walls in red) vs RACE map. This is the maze with the largest gap; averages below.
rankmethodmeancleanliars < 50%liars ≥ 50%worst cell
1 (1–1)race_net_m + reach-repair70.276.873.166.80.0
2 (2–2)race_net_m69.876.872.666.40.0
3 (3–3)race_map66.276.869.761.80.0
4 (4–5)wmsr_trim64.176.865.661.513.9
5 (4–6)reputation63.876.867.958.50.0
6 (5–7)eigentrust63.075.066.858.20.0
7 (6–8)dawid_skene62.076.367.455.50.0
8 (7–8)solo61.661.562.061.213.9
9 (9–9)trust_all56.776.859.951.70.0
10 (10–10)majority55.776.861.548.10.0
refREF oracle71.376.874.567.50.0
SPL as liars take over the maze swarm.
SPL as liars take over the maze swarm.
RACE-Net-M, unchanged, on swarm and map sizes it was never trained on.
RACE-Net-M, unchanged, on swarm and map sizes it was never trained on.
LLM agents on text mazes: one-shot vs validator-feedback repair.
LLM agents on text mazes: one-shot vs validator-feedback repair.

In context: published numbers

Grey = numbers from the cited papers' tables; teal = measured here. Context only: models, dates and instance subsets differ.
Grey = numbers from the cited papers' tables; teal = measured here. Context only: models, dates and instance subsets differ.

Validation

Limitations