RoboRACE
Byzantine-robust multi-agent robotics: when some of your robots, cameras or AI agents are wrong on purpose. Two public benchmarks (ERQA, PlanBench) and five RoboRACE tracks, run on a CPU with real VLMs, real LLMs, two simulated robot arms and a maze swarm. Every number below is generated from cached agent outputs in the dataset repository.
The idea
Majority vote and classical truth inference (Dawid–Skene, GLAD, MACE) fail once liars are the majority, because the latent-truth model becomes symmetric. RACE (from the Liars Are Information program) anchors on the receiver's own honesty and gives every peer a signed channel weight: trust, invert or discard. This work adds RACE-S (score level), online variants against sleepers, RACE-C + W-MSR for robot consensus, confidence-gated anchoring (reported as a compromise, not a free lunch), and an engineered / LLM-Modulo planning layer.

Architecture




Public benchmark · ERQA
Cosmos-Reason2-2B reproduced on 4 CPU cores: 33.8% (95% CI 29.3–38.5, n = 400), reported 37.75%. At ≤768² px: 35.5% (19 of 400 answers change). Swarm of 4 VLM agents on 283 shared questions:
| system | accuracy | 95% CI | reweighted |
|---|---|---|---|
| ⊕ Dawid-Skene (crowd-kit) | 54.8 | 48.9–60.5 | 54.5 |
| Qwen3.8-27B @Cerebras | 54.1 | 48.2–59.8 | 53.7 |
| ⊕ RACE · receiver = Gemini-3.1-flash-lite | 53.0 | 47.2–58.7 | 52.9 |
| ⊕ RACE · receiver = Gemini-3.5-flash-lite | 53.0 | 47.2–58.7 | 52.9 |
| ⊕ RACE · receiver = Qwen3.8-27B @Cerebras | 53.0 | 47.2–58.7 | 52.9 |
| ⊕ RACE sign-only anchor · receiver = Gemini-3.1-flash-lite | 53.0 | 47.2–58.7 | 52.9 |
| ⊕ RACE sign-only anchor · receiver = Gemini-3.5-flash-lite | 53.0 | 47.2–58.7 | 52.9 |
| ⊕ RACE sign-only anchor · receiver = Qwen3.8-27B @Cerebras | 53.0 | 47.2–58.7 | 52.9 |
| ⊕ RACE sign-only anchor · receiver = Cosmos-Reason2-2B (CPU, ours) | 52.3 | 46.5–58.0 | 52.1 |
| Gemini-3.5-flash-lite | 51.9 | 46.1–57.7 | 51.4 |
| ⊕ plurality | 51.2 | 45.4–57.0 | 50.8 |
| Gemini-3.1-flash-lite | 50.2 | 44.4–56.0 | 50.0 |
| ⊕ RACE CGA · receiver = Cosmos-Reason2-2B (CPU, ours) | 43.8 | 38.2–49.6 | 43.3 |
| Cosmos-Reason2-2B (CPU, ours) | 37.1 | 31.7–42.9 | 36.5 |
| ⊕ RACE · receiver = Cosmos-Reason2-2B (CPU, ours) | 37.1 | 31.7–42.9 | 36.5 |



Public benchmark · PlanBench Blocksworld
An exact simulator (accepts all 500 official ground-truth plans) turns into a repair signal: LLM-Modulo with ≤3 repairs.

| agent / strategy | success | 95% CI | LLM calls / task |
|---|---|---|---|
| qwen3.8-27b@cerebras · LLM-Modulo (≤3 repairs) | 100.0 | 96.0–100.0 | 1.02 |
| swarm cascade-Modulo (cheap → expensive) | 100.0 | 96.0–100.0 | 1.02 |
| gpt-oss-120b@groq · LLM-Modulo (≤3 repairs) | 100.0 | 96.0–100.0 | 1.10 |
| swarm verify-select (one-shot, 4 agents) | 100.0 | 96.0–100.0 | 4.00 |
| qwen3.8-27b@cerebras · one-shot | 97.8 | 92.4–99.4 | 1.00 |
| gemini-3.1-flash-lite · LLM-Modulo (≤3 repairs) | 95.7 | 89.3–98.3 | 1.47 |
| gpt-oss-120b@groq · one-shot | 91.3 | 83.8–95.5 | 1.00 |
| swarm plurality (one-shot, 4 agents) | 90.2 | 82.4–94.8 | 4.00 |
| qwen3.8-27b@groq · LLM-Modulo (≤3 repairs) | 72.8 | 63.0–80.9 | 2.26 |
| gemini-3.1-flash-lite · one-shot | 69.6 | 59.5–78.0 | 1.00 |
| qwen3.8-27b@groq · one-shot | 43.5 | 33.8–53.7 | 1.00 |
RoboRACE · Track A — SO-101 camera swarm

| rank | method | mean | clean | liars < 50% | liars ≥ 50% | worst cell |
|---|---|---|---|---|---|---|
| 1 (1–1) | race_net (new, learned) | 83.4 | 88.2 | 85.1 | 80.0 | 53.6 |
| 2 (2–2) | race_s_online (new) | 82.7 | 87.3 | 84.2 | 79.8 | 54.5 |
| 3 (3–3) | race_pool (new) | 82.0 | 86.2 | 83.8 | 78.7 | 54.5 |
| 4 (4–4) | race_online (new) | 77.7 | 83.6 | 79.7 | 73.9 | 49.1 |
| 5 (5–6) | race CGA (new) | 77.1 | 83.6 | 79.7 | 72.1 | 39.1 |
| 6 (5–7) | race_s (new) | 76.3 | 85.1 | 78.6 | 71.9 | 49.1 |
| 7 (6–8) | race_online sign-only anchor (ablation) | 76.0 | 83.6 | 79.6 | 69.1 | 16.4 |
| 8 (7–8) | race | 75.0 | 83.3 | 78.0 | 69.3 | 44.5 |
| 9 (9–10) | receiver_only | 71.3 | 71.3 | 71.3 | 71.3 | 55.5 |
| ref | REF stack_lr (labelled) | 82.6 | 92.7 | 84.9 | 78.1 | 48.2 |



RoboRACE · Track A-Panda — replication on a different robot
All methods and hyper-parameters were frozen before this data existed. Same tasks, cameras and physics labels, new arm (Franka Panda).
| rank | method | mean | clean | liars < 50% | liars ≥ 50% | worst cell |
|---|---|---|---|---|---|---|
| 1 (1–1) | race_s_online (new) | 87.0 | 91.8 | 88.7 | 83.6 | 53.6 |
| 2 (2–2) | race_pool (new) | 85.9 | 90.5 | 87.7 | 82.4 | 53.6 |
| 3 (3–3) | race_net (new, learned) | 84.2 | 86.0 | 85.6 | 81.5 | 54.5 |
| 4 (4–4) | race_online (new) | 83.2 | 90.0 | 85.9 | 78.0 | 49.1 |
| 5 (5–6) | race CGA (new) | 82.7 | 90.0 | 85.9 | 76.6 | 34.5 |
| 6 (6–8) | race_online sign-only anchor (ablation) | 81.8 | 90.0 | 85.9 | 74.2 | 5.5 |
| 7 (5–7) | race | 81.7 | 94.2 | 86.8 | 71.9 | 5.5 |
| 8 (7–8) | race_s (new) | 80.4 | 91.3 | 82.9 | 75.3 | 50.0 |
| 9 (9–9) | receiver_only | 74.5 | 74.5 | 74.5 | 74.5 | 56.4 |
| ref | REF stack_lr (labelled) | 84.9 | 96.4 | 87.0 | 80.7 | 50.0 |


Our two models: one design
Both models answer the same question in two domains: a robot receives reports from peers, some of whom lie in a structured way; what is true? They share one design from RACE and AIP: anchor on the receiver's own evidence, estimate every peer's channel in context, and invert consistent liars instead of discarding them. Peers and items are sets, and both models are trained on simulated swarms only. (design note)


RACE-Net-V — the learned verification model
A 0.2 M-parameter axial set transformer that has learned receiver-anchored truth inference from synthetic swarms and is applied unchanged to the real camera swarms. RACE-Pool is the AIP-style pool of RACE-Net-V, RACE-S and RACE. Weights: Nabidnur/RoboRACE-Net.


Hold-out: a test nothing was tuned on
New episodes of both arms (new seeds). Their views were scored by the frozen VLM pipeline, and no fusion method was run on them until a commit had
frozen every model, rule and threshold (results/holdout/FREEZE.json).
SO-101 hold-out
| rank | method | mean | clean | liars < 50% | liars ≥ 50% | worst cell |
|---|---|---|---|---|---|---|
| 1 (1–1) | race_s_online (new) | 82.6 | 86.7 | 84.1 | 79.7 | 53.6 |
| 2 (2–2) | race_pool (new) | 81.2 | 84.9 | 82.9 | 78.0 | 52.7 |
| 3 (3–3) | race_online (new) | 79.9 | 86.5 | 82.1 | 75.6 | 52.7 |
| 4 (4–5) | race CGA (new) | 79.4 | 86.5 | 82.1 | 74.3 | 37.3 |
| 5 (4–6) | race_net (new, learned) | 79.0 | 77.1 | 80.0 | 77.2 | 55.5 |
| 6 (5–6) | race_online sign-only anchor (ablation) | 78.4 | 86.5 | 82.2 | 71.3 | 11.8 |
| 7 (7–7) | race | 74.3 | 80.0 | 76.8 | 69.7 | 33.6 |
| ref | REF stack_lr (labelled) | 82.6 | 93.6 | 85.3 | 77.5 | 45.5 |
Franka Panda hold-out
| rank | method | mean | clean | liars < 50% | liars ≥ 50% | worst cell |
|---|---|---|---|---|---|---|
| 1 (1–1) | race_pool (new) | 86.7 | 90.9 | 88.7 | 83.1 | 56.4 |
| 2 (2–3) | race_online (new) | 85.9 | 92.7 | 89.0 | 80.1 | 51.8 |
| 3 (2–4) | race_s_online (new) | 85.4 | 89.1 | 87.0 | 82.5 | 58.2 |
| 4 (3–5) | race CGA (new) | 84.8 | 92.7 | 88.9 | 77.1 | 35.5 |
| 5 (4–6) | race_net (new, learned) | 84.3 | 87.5 | 85.7 | 81.6 | 54.5 |
| 6 (5–7) | race_online sign-only anchor (ablation) | 83.4 | 92.7 | 88.9 | 73.0 | 6.4 |
| 7 (6–7) | race | 82.1 | 89.5 | 85.3 | 76.1 | 48.2 |
| ref | REF stack_lr (labelled) | 82.6 | 93.6 | 84.7 | 78.5 | 50.0 |
RoboRACE · Track B — Byzantine rendezvous
| rank | method | mean | clean | liars < 50% | liars ≥ 50% | worst cell |
|---|---|---|---|---|---|---|
| 1 (1–1) | race_c+wmsr | 94.3 | 100.0 | 94.1 | – | 0.0 |
| 2 (2–3) | geomedian | 87.1 | 100.0 | 86.7 | – | 0.0 |
| 3 (3–5) | median | 84.3 | 100.0 | 83.8 | – | 0.0 |
| 4 (3–5) | trimmed | 83.7 | 100.0 | 83.2 | – | 0.0 |
| 5 (3–6) | wmsr | 82.1 | 100.0 | 81.5 | – | 12.5 |
| 6 (2–6) | race_c | 72.9 | 82.5 | 72.6 | – | 0.0 |
| 7 (7–7) | mean | 3.2 | 100.0 | 0.0 | – | 0.0 |

RoboRACE · Track C — LLM planners

| rank | method | mean | clean | liars < 50% | liars ≥ 50% | worst cell |
|---|---|---|---|---|---|---|
| 1 (1–1) | receiver_only | 97.7 | 97.7 | 97.7 | 97.7 | 90.9 |
| 2 (2–3) | race | 93.2 | 97.7 | 97.0 | 90.0 | 9.1 |
| 3 (2–3) | plurality | 91.7 | 100.0 | 100.0 | 84.8 | 0.0 |
| 4 (4–4) | race_unanchored (DS-style) | 85.5 | 100.0 | 100.0 | 73.4 | 0.0 |
| ref | REF best_single (labelled) | 86.2 | 100.0 | 91.7 | 80.9 | 0.0 |
RoboRACE · Track D — maze exploration with lying robots · RACE-Net-M
Six mobile robots explore an unknown maze and share what they sense; up to four corrupt their broadcasts. RACE-Net-M (45 K parameters, trained on simulated mazes with random liars; the benchmark's attack names never appear in training) verifies each peer against cells the robot sensed itself and inverts consistent liars. It is compared with six rules from the literature and with our hand-written RACE map. Score = SPL of honest robots. Weights: Nabidnur/RoboRACE-Net-M.
| rank | method | mean | clean | liars < 50% | liars ≥ 50% | worst cell |
|---|---|---|---|---|---|---|
| 1 (1–1) | race_net_m + reach-repair | 70.2 | 76.8 | 73.1 | 66.8 | 0.0 |
| 2 (2–2) | race_net_m | 69.8 | 76.8 | 72.6 | 66.4 | 0.0 |
| 3 (3–3) | race_map | 66.2 | 76.8 | 69.7 | 61.8 | 0.0 |
| 4 (4–5) | wmsr_trim | 64.1 | 76.8 | 65.6 | 61.5 | 13.9 |
| 5 (4–6) | reputation | 63.8 | 76.8 | 67.9 | 58.5 | 0.0 |
| 6 (5–7) | eigentrust | 63.0 | 75.0 | 66.8 | 58.2 | 0.0 |
| 7 (6–8) | dawid_skene | 62.0 | 76.3 | 67.4 | 55.5 | 0.0 |
| 8 (7–8) | solo | 61.6 | 61.5 | 62.0 | 61.2 | 13.9 |
| 9 (9–9) | trust_all | 56.7 | 76.8 | 59.9 | 51.7 | 0.0 |
| 10 (10–10) | majority | 55.7 | 76.8 | 61.5 | 48.1 | 0.0 |
| ref | REF oracle | 71.3 | 76.8 | 74.5 | 67.5 | 0.0 |



In context: published numbers

Validation
- ERQA pipeline: reported 37.75% inside our 95% CI.
- Blocksworld validator: accepts 500/500 official ground-truth plans, rejects corrupted ones.
- W-MSR: the safety theorem holds in every tested case (honest states stay in the honest hull).
- Dawid–Skene: ≥90% decision agreement with the crowd-kit reference implementation.
- A fresh clone regenerates every leaderboard, figure and the README byte-for-byte.
Limitations
- Track A renders are simulated (real SO-101 mesh, synthetic lighting); all cameras share one VLM.
- ERQA swarm covers the questions every hosted agent answered within free-tier quotas; the SOTA reference covers 27 questions.
- PlanBench uses a seeded 100-instance subset; Track C saturates after engineering.
- Confidence-gated anchoring was proposed post hoc; its robust floor did not replicate on the full data.