EmbodiedSWE: Coding Agents for Long-Horizon Dexterous Robotics

Overview

We study how frontier coding agents can help robotics. EmbodiedSWE has four parts. EmbodiedSWE-Bench is an agent-native benchmark of everyday robotics tasks. A frontier evaluation measures how well today's coding agents perform on them along several dimensions. EmbodiedSWE-Gen then diversifies the agents' verified solutions into a large dataset for training a general robot policy. Finally, we generate new tasks from existing ones to further improve the coding agents via RL.

From coding agents to the physical world

EmbodiedSWE Overview

Improved agent becomes the solver

01 / Benchmark

EmbodiedSWE-Bench

Explore the benchmark ↗

A new agent-native benchmark covering a variety of everyday tasks, with horizons up to half an hour. Twenty-eight tasks across six suites span assembly, organizing, puzzles, deformables, cutting, and loco-manipulation, on seventeen embodiments from single arms to humanoids. Coding agents interact with the simulator directly, and hidden graders score physical task progress offline.

01 EmbodiedSWE-Bench

An agent-native robotics benchmark powered by Isaac Sim and Newton. The tasks are long-horizon, precise, and dexterous, drawn from everyday scenarios that include liquids, deformables, cutting, and contact-rich assembly.

task suites

new scenes

robot embodiments

controllers

~30 min

task horizon

Design principles

01 Modularity

Every environment decomposes into a scene, a robot, and a controller, each defined independently. A new scene or embodiment composes with everything already there, so the task set grows quickly. We ship 28 scenes and 17 embodiments, each with its own variants, such as object types or supported end-effectors, to widen the range further.

02 Standardization

All environments share a Gym-like API and run massively parallel on the GPU. Evaluations run in isolated containers with identical system settings, and tools and prompts are registered from a configuration file, so ablations are easy. Every submission is stamped with the tokens and wall clock spent to record performance against cost.

03 Agent-native

We provide no task-specific macro primitives, only generic scene-inspection tools and controllers such as IK, with gains the agent tunes itself. Tasks arrive in natural language, and the agent has full simulation access to explore the scene. Grading is purely offline against a hidden rubric.

Please refer to our paper and code for more details on the benchmark and evaluation. Below are representative tasks.

02 Frontier agents evaluation

We evaluate six frontier coding models, each in its own standard coding harness, on one NVIDIA RTX 4090 per run with a 4-hour wall-clock budget. Submissions are graded offline against a hidden rubric to prevent reward exploits or hacking of the evaluation system.

Model	Harness	Mean score	Success rate	Hack rate	Wall-clock (min)	Median min. to solve
GPT-6 Astra	Codex	0.94 ± 0.03	82%	0%	154	39
Fable 5.1	Claude Code	0.75 ± 0.07	61%	4%	151	73
Opus 5	Claude Code	0.66 ± 0.08	50%	7%	211	103
Opus 4.8	Claude Code	0.52 ± 0.08	25%	11%	191	98
GPT-5.6 Sol	Codex	0.44 ± 0.08	18%	43%	117	58
GPT-5.6 Terra	Codex	0.26 ± 0.07	11%	39%	151	119

Scores lie in [0, 1]. A score of 1 means the run fully achieves the natural-language goal; partial credit is given for reaching significant intermediate stages, such as attaching one leg in the table-assembly task.

Best-so-far score against wall-clock budget

Mean over tasks of the highest score reached by each time. GPT-6 Astra passes 0.9 inside two hours; every other curve flattens well below ceiling.

Share of the benchmark solved

Fraction of tasks reaching a successful solution within the 4-hour budget.

Best-so-far score against spend

API cost per task, log scale. The paper also reports these curves against generated tokens.

Transfer learning

We further evaluate transfer learning: given a previously verified solution as a hint, does the agent learn faster or reach a higher score?

We consider two settings. In cross-task transfer the agent is given a verified solution to a different task on the same embodiment; in cross-embodiment transfer it is given the solution to the same task but must perform it with another robot arm. For the cross-task study we further split the hints into similar tasks, which share the target's dominant underlying skill (e.g. threading a nut as the hint for screwing in a light bulb), and dissimilar tasks, which do not.

Cross-task transfer

A hint from another task, similar or not, against no hint.

Cross-embodiment transfer

The verified Franka solution, ported to two other arms.

The transfer learning experiments are performed on Opus 5 and GPT-5.6 Sol with three seeds per configuration over six different tasks.

We find that coding agents demonstrate strong transfer learning capability on robotics tasks: given a verified solution to a similar task, or to the same task on another robot, as a hint, they solve the target noticeably faster and reach a higher final score.

Please refer to our paper and blog for more technical details, including failure analysis, harness comparisons, and comparisons with RL.

03 EmbodiedSWE-Gen

A single coding-agent solve costs hours of wall clock and a substantial token budget, and in real deployment the latency and the risk of damage during iteration make that loop hard to afford. In this section we therefore consider how to use the coding agent as a teacher to scalably generate demonstrations for training a VLA. We propose a pipeline that systematically perturbs agent-generated solutions to produce a large and diverse trajectory dataset for policy training.

Grids of rendered tiles showing scene, strategy, phase, dynamics and visual variation for one task.

The diversification levels compose multiplicatively, expanding both the number and diversity of trajectories from a single solved task. As more trajectories are generated, the agent cost is increasingly amortized, causing the token cost per trajectory to drop sharply.

Verified successful trajectories

Composing diversification levels rapidly increases the number of successful trajectories.

Agent token cost

Token cost per trajectory drops significantly as diversification scales, while cumulative agent cost grows slowly.

We use the generated trajectories to train a SmolVLA policy. Both success rate and score climb steadily with the number of demonstrations per task: mean success rises from 14% at 10 demonstrations to 66% at 400.

Success rate

Score

SmolVLA trained for 100k steps at each dataset size, three runs per setting, each evaluated over 32 trials per task.

04 Coding agent improvement

Robotics simulation provides a natural training environment for coding agents: outcomes are directly verifiable, and multi-stage tasks offer meaningful partial credit. We then propose a pipeline that generates new tasks at scale, creating a large and diverse training set to improve coding agents on robotic tasks.

01 Seed and mutate

A seed task is sampled from a deduplicated pool of 194 manipulation tasks in RoboVerse. An agent uses the seed as a reference to construct a new task from scratch in the benchmark’s format.

02 Check task mechanics

To establish physical feasibility the agent first writes a simplified "teleport" solution that validates contact dynamics without the real robot. Only then does it build the scoring rubric, write tests that confirm the rubric penalizes incorrect states, and draft the task description.

03 Validate and judge

Six validation steps check simulation stability, scoring accuracy, and robust success of the teleport solution. An LLM judge then rates novelty against the seed, solution legitimacy, and description clarity, and rejects the task if any of the three fail.

We evaluate the learnability of the generated tasks by training Seed-2.1-Lite with turn-level PPO on 400 environments. Mean task reward increases from 0.09 to 0.18, while success rate rises from 2.7% to 7.5%.

Reward

Success and non-zero rate

Cite

@misc{embodiedswe2026,
  title        = {EmbodiedSWE: Coding Agents for Long-Horizon Dexterous Robotics},
  author       = {Shen, Zeyu and You, Haoxiang and Liu, Yilang and Zheng, Zhicheng and Zha, Lihan and Yamazaki, Kashu and Zhang, Mingtong and Huang, Suning and Sun, Jiankai and Chen, Qianzhong and He, Lucy and Chang, Haoran and Shah, Dhruv and Schwager, Mac and Fragkiadaki, Katerina and Henderson, Peter and Abraham, Ian and Xu, Canwen},
  year         = {2026},
  howpublished = {\url{https://embodiedswe.github.io/}},
  note         = {Project page},
}

Cite the project page for now — a formal citation will be added when the paper is public.

EmbodiedSWE Coding Agents for Long-Horizon Dexterous Robotics