so101-block-stack-eval

The SO-101 block stack in ManiSkill, as an Inspect Robots embodiment and task.

Leaderboard

95% interval
1 claude-opus-5-5 anthropic 88% 95% interval 78%–96% 100% 0.237 18 136,590 2,876 1.0
2 pi0.5 pi 82% 95% interval 72%–92% 100%
3 pi0 pi 76% 95% interval 64%–86% 94%
4 lawam rlinf 62% 95% interval 50%–74% 90%
5 claude-sonnet-5-5 anthropic 52% 95% interval 40%–66% 92% 0.234 29 336,394 5,743 1.6
6 vla-jepa ustc 50% 95% interval 36%–64% 86%
7 claude-haiku-5-5 anthropic 34% 95% interval 22%–48% 86% 0.020 38 523,773 10,333 1.3
8 deepseek-v4.1-flash deepseek 30% 95% interval 18%–42% 90% 0.082 62 723,514 76,012 8.2
9 glm-5.3-flash z-ai 24% 95% interval 12%–36% 70% 0.039 50 475,919 7,002 2.1
10 gpt-6-luna openai 16% 95% interval 6%–26% 34% 0.011 24 127,304 2,052 1.5
11 nemotron-3.5-super-vl nvidia 0% 95% interval 0%–0% 12% 0.055 52 636,514 17,841 1.5
11 smolvla huggingface 0% 95% interval 0%–0% 16%

Lines show 95% bootstrap intervals. Updated 2026-11-07. Download results.json

50 layouts, one trial each; 95% bootstrap interval over layouts. Cost in US dollars, calls, tokens and minutes waiting on the model are means per trial. Prompt version 8dd3b142780d.

The pi0.5, pi0, LaWAM, VLA-JEPA and SmolVLA rows are VLAs, each a LoRA fine-tune of the base model on the cell's oracle demonstrations: pi0.5, pi0, LaWAM, VLA-JEPA, SmolVLA. Each commands the joint targets directly and executes 10 actions of each chunk before predicting the next (VLA-JEPA 7, SmolVLA all 50), read back off a smooth cubic through 13 of its steps. pi0.5, pi0 and SmolVLA stitch their chunks with Real-Time Chunking.

Each model on layout 42, "stack the red block on the green block", from the top camera, in leaderboard order. An agent's trial ends when it stops; a VLA's is shown up to the step it stacks the block, or all 600 if it never does.

claude-opus-5-5 on layout 42 pi0.5 on layout 42 pi0 on layout 42 lawam on layout 42 claude-sonnet-5-5 on layout 42 vla-jepa on layout 42
claude-opus-5-5 pi0.5 pi0 lawam claude-sonnet-5-5 vla-jepa
83 steps 110 steps 102 steps 106 steps 81 steps 142 steps
claude-haiku-5-5 on layout 42 deepseek-v4.1-flash on layout 42 glm-5.3-flash on layout 42 gpt-6-luna on layout 42 nemotron-3.5-super-vl on layout 42 smolvla on layout 42
claude-haiku-5-5 deepseek-v4.1-flash glm-5.3-flash gpt-6-luna nemotron-3.5-super-vl smolvla
459 steps 185 steps 82 steps 83 steps 74 steps 600 steps

The trials behind the table, with each run's actions, transcripts and model calls, are in the Hugging Face dataset.

Task

so101-block-stack has one scene per layout in src/so101_block_stack_eval/layouts.jsonl. Each layout gives:

A trial succeeds on the first step the named block sits on its target, out of the jaws, with neither of the other blocks pushed out of place. A second scorer, lifted, records whether the jaws ever held the named block off the table.

so101-block-stack-until-done runs each trial until the agent calls done or give_up, or its steps run out. The last step repeats its command for a second, puts the arm pointing straight up without sweeping it through the blocks, and waits another second. standing records whether the named block was out of the jaws on its target before the arm moved, and still is, with neither other block pushed out of place and no part of the arm touching the stack; stacked records whether the stack ever held as the default task scores it. The leaderboard counts this task, with standing as Stacked.

Embodiment

so101_sim is one ManiSkill environment with physics on the CPU. Each action is an absolute pose for the grip point: x, y and z in the arm's base frame, the wrist's yaw, the tool's pitch below level, and the jaw opening. The grip point is where a block's centre sits once the jaws close on it. A closed-form IK turns each pose into joint targets. A target the arm cannot reach, or one that would turn a joint too far in one step, is refused and the arm holds its last target. Observations are the wrist and overhead camera images, the commanded pose read back from the arm, and the joint positions.

Running an agent

uv sync
export OPENROUTER_API_KEY=...  # or ANTHROPIC_API_KEY for anthropic/claude-* models
uv run python scripts/run.py z-ai/glm-5.3-flash --until-done

scripts/run.py passes the agent two things the inspect-robots run CLI cannot: SimEmbodiment.pre_check, which gives the reason a motion would be refused, and LOCATE_POINTS, a custom tool that maps top-camera pixels to base-frame points on a horizontal plane. Every model runs at low reasoning effort, through OpenRouter, except Anthropic's, which go to Anthropic's Messages API under the names Anthropic gives them (anthropic/claude-haiku-5-5).

Only the cameras render on the GPU. To run without one, hide it and render with Mesa's software Vulkan:

CUDA_VISIBLE_DEVICES= VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/lvp_icd.json \
    uv run python scripts/run.py z-ai/glm-5.3-flash --until-done

The agent plugin's custom tools are not released yet, so inspect-robots-agent and inspect-robots come from the agent-custom-tools branch of a fork; uv.lock records the commit.

Tests

uv run pytest

License

Apache 2.0. The arm's description in src/so101_block_stack_eval/description/ is adapted from the SO-101 in TheRobotStudio's SO-ARM100, under the same license.

Citation

@misc{wahl2026so101blockstackeval,
  title = {{so101-block-stack-eval: LLM agents stacking blocks with a simulated SO-101 arm}},
  author = {Wahl, Dan},
  year = {2026},
  url = {https://github.com/danwahl/so101-block-stack-eval},
  note = {Version 0.1.0}
}