Ai2 releases ToyBlocks to test how robots handle unfamiliar tasks

The Allen Institute for AI (Ai2) released ToyBlocks on Nov. 6, an open simulation framework for training and testing robots on tabletop tasks. Researchers can use it to generate training data automatically, then test whether a robot can apply what it learned to unfamiliar block layouts and tasks. huggingface.co

The initial release includes 85,336 demonstrations of multistep tasks, representing nearly 1,600 hours of simulated robot activity. Ai2 also released two trained models, simulation assets, task and demonstration generators, and code for training and evaluation. huggingface.co

The simulator centers on a robot arm that can push, flip, rotate and stack blocks. In one task, the robot must turn blocks over to find hidden letters before arranging them to spell a word. Researchers can vary block size and table height, reserve layouts for testing, and track how far the robot gets, how fast it moves and how much force it applies. For multistep tasks, they can also test planning and movement separately to pinpoint where a failure occurred. huggingface.co

In early block-spelling trials with letters already visible and upright, a system trained entirely on ToyBlocks data succeeded in 111 of 200 simulated attempts and 15 of 30 attempts with a physical robot. Ai2 said more trials are needed to establish statistical confidence across tasks. huggingface.co

ToyBlocks is the first release in Ai2’s FoundationWorlds initiative, which aims to build controlled environments for studying robot learning. In its Hugging Face announcement, Ai2 said it was targeting a second release within weeks, with a larger dataset, more robot skills and broader benchmarks. huggingface.co