Andon Labs

Andon Labs: stress-testing how far you can trust an autonomous AI agent

Andon Labs is an AI safety lab preparing for a future where organizations are run by AI with minimal human oversight. Rather than publishing abstract safety papers, the team benchmarks and deploys frontier models in real businesses and measures what actually happens over long horizons. Its best-known instrument is Vending-Bench, a simulation where a model must run a vending machine business for a full year — ordering stock, negotiating with adversarial suppliers, handling customer complaints, and staying solvent. The headline finding: some frontier LLMs can turn a profit, but every model has runs that derail completely.

The lab pairs simulations with physical deployments. It partnered with Anthropic on Project Vend, in which Claude ("Claudius") ran a real office shop, and its Butter-Bench evaluation puts LLMs in control of an actual robot to test practical, embodied intelligence — where the best model scored roughly 40% against a 95% human baseline. Founded in 2023 by Lukas Petersson, Axel Backlund, and Emil Froberg, the San Francisco team works with frontier labs that need honest answers about long-horizon agent reliability.

Key Features

• Vending-Bench: a long-horizon benchmark where models run a simulated vending business for a year, measuring coherence, profit, and failure modes
• Real-world autonomous deployments, including Project Vend with Anthropic where Claude ran a physical office shop
• Butter-Bench: evaluation of LLM-controlled robots on household delivery tasks (best model ~40% vs 95% for humans)
• Custom evaluations built for frontier AI labs and published safety research
• Public leaderboards and papers documenting where and how autonomous agents derail
• Open invitation for labs to submit models for benchmarking

Ideal Use Case

Andon Labs is not an off-the-shelf SaaS product; it serves frontier model labs, agent developers, and safety researchers who need rigorous evidence about long-horizon autonomy. If you are shipping an agent that will control money, inventory, or hardware without a human approving each step, Andon Labs' evaluations are the closest thing to a crash-test lab for that scenario. Enterprises evaluating agentic vendors also use its published results as an independent reference point.

How Andon Labs differentiates

Most eval shops test single-turn accuracy; Andon Labs tests whether an agent stays coherent across a simulated year of business decisions. Its credibility fact is hard to match: Anthropic itself partnered with Andon Labs on Project Vend, the widely covered experiment where Claude autonomously ran a small shop in Anthropic's San Francisco office, and Vending-Bench results are cited in frontier-lab model cards and press. The company is Y Combinator-backed and was founded in 2023.

FAQ

What is Vending-Bench? A benchmark where an LLM manages a simulated vending machine business for a year — procurement, pricing, supplier negotiation, and customer complaints — to measure long-term coherence and profit.

Is Andon Labs a product I can buy? It is a research lab offering custom evaluations and deployments to AI labs and agent builders; engagements are scoped directly — contact the founders to inquire.

What was Project Vend? A collaboration with Anthropic in which Claude ran a real office shop, surfacing failure modes like identity confusion and unprofitable discounting.

Do they test physical robots? Yes. Butter-Bench evaluates LLMs orchestrating a real robot on delivery tasks; top models scored around 40% versus 95% for humans.

tl;dr

Andon Labs is the YC-backed safety lab behind Vending-Bench and Anthropic's Project Vend, benchmarking whether autonomous agents can run real businesses without derailing. Custom evaluations by inquiry.

Related

Looking for more options? Browse the Security & Governance directory or read our best AI security tools listicle. Andon Labs is also tracked on Crunchbase.