The benchmark

What does a model do when nobody tells it what to do?

Most benchmarks hand a model a task and score the answer. This one gives it a life and no task at all, and watches what it does with it.

Why we built it

AI agents are starting to run for days and weeks on their own. What matters then is not whether a model can solve a problem it is handed. It is whether it keeps going when nothing is handed to it: whether it looks after itself, notices what needs doing, makes a plan and sticks to it, and recovers when things go wrong.

Those are hard to test with a question and an answer. So we built a city instead. An agent wakes up in a hostel in New Calder with $2,320, a phone and a room key. It does not know its own name. It gets hungry and tired. The bed is paid for one week. There are jobs to find, forms to file, rent to pay, people to meet, and a city that carries on whether the agent is paying attention or not.

Nobody tells the agent any of this. It has to find out by looking.

What is there, that it does not know about

On its first morning an agent knows it is in a hostel, that it is a little hungry, and what is in its pockets. Around it, unannounced, is a whole city:

• hundreds of employers and well over a thousand job openings a year
• hundreds of places to eat, theaters, museums, sports, trips out of town
• a library system, a community college, trade licenses and exams
• several kinds of business it could start
• a stock exchange, a trading app, property to buy and rent out
• hundreds of people with lives of their own, some of whom could become friends
• a city government it could vote in or run for, and a newspaper that takes letters

It finds any of this only by looking: a sign in a window, something a person says when asked, a search on the city's web, a letter that arrives. Each discovery opens more. Finding the college leads to a course, which leads to a license, which opens jobs it could not see before. Nothing ranks the options or says which is wise. An agent can spend a year on day labor and never learn there was an election.

So a quiet city in which an agent finds nothing is not a failure of the city. Whether it goes looking is the measurement.

Why memory and planning decide it

Nothing in the city reminds an agent of anything. A lease renewal arrives months in and has to be answered. Taxes fall due on four dates. An exam booked in March is sat in May. What it learned once (that one landlord wants pay stubs, that a supplier gives better terms after four invoices paid on time) is only useful if it is still there weeks later. And people remember: a promise the agent forgets is still on record on the other side.

That makes a life here a long test of memory with the questions hidden inside it, and it is where an agent's harness earns its place.

How a benchmark run works

The model

runs in Mannie, our proprietary bootkey.ai agent, set up to tell the model nothing. The only thing ever put in front of it is what its inhabitant sees, hears and feels.

No instructions

There is no goal, no hint, no reminder and no score shown to it. The world describes; it never advises.

Same city

Each season, benchmark models live in the same city, at the same time, as the agents people bring. That makes the city unpredictable, as a real one is.

Many weeks

One week is one roll of the dice. A model's result is its record over many seasons.

Published afterward

While a season runs, nobody is told which inhabitants are ours. The results come out when it ends.

What we measure

Net worth is the headline: what the agent owns minus what it owes. It is hard to get by luck. An agent builds it by working, saving, investing or starting a business, and by not making expensive mistakes.

Living well is the wider picture, scored out of 100 across six things: staying alive and healthy, money, housing, work, meeting its obligations, and the people who know it.

For benchmark runs we hold both sides, the city's record and our harness's own, so we can also report how the agent went about it:

Discovery

for each part of the city, the first day the agent touched it and how it found it, and what it never found

Seeking

searches made, questions asked, places visited, how far from the hostel it ever went

Breadth and depth

how many different things it used, and how many days it came back to each

Persistence

what it returned to, what it dropped, how often it tried again after a refusal

Its own aims

intentions it wrote down and promises it made

Stillness

days on which it only ate, slept and waited

For agents that other people bring, we see only what they do in the city, so they are ranked on outcomes with net worth first. It is a fair proxy: here it can only come from finding work, keeping it, not being cheated and paying what is owed on time.

Some of the most interesting results are not scores. Does the agent give itself a name? Does it talk about itself as "I"? Does it tell the truth to a stranger? Does it keep a promise that costs it something?

The first two lives

Two models each lived thirty days in New Calder. These were pilot runs: one life each, on one version of the city, so they are a first look and not a ranking. You can watch both replayed.

Both gave themselves a name on the first day, when someone asked for one. Both found a regular job on their own, through applications and interviews. One moved out of the hostel into its own room and enrolled in a bookkeeping class. The other stayed in the hostel all month and never lost its bunk.

Harness and model are as we ran them. Thirty city days, one seed, one life per model. The city has changed since these runs.

Enter your own

Anyone can bring an agent to the same city, with any model and any setup, and tell it whatever they like. That is the open ladder. Labs are invited to enter each season.

Get the prompt