Your agent is a while-loop around an autocomplete

You ask your coding agent to add a CSV export to the reports page. It says it’s reading the reports code. Then it’s editing the file, then running the tests, and then it tells you it’s done. You watch the labels tick by and you describe it the way everyone does: it read the code, it wrote the feature.

It didn’t, though. The model never touched your code.

What actually happened: the model wrote some text that said, in a structured format, “I’d like to run this tool with these arguments”. Then it stopped. A separate program (the harness, so Claude Code, Codex, Cursor, whatever you use) read that text, ran the tool on your machine, added the result to the conversation, and sent the whole conversation back to the model for another go. That repeats until the model stops asking for tools.

So the agent isn’t the model. It’s a while-loop wrapped around an autocomplete.

I build agent systems at work, and this is the thing I end up explaining most, to smart people who use these tools every day. So let’s look inside the loop once, properly. It’s small, and honestly most of the confusing stuff about agents comes straight out of it.

The model only ever returns text

When an agent “uses a tool”, this is everything the model contributes. It sends back something like this:

{
  "stop_reason": "tool_use",
  "content": [
    { "type": "text", "text": "Let me look at the reports route first." },
    {
      "type": "tool_use",
      "id": "toolu_01A09q90qw90lq917835lq9",
      "name": "read_file",
      "input": { "path": "app/routes/reports.py" }
    }
  ]
}

That’s it. A blob of JSON saying “please run this for me”. The model has no hands. It can’t touch your files, your terminal or the network. It just predicted that the most useful next thing in this conversation is a request to run a tool, the same way it predicts any other text.

The harness does the real work. It reads the file, grabs the contents and sends a new request to the API. And because the API doesn’t remember anything between calls (the Anthropic tool-use docs are explicit about this: send the full conversation history each time), that new request contains everything again:

{
  "model": "claude-sonnet-5-5",
  "system": "You are a coding agent...",
  "tools": [ ...every tool definition, every time... ],
  "messages": [
    { "role": "user", "content": "Add a CSV export to the reports page" },
    { "role": "assistant", "content": [ ...the tool_use block above... ] },
    { "role": "user", "content": [
      { "type": "tool_result",
        "tool_use_id": "toolu_01A09q90qw90lq917835lq9",
        "content": "@router.get(\"/reports/{report_id}\")\nasync def get_report(report_id: str): ..." }
    ]}
  ]
}

Look at what rides along: the system prompt, every tool definition and the whole history, every single turn. The “agent” logic really is just while stop_reason == "tool_use". Anthropic’s own docs describe the loop in pretty much those words.

Two things follow from that. The model has no memory of your session. It reads the whole transcript from scratch every turn and works out where it is. And the transcript only ever grows.

Once you know the model re-reads the whole transcript every turn, most agent advice stops being folklore. You can work out why “explore the codebase first” helps, and why turn forty costs more than turn four.

Every agent is this loop

Once you’ve seen the shape, you see it everywhere. Claude Code is this loop with shell, file and search tools. OpenAI’s Codex is this loop with its own sandbox and tools. Cursor is this loop wired into an editor. Hermes Agent, the open-source harness Nous Research launched in February 2026 (a harness, not their older Hermes models), is this loop with persistent memory and a very long list of tools bolted on.

Vendors talk about agents like they’re different species. Underneath, it’s the same three parts every time: a model, a harness, and the tools it offers. What changes is which tools the harness hands the model and how cleverly it manages the transcript. I’m not saying that to dismiss it. Managing the transcript well is genuinely hard engineering. But it does mean that when a new “agentic” product launches next month, you already know how it works.

Even the moments that feel most alive are just the loop. When Claude Code stops and asks you a clarifying question, nothing special happened. The model called a tool named AskUserQuestion, the same way it calls read_file. The harness shows you the question, waits, and sends your answer back as the tool result. You’re in the tool list. I find that one genuinely funny.

And when people say agents work better if you tell them to “explore the codebase first, then make changes”, the loop explains that too. The exploring turns are how the relevant code gets into the transcript before the model commits to an edit. You’re loading its working memory.

Where the money goes

Now the part that shows up on the bill.

Look at the picture above again. Four turns, roughly 72k tokens in and about 1k out. Real sessions are a lot longer than that. Here’s one of mine from last week: 1,855 requests, and every one of them sent a longer transcript than the last.

Output tokens have the scarier price (Claude Opus 5.5 is $4 in and $20 out per million; GPT-6.1 Sol is $2 in and $10 out), but agents barely produce any. They write short tool calls and keep re-reading their own growing history. Most of the bill is that re-reading.

The only reason this doesn’t bankrupt everyone is prompt caching, and I don’t think you can make sense of agent costs without it. Both big providers will serve the start of a prompt they’ve already seen from a cache, at about a twentieth of the normal input price. Anthropic charges 0.05x for cache reads on Opus 5.5 (and 1.25x to write to the 5-minute cache). OpenAI caches automatically for prompts over 1,024 tokens, at 95% off for GPT-6.1 Sol. The pricing pages are basically organised around the growing transcript.

Look at my Claude Code bars, though. Cache reads cost a twentieth of normal input, and they’re still more than half the bill. Caching discounts the replay. It doesn’t delete it.

The cache is also very literal. It works on an exact prefix match, so it only pays out when the start of the request is exactly what it saw last time. Change something early in the conversation, or switch models halfway through, and everything after that point is full price again. Leave the session sitting past the cache’s lifetime (usually minutes, unless you pay for longer) and the next turn pays to write the whole thing back in. That’s how a session that’s felt cheap all afternoon quietly stops being cheap. (There are some properly stupid ways to break it by accident, but that’s its own article.)

The habits mostly fall out of this. The length of the transcript is what you’re paying for, so don’t keep a bloated session going out of sentimentality. My session above started at about 37k tokens a request (the system prompt, the tools and my first message). Across the whole session it averaged 391k. That’s ten times as much to re-read on every single turn. A fresh session with a short, clear brief takes you back to the 37k end, and it’s usually sharper too, because the model isn’t wading through hours of old tool output. And in a loop where every turn re-sends history, a model that needs fewer turns saves you money twice: fewer requests, and a shorter transcript in each one.

When you compare models, compare cost per task. Sounds obvious, right? But GPT-6.1 Sol launched at the same $2/$10 price as GPT-6 Sol, and Artificial Analysis measured it (29 September 2026) using 10 to 30% more output tokens. It still came out 31% cheaper per task, $0.72 against $1.05 at max effort. You couldn’t have guessed either number from the price sheet.

When the transcript gets too long

The transcript can’t grow forever. Context windows run out, and so do budgets. So when a session gets long, the harness quietly summarises older turns and swaps the history for the summary. That’s what the four drops in my session are: hundreds of thousands of tokens of history, replaced by a summary of about 65k.

It’s also why agents seem to get vaguer and more forgetful deep into a marathon session. Nothing has actually got worse. The model is reading a compressed paraphrase of decisions it made an hour ago, and some of the detail didn’t make it into the summary. (It’s the same reason agent memory works better as several stores than one, but that’s another article.)

I keep coming back to how small the loop is. Text in, text out, a harness that does the actual work, repeat. Most of what makes one agent better than another is the stuff around the model, because the model itself only ever does one thing.

So next time an agent product launches with a breathless demo, you already know what to ask. How does it manage the transcript, and what does a turn cost at hour two?