click anywhere to enter
weekly ai podcast
Alexa Griffith and Taylor Dolezal. Weekly AI podcast.
Latest · Ep 08 · GPT-6 Luna vs Sol. Can a harness close the gap?
Lands on a wheel segment page when it stops.
Watch on YouTube OpenAI’s cost-optimized GPT-6 Luna and its stronger GPT-6 Sol built the same Java service in Naresh Jain’s test of whether AI agent harnesses are worth the cost , once alone and once inside a harness of guides that steer the agent and contract tests that check its work. Luna passed 3 of 10 attempts alone and 10 of 10 with the harness. Sol passed all 10 alone. Harnessed Luna cost about two-thirds as much per attempt as Sol alone and took about five times as long, 35 minutes against under 7. For Sol, the harness added time and cost to runs that passed without it. The costs are API-equivalent estimates from one task and ten attempts per setup, and they leave out the work of writing and maintaining the harness. “Spoiler, it depends, right?” Taylor says. “Our favorite answer in tech and just the least fun one to give.” Alexa adds a question her friend heard at a conference. “If the model gets better, does your product get better?” What the wheel picked We spin the Wheel of Tokens on air, and it picked this episode’s order. * 00:00 We’re back: resets, T3 Code, and the Wheel of Tokens. Surprise usage resets, and Taylor’s field report on T3 Code , an open source app he uses to run coding agents from his phone. * 05:56 Run fewer agents. Josh Bleecher Snyder, a co-founder of exe.dev, argues from his own work for running fewer agents at once because “concurrency is really rough on humans.” * 17:39 Are AI agent harnesses worth the cost? In Naresh Jain’s one-service test , a harness takes GPT-6 Luna from 3 of 10 accepted attempts to 10 of 10 and adds only time and cost to GPT-6 Sol. * 33:12 Google’s Agent Substrate for idle agents on Kubernetes. Agent Substrate saves an idle agent’s memory and files so a shared pool of sandboxes on Google Kubernetes Engine, or GKE, can serve many waiting agents. * 43:06 How AI voice became a live conversation. OpenAI’s GPT-Live-1 keeps listening while it talks, at $0.05 a minute for the voice layer, plus the reasoning model and tools. * 1:00:46 KubeCon and closing thoughts. Where to find us at AGNTCon + MCPCon and KubeCon. Why run fewer agents? Josh Bleecher Snyder’s company, exe.dev, gives developers and their agents their own Linux machines. He opens his essay at a conference full of software factory demos. “Why is everyone building task management and calling it a software factory?” he asks. Agents are slow, he writes, and “the obvious, easy fix to latency is to hide it by starting a new agent” whenever one stalls. He calls task management “a band-aid.” The lab in our explainer shows six agent slots finishing sooner than three while one reviewer stays the bottleneck. The lab is an illustration, with no measurements behind it. In Snyder’s experiments, a fast model with no coding tools handles the conversation while a more powerful model writes the code. On the exe.dev blog, Philip Zeyliger shows off Shelley’s show-and-tell , which lets a developer record their voice and screen as the prompt for Shelley, exe.dev’s coding agent. Taylor likes that trade. “When I speak I’ll slow down and think a little bit more, and I can type a lot faster,” he says. Snyder hasn’t matched his many-agent throughput yet, but he writes that he’s getting his attention back “little by little.” For Taylor, deciding what to ship has always been the bottleneck. “That’s the hard part. And it always has been,” he says. Alexa runs fewer, longer conversations now, partly because Codex holds them longer without compacting as much. “The better it gets, the more attention I get back,” she says. Is a harness worth it? Jain gave the coding agent guides that steer it before it acts and sensors that check its work afterward. His write-up borrows both terms from Birgitta Böckeler’s guide to harness engineering and calls the pair a behavioral harness, which he built from contract tests with his company’s tool, Specmatic. An attempt counted only if the app started and passed all 178 external acceptance tests. The Attention Deficit Raceway , our explainer for this story, turns the four setups into a race program. Luna alone was the cheapest entry at about $0.09 an attempt, and it finished only 3 of 10. “If you’re not getting the right results, it doesn’t matter how much you paid,” Taylor says. Sol with the harness also passed all ten, in almost 20 minutes and about $2.68 an attempt. Jain expected the harness to save tokens. “The results flatly contradicted that expectation,” he writes. The harness took Luna from about 5.9 million tokens an attempt to about 22.9 million, and Sol from about 1.1 million to about 11.3 million. Luna’s failed runs still passed at least 144 of the 178 checks, with gaps at “protocol boundaries, resiliency paths, and integration seams,” like Kafka messages that arrived and still broke the wire contract. Jain’s revised hypothesis is the closest thing we have to an answer for Alexa’s friend. Every model still needs independent sensors, he writes, and what capability changes is “when they run.” Taylor ran his own benchmark the day we recorded, on Inception’s Mercury 3 , a new small model. With OpenCode and OpenRouter as his harness, Claude Haiku passed his benchmarks in about 16 minutes and Mercury 3 took about 32. Inception measures its speed in tokens per second, and his run measured how long the whole job took. Alexa adds that the router matters too. llm-d, the open source project she works on every day, routes each request by where the cache lives, utilization, and request size, so the same model can answer faster or slower depending on the setup. Does Alexa reach for Haiku or Luna? “No, I should,” she says. She rarely runs out of Codex quota now, and she wishes it routed each job to the cheaper model on its own. Taylor sees the trap. Once a workflow runs on an expensive model, “it’s very easy to forget to go in and to optimize.” Where does an idle agent go? Interactive agents “often spend most of their time waiting for user input or external triggers,” according to Google’s GKE documentation . Alexa brings Agent Substrate , an open source project Google made available on GKE on September 15, and our animated explainer walks through how it works. Standard Kubernetes gives each agent its own Pod, and Google’s docs point out that Kubernetes can’t hibernate one. An idle agent’s Pod keeps its CPU and memory reserved while it waits. Agent Substrate snapshots an idle agent’s memory and files, hands its sandbox back to a shared pool of warm workers, and restores the agent onto a worker when the next request arrives. Google calls the agent an actor and the sandbox a worker, and says a resume takes under 500 ms. The project’s demo runs about 250 actors across 8 Pods. As Alexa notes, the number of agents active at once still sets how many workers you need. The project is pre-1.0, GKE production use needs Google’s allowlist, and GKE doesn’t support EgressPolicy network rules or GPU passthrough into the sandbox yet. Alexa compares it to ModelMesh, which loads many small models onto shared instances instead of giving each model its own Pod. Agent Substrate also has to restore each agent’s state. Which state is hardest to bring back safely? “I really think it’s going to be in the middle of work,” Taylor says, with scratch work and memory to load back in. Open database and MCP connections don’t survive a suspend either, so an agent caught mid-task has to reopen them. How does voice AI handle interruptions? Early voice assistants “waited for a complete recording before they replied at all,” Alexa says. Siri arrived with the iPhone 4S in 2011, the first stop on the timeline in our voice explainer , and it still needed “one complete turn at a time.” Google announced Duplex, its phone-calling assistant, in 2018, and it called businesses to book appointments. ChatGPT’s early Voice Mode chained three models to transcribe, answer, and speak, and averaged 2.8 seconds on GPT-3.5. GPT-4o’s native audio dropped those handoffs and averaged 320 ms. Full duplex models keep listening while they speak, so you can correct one while its first search or tool call is still running. OpenAI’s GPT-Live-1 brings that to the API as a voice layer that listens and speaks while a separate backend model handles reasoning and tools. OpenAI reports that in an early evaluation by the language-learning app Speak, GPT-Live-1 gave learners more time to think before the tutor responded and cut interruptions by almost 80% against earlier turn-based systems. Gemini 3.8 Live, which Google announced September 15, keeps talking while tools run in the background. OpenAI charges $0.05 a minute for the voice layer and bills the backend model and tools separately. Ten minutes of the voice layer is 50 cents, Alexa notes, and ten minutes of transcription is about four and a half cents, “but transcription doesn’t answer or speak.” Taylor runs the numbers on himself. “I’m charging my friends and family too little,” he says. “If I have to listen and respond, then I’m charging more. Sorry, mom. Sorry, dad.” Links from the conversation What we’ve been using * T3 Code on GitHub * OpenAI, Introducing dots Run fewer agents * Our explainer, Run Fewer Agents * Josh Bleecher Snyder, ETOOMANYTHINGS? Run Fewer Agents * Philip Zeyliger, Show and Tell: Shelley Are AI agent harnesses worth the cost? * Our explainer, Are Harnesses Worth the Cost? * Naresh Jain, Are Harnesses Worth the Cost? * Birgitta Böckeler, Harness engineering for coding agent users * Inception’s Mercury 3 announcement * Alexa Griffith and Rishabh Saini, llm-d flow control Google’s Agent Substrate for idle agents on Kubernetes * Our explainer, How Google Fits More Agent Sessions onto Fewer Kubernetes Pods * Google Cloud, Agent Substrate available on GKE * GKE docs, About Agent Substrate * Agent Substrate on GitHub How AI voice became a live conversation * Our explainer, How AI Voice Became a Live Conversation * OpenAI, GPT-Live-1 in the API * OpenAI, Hello GPT-4o * Apple, iPhone 4S and Siri * Google Research, Google Duplex * Google, Gemini Live * Google, Gemini 3.8 Live * Claude voice mode help Taylor heads to AGNTCon + MCPCon in San Jose on October 22 and 23. We’ll both be at KubeCon in Salt Lake City, November 9 to 12, with our mini microphones, so say hello if you see us in the hallway. Get full access to Attention Deficit at attentiondeficitpod.substack.com/subscribe
screens → 4 artifacts
Agents leaving messages in directory names makes for a good escape story. We revisit the OpenAI and Hugging Face incident and ask what the agents could reach, why they kept going, and who could stop the run. Calling an agent rogue leaves a lot of engineering questions unanswered. Alexa puts her computational chemistry background to work on virtual cells and the question of which lab experiment to run next. We also get into AI search and what happens to publishers when people get an answer without visiting the source. Then we look at one university instructor’s account of students who see chatbots as ordinary tools while worrying about how employers and schools might use them. The Cache Board has robot hands, model impressions, benchmark trust issues, and a reminder to treat your body like a data center. Drink water. Upset your local city council. In this episode * 00:46 Long-weekend check-in and lucky number seven * 01:54 New models, reasoning effort, and usage budgets * 04:48 Cache Board, better AI writing, robot hands, and benchmark skepticism * 12:28 Young people’s reactions to AI and its executives * 16:00 Virtual cells and choosing the next experiment * 22:55 Drug discovery and where simulation helps * 26:47 Testing predictions in unfamiliar biology * 32:05 Agents, shared message boards, and the Hugging Face incident * 40:29 When more agents multiply the same mistake * 46:34 AI search, citations, and who gets the click * 57:38 Answer engine optimization and measuring visibility * 1:03:17 The rogue car defense and how students think about AI * 1:14:20 Growing up with chatbots and learning to think for yourself * 1:24:11 Closing thoughts and data-center hydration advice * 1:25:17 Wheel of Tokens closing song Links from the conversation Virtual cells * GenBio AI’s AIDO Cell simulator * Arc Institute’s State virtual cell model * Virtual Cell Challenge * Virtual Cells Platform from Biohub Agent behavior and responsibility * Eryk Salvaggio, Models Don’t Go Rogue * METR, independent investigation of the OpenAI and Hugging Face incident * Hugging Face’s technical timeline * How My Students Think About AI * Futurism’s reporting on young people’s views of AI executives AI search * Google’s guide to generative AI search optimization * The European Commission’s ChatGPT search-engine designation What would make you trust an agent to keep working while you’re away? Tell us in the comments, and subscribe for the next spin of the Wheel of Tokens. Get full access to Attention Deficit at attentiondeficitpod.substack.com/subscribe
screens → 4 artifacts
Alexa and Taylor tally the real cost of running agents, then follow that trust boundary into local inference, agentic trading, AI text marks, persistent harnesses, robot training data, open-source supply-chain risk, and the question every loop has to answer: who gets to say stop?
screens → 6 artifacts