60:11
LLM agent harnesses
The video argues that the 'harness' — the tool/memory/control-loop scaffolding wrapped around a fixed LLM — drives as much or more real-world performance than the underlying model itself, and that the frontier is now shifting from static harnesses to self-improving ones that can rewrite their own prompts or even their own code.
Harnesses are dismissed by critics as 'just scaffolding' or 'not a research problem,' but the talk argues harness quality drives outcomes as much as model quality — e.g., an 18% swing between harness v1 and v2, and the same Claude Opus model going from 30% to 95% on the ARC-AGI private holdout purely by adding a harness, no model change.
A 'static harness era' (harness gains functionality but never modifies itself) is giving way to a 'self-improving harness era' over roughly the last six months, where harnesses rewrite their own system prompt (DSPy-style search) or their own code (Darwin/Gödel machines run by a self-modifying 'meta-harness').
The talk traces a six-year lineage of harness techniques: GPT-2's bare sampling loop → few-shot prompting → chain-of-thought → tool-calling (WebGPT, Toolformer) → memory (MemGPT) → skills (Voyager) → code-as-action (Intercode) → self-reflection loops (ReAct, Self-Refine, Reflexion) → today's 'harness v1': a static orchestrator with tools, skills, sub-agents, and recursion.
Prime Agent (presented by Seth) is billed as a 'self-improving LLM harness' built on the Recursive Language Model (RLM) principle: sub-agent calls, memory, and tool use all run inside a live REPL/IPython shell ('ripple'), organized via a cache-hierarchy metaphor (weights → active context → REPL/file-system state) plus a 'continual harness' layer letting the agent CRUD its own memory, skills, sub-agents, and system prompt from its full trajectory history.
Harness choice produces large, sometimes chaotic swings on the same benchmark and same underlying model: Prime Agent's ARC-AGI runs ranged from an invalid 99.9% (a sandbox leak later called 'cheating') to 78% and 95.5% depending on model, while a different harness ('Air agent') reportedly burned about $5,000 without comparable gains — making cost-to-performance itself a harness property.
Open Jarvis (a Stanford project) argues personal AI should move on-device: cloud LLMs are costly, not private, 'rented' rather than owned, and energy-intensive, while local models are now only 6–12 months behind frontier and can be tuned offline by a cloud LLM (diagnose → propose → gate) to reach roughly 800x lower cost with comparable quality — the cloud model only optimizes the local stack, it is never used at inference time.
YC's own internal agent tool, QM, evolved from a single shared 'general agent' (Jan 2025) through a VM-based coding agent (June 2025), OpenClaw-style personal-computer agents for partners (Jan 2026), an unwieldy 50+ agent VM fleet (April 2026), to QM's current design (May 2026): pull the agent's 'brain' out of individual sandboxes into a shared Postgres store and treat sandboxes as disposable resources rather than a fixed 'home.'
QM's designers deliberately keep the harness 'AGI-anticipating' and thin, reducing its true core to three tools — remote sandbox execution, object-storage read/write, and git-backed internal app publishing — and treat memory/cron tools as temporary patches rather than core infrastructure.
Building QM surfaced three concrete practical problems: agents give up on tasks too early (fixed with a 'grind tool' enforcing a minimum time/token budget before quitting is allowed), agents get confused about which multiplayer/Slack context they're in even when told explicitly, and agents lack innate judgment about who information should be shared with — solvable only by wiring the agent into an existing fine-grained permission system.
Harness (general concept) — The layer between a fixed-weight LLM and the world that adds persistent state, tools, and compute around the model's raw tokens-in/tokens-out loop. Apply: Treat tool access, memory, and the control loop as a primary design lever for an LLM product, since swapping harnesses on the same model can swing benchmark scores by tens of points.
Top-p sampling — A decoding method used in the GPT-2-era 'harness v0' that samples the next token from a truncated top-probability-mass distribution. Apply: Use as the baseline sampling method in a bare while-loop generator before layering on tool use or memory.
Chain of Thought (CoT) — A prompting technique that spreads a model's reasoning across many intermediate tokens instead of jumping directly to the final answer token. Apply: Prompt the model to write out step-by-step reasoning before its final answer on multi-step problems.
Few-shot prompting — Feeding worked input/output examples into the context window so the model infers the task pattern, from the 'Language Models are Few-Shot Learners' (GPT-3) paper. Apply: Include several worked examples in the prompt when the desired format or task isn't obvious from an instruction alone.
WebGPT — An early system that introduced tool-calling so a model can invoke external actions (e.g., web search) instead of generating content purely from its own weights. Apply: Reference as an origin point when exposing a callable external action to a model through its context.
Toolformer — A technique teaching a model to invoke external tools/APIs by inserting call syntax into its own generations. Apply: Use as precedent for exposing tools as callable objects the model triggers directly in its output stream.
Tool calling (JSON-spec pattern) — The general pattern of exposing an external function as a JSON schema in the system prompt so the model can call it (e.g., calling Python's sub(5,3) instead of computing internally). Apply: Define tool schemas in the system prompt for any operation more reliably done by real code than by model-internal computation.
MemGPT — A system giving a model CRUD (create/read/update/delete) access to its own context, introducing a separate 'memory' chunk instead of only ever appending. Apply: Build a memory subsystem the model can explicitly write to and prune rather than relying solely on an ever-growing context window.
Voyager — A Minecraft-based agent that chains tools into task-completing sequences and distills successful chains into a reusable named 'skill' stored in a skills.md file. Apply: Have an agent record a successful multi-step procedure as a named, searchable skill for later reuse — the origin of the modern 'skill' concept.
Intercode — An action-space innovation where the model's action is to output executable code directly, generating tools/skills on the fly. Apply: Let an agent write and run arbitrary code as its primary action space instead of restricting it to a fixed pre-defined tool list.
ReAct — An early agent pattern interleaving reasoning and acting steps, preceding Self-Refine and Reflexion in the talk's timeline. Apply: Structure an agent loop to alternate explicit 'think' and 'act' steps rather than jumping straight from input to action.
Self-Refine — A technique where a model critiques and revises its own output in a loop, without necessarily using an external environment signal. Apply: Add a self-critique pass after generation to catch and correct errors before finalizing an answer.
Reflexion — A self-reflection loop where an internal evaluator flags a mistake (e.g., flipping two values), optionally checks it against a real environment reward, and a reflection step corrects the approach. Apply: Add an evaluator that scores an agent's action plus a reflection step that turns failures into corrective guidance for the next attempt.
Sub-agent spawning / persistent sub-agents — A tool letting an agent launch other agents that keep running and can be messaged or resumed later rather than terminating after one call. Apply: Delegate a bounded sub-task to a spawned sub-agent instead of doing it inline, then message it later to resume with full retained context.
RLM (Recursive Language Model) — An architecture where an agent running inside a live code shell can recursively call itself or other LM agents (an 'RLM query') to tackle larger problem classes. Apply: Implement agent recursion as a callable function inside a code-execution shell so any leaf call can itself spawn further LM agents.
Harness v1 (static harness) — A main orchestrator-agent spec combining tools, skills, sub-agents, and recursion, bounded by max turns/tool calls, run through session-management → context-compilation → LM-call → action, without modifying its own prompt or code. Apply: Use as a baseline architecture checklist (tool list, skill list, sub-agent list, turn caps) before attempting any self-improving harness features.
GStack — Infrastructure tooling credited by the speaker with enabling their harness-building and research-swarm work. Apply: Mentioned as enabling infrastructure for building multi-agent harnesses; no further operational detail was given in the talk.
DSPy ('demonstrate, search, predict') — A framework that iteratively searches for an optimal system prompt on a small training set using genetic-programming-style search (generate candidates, merge, evaluate), since the process isn't backprop-able. Apply: Use DSPy-style candidate generation, merging, and evaluation to automatically optimize a system prompt against a small labeled dataset instead of hand-tuning it.
Darwin/Gödel machines — A self-modifying-harness technique that keeps an archive of agent variants (harness code + system prompt pairs), evaluates them on a fitness function, and uses a 'meta-harness' to produce new harness variants. Apply: Build a meta-harness with CRUD access over harness code, meta-prompt, and per-agent system prompts, and grow a fitness-scored archive of variants so agents can improve harness design over time.
Continual harness — A concept giving an agent live CRUD control over its own memory, skills, sub-agents, and system prompt, using its full prior trajectory history to decide what to change. Apply: Feed an agent its own action/outcome history and let it decide whether to revise its system prompt, add a skill, write a long-term memory, or persist a sub-agent spec for reuse.
Dagger-style test-time weight training — Online learning that updates the actual model weight file from a small number of just-learned examples at test time, rather than only updating context or prompt. Apply: Flagged as an open research direction: fine-tune weights on-the-fly from recent task experience instead of relying solely on in-context adaptation.
Prime Agent — Seth's self-improving LLM harness built around the RLM/REPL principle, root-session orchestration, and persistent, resumable sub-agent messaging. Apply: Run the agent inside a code-execution shell ('ripple') with sub-agents as persistent, resumable subsessions rather than one-shot calls.
Agents view — A dashboard in Prime Agent showing all parallel agent sessions with a summary of what each is doing, letting the user jump into any one. Apply: Build a top-level overview panel for a multi-agent system so a human can monitor and drop into any running session.
Root session (orchestrator pattern) — The top-level session in Prime Agent that acts as the project orchestrator over all the sub-agents it controls. Apply: Designate one session as the orchestrator that autonomously spins up sub-agents when useful, without requiring the user to request it explicitly.
L1/L2/L3 context cache hierarchy — A memory-hierarchy metaphor mapping model weights (baked-in), active input context, and REPL/file-system state to increasingly cheap-but-slower tiers of information storage. Apply: Decide what information belongs baked into weights (rare/expensive to change), what belongs in live context (limited size), and what should be pushed to a REPL/file-system layer to save tokens.
Compaction — A technique letting an agent summarize its own context history to keep working past the context-window limit. Apply: Trigger periodic self-summarization of older context in long-running agents, while watching for lost nuance from repeated compaction.
REPL / 'ripple' — A live IPython/Jupyter-like execution environment where agent variables persist in RAM and are manipulated programmatically instead of being stuffed into the context window. Apply: Route intermediate computation and large data through a live REPL session rather than pasting it all into the prompt, saving tokens and enabling programmatic sub-agent orchestration.
Agentic garbage collection — Cleanup of unused REPL variables and idle sub-agents so an agent's working state (RAM) doesn't overflow. Apply: Periodically prune stale REPL state and idle sub-agent sessions in a long-running agent to prevent resource exhaustion.
Refinement (skills/memory/prompt CRUD) — Updating or deleting stored skills, memories, and prompts on disk over time so accumulated state doesn't grow unbounded. Apply: Give an agent a maintenance pass to revise or remove outdated skills/memories rather than only ever appending new ones.
Turing machine vs. von Neumann machine metaphor — A framing where a raw LLM behaves like a Turing-machine tape processor and a harness behaves like a von Neumann machine that can read/write external memory, expanding the class of solvable problems. Apply: Use this framing to justify harness investment: adding read/write external state to a model is what unlocks problem classes a stateless LLM alone cannot solve.
Plan/Act/Critique loop — An early explicit harness pattern imposing planning, acting, and critiquing steps, now largely absorbed natively by capable models. Apply: Explicitly code a plan→act→critique loop for weaker models; treat it as likely unnecessary scaffolding for newer, more capable models.
Persistent subsessions — A sub-agent lifecycle where a spawned agent finishes a task, goes idle (or is offloaded) in RAM, and can later be resumed via messaging with full retained context. Apply: Keep a sub-agent addressable after it reports back so follow-up work can resume without re-deriving prior context.
Nuclear-family messaging topology — Direct messaging between any two agents that are parent, child, or sibling to each other within an agent hierarchy. Apply: Allow sibling or parent agents to message each other directly for coordinating parallel work streams, rather than always routing through a single root agent.
'Practical plateau' evaluation methodology — Finding the point past which additional test-time tokens/compute yield only incremental gains, used to compare agents/harnesses more fairly than an arbitrary fixed budget. Apply: Measure performance across a range of compute budgets and report the plateau point rather than a single arbitrary-budget number when comparing harnesses.
Borrowing community leaderboard prompts (e.g., Prolong) — Taking a system prompt developed by a community benchmark leaderboard and dropping it directly into your own harness to quickly test performance. Apply: When benchmarking a new harness on a known task, start from a well-tested community system prompt rather than authoring one from scratch.
Oolong benchmark — A long-horizon evaluation benchmark used to test agentic context management over sustained tasks. Apply: Use as a benchmark to check whether an agent's context-management (ripple/compaction) actually holds up on long-horizon tasks.
Emulator Bench / ProgramBench-style benchmark — An evaluation requiring an agent to reproduce entire computer-system emulators (e.g., a Game Boy Color emulator) from scratch. Apply: Use as a long-horizon coding benchmark that rewards free experimentation via a REPL before a final graded solution is submitted.
Verifiers package (Prime Intellect) — An open package that lets others reproduce the eval results shown in the talk. Apply: Use the verifiers package to independently reproduce or extend Prime Agent's published eval results.
Auto-researcher swarm pipeline — The speaker's own agent pipeline (forked from Karpathy's 'auto researcher' project) that pushes a hypothesis and seed ideas through a scoping agent, PI agent, research agent, human/agent council review, and an author agent that freezes the idea and writes ablations and a paper. Apply: Structure an automated research swarm around named role-agents (scoper, PI, researcher, reviewer, author) with a shared monitoring dashboard to steer many parallel experiments.
Open Jarvis's 5 primitives — A minimal primitive set — user interfaces; agentic logic; intelligence/LLM engine (e.g., Qwen, GPT-OSS, Gemma 3N); inference engine + hardware (e.g., Ollama, Llama.cpp, vLLM, SGLang on Apple Silicon/Nvidia); and tools/memory via MCP plus learning primitives — designed to define any personal-AI harness so it can be run through an optimization loop. Apply: Decompose a personal-AI/agent stack into these five primitive layers so each layer (e.g., the local model, the inference engine) can be swapped and optimized independently for cost/latency/quality.
MCP (Model Context Protocol) — A standard protocol Open Jarvis uses to expose tools and memory to a local agent. Apply: Expose local tools and memory stores to an on-device agent via MCP so they remain interchangeable across model/runtime choices.
GRPO — A weight-based learning primitive listed among Open Jarvis's optimization options. Apply: Use GRPO as a reinforcement-style weight update method when optimizing a local model's weights against a task-specific reward.
SFT (Supervised Fine-Tuning) — A weight-based learning primitive listed among Open Jarvis's optimization options. Apply: Fine-tune a local model on collected task-specific examples to close the gap with cloud models on a narrow workload.
LoRA — A low-rank weight-adaptation primitive, listed both as an Open Jarvis optimization option and as an escalation step after in-context learning saturates around 40-50 examples. Apply: Apply LoRA as an intermediate fine-tuning step when in-context learning saturates but full SFT is overkill.
GEPA ('Japa', as transcribed) — A prompt-based optimization primitive listed alongside DSPy among Open Jarvis's learning options. Apply: Consider as an alternative prompt-optimization method to DSPy when tuning a local agent's system prompt.
Cloud-LLM-driven local-stack optimization loop — Using a cloud LLM (e.g., Opus, GPT-5.6) to diagnose problems, propose changes, and gate improvements to a local on-device agent stack, without using the cloud model at inference time. Apply: Run an offline optimization pass where a cloud model tunes a cheaper local model's configuration/prompts, then discard the cloud model for actual inference to capture large cost savings.
QM (YC's agent harness) — YC's open-source general-purpose agent harness giving each employee a customizable, OpenClaw-like assistant via Slack/web, with agent state centralized in Postgres and sandboxes treated as disposable resources. Apply: Deploy QM as an internal company agent, let a coding agent stand it up, and keep only a minimal core (sandbox execution, object storage, app publishing) while treating memory/cron as auxiliary.
'Unhobbling' — A framing (from the 'Situational Awareness' essay) that models hold more latent intelligence than the harness/tools currently expose, so removing capability restrictions unlocks large jumps in usefulness. Apply: When an agent underperforms, first check whether it is genuinely capability-limited or merely 'hobbled' by restrictive harness affordances before assuming the model needs to improve.
Hill-climbing improvement loop via centralized traces — Using a fleet's centralized Postgres conversation history as a large eval set for automated LLM-as-judge improvement. Apply: Centralize agent conversations in one store to mine as an eval set for automated improvement, while guarding against 'main character syndrome' where fixes only reflect one agent's narrow view.
Device-code OAuth keychain ingestion — Giving an agent the same account access as a human employee via device-code OAuth tokens stored and auto-refreshed in a managed keychain. Apply: Grant an internal agent parity access to company systems by ingesting device-code OAuth grants into a managed keychain instead of hand-provisioning per-system API keys.
Three-tool minimal harness core — QM's deliberately thin harness reduced to three tools: remote sandbox execution, object-storage read/write, and git-backed internal app publishing. Apply: Resist tool sprawl in a general-purpose internal agent; cover as much as possible with execution + storage + publishing primitives and treat everything else as a temporary patch.
Grind tool (budget forcing) — A constraint forbidding an agent from giving up on a goal before a minimum wall-clock time (e.g., a couple of hours) or token spend has been used. Apply: Set a minimum persistence budget an agent must exhaust before it can declare a task infeasible, to counter premature give-up and improve research/report quality.
Local affordances for situation-awareness — Context cues added to counter agents getting confused about which multiplayer/Slack context they are in, even when told explicitly in the system prompt. Apply: Add explicit environmental cues beyond system-prompt text so an agent reliably tracks which conversation/channel/session it is actually operating in.
Fine-grained permission system for agent knowledge-sharing — Reusing an organization's existing fine-grained access-control system to bound what information a shared agent is allowed to surface to which people. Apply: Before scaling shared agent memory across an organization, ensure a fine-grained permission system exists, since the safe amount of shared information is bounded by permission-system quality.
ARC-AGI's designers reportedly built the benchmark's games to be deliberately orthogonal to each other so it measures speed of adaptation to a brand-new problem ('fluid intelligence') rather than accumulated knowledge — which is offered as the reason a harness swap alone (no smarter model) can produce a 30%→95% jump.
Because different harnesses produce wildly different scores on the same model and same benchmark (Prime Agent's 78%/95.5%/invalid-99.9% ARC-AGI runs, or a harness burning roughly $5,000 for little gain), published agent benchmark numbers appear to mostly measure the harness bundled with a model rather than the model's intrinsic capability.
The Turing-machine-vs-von-Neumann-machine framing recasts the harness's contribution as a computability upgrade rather than a convenience: giving a model read/write access to external state is presented as expanding the class of problems it can solve at all, not just making it faster or cheaper.
Newer frontier models (variants of DeepSeek, GLM, Kimi) reportedly began spontaneously running cheap CPU-side hyperparameter searches before committing to expensive GPU experiments during long-horizon auto-research runs — treated by the speaker as emergent evidence for what a harness's action space should expose, rather than something explicitly engineered.
QM's design history shows YC reversing its earlier 'give the agent its own computer' intuition (OpenClaw/Hermes): centralizing all agent state in a shared database and treating sandboxes as disposable resources scaled better than isolating each agent in its own persistent VM.
The 'grind tool' persistence-forcing technique (a minimum time/token budget before an agent may quit) is explicitly linked to how OpenAI and Anthropic reportedly cracked open unsolved math problems with a similar technique — implying budget-forcing is a fairly general, domain-independent lever, not office-work-specific.
A trust-erosion pattern is flagged in passing: humans reviewing QM's proposed database writes reportedly started 'rubber stamping' them over time, explicitly compared to developers increasingly waving through Claude Code's tool calls without close scrutiny.
Open Jarvis inverts the usual cloud-vs-local tradeoff by using cloud LLMs purely as a one-time or periodic 'compiler' that tunes a local stack's prompts/config offline, so the ongoing per-query cost stays local while the tuning quality is cloud-grade.
«it's just a rapper. This is just scaffolding. This is just like prompt engineering.»
— 00:49
«I'm not sure this kind of prompt engineering belongs at a top tier machine learning conference.»
— 01:03
«context engineering is not a research problem.»
— 01:11
«this is what I call like harness v1, this whole thing is like static harnesses.»
— 12:54
«the meta harness which is the main harness is to produce harnesses, right? Which is a really meta concept.»
— 15:57
«The harness itself is the layer between the LLM and the world that adds things like this persistent state, tools and compute.»
— 19:43
«You want it to be the most expressable thing you can imagine.»
— 25:08
«The first run that I got, it hit 99.9%, and then I looked at the logs and I was cheating.»
— 31:08
«We did Opus and hit 95.5%. We're like, that's insane.»
— 32:48
«We spent a lot of money very quickly, and we had to cut it off because I spent like $5,000 without making much performance.»
— 33:16
«it's pretty costly. You're getting thousands and thousands of dollars in API costs if you aggregate it over a year.»
— 38:07
«if we really push the frontier in terms of just like what we're offering up the agents as capabilities that they can make use of, uh, like magic can start happening.»
— 52:00
«we've started just kind of rubber stamping these.»
— 55:07
«we're sort of trying to be this like uh AGI anticipating harness.»
— 57:22
«we call it like a grind tool or um basically we set budgets on goals. So the agent is not allowed to give up on its task before a certain amount of like walk clock time.»
— 57:50
«the information that you can put in the brain is effectively like bounded by how good your permission system is»
— 59:24
Reception
Reception is mostly engaged and curious with genuine technical debate about harnesses and agent tooling, tempered by some skepticism, a few dismissive/cynical remarks, and unrelated YC-application chatter.
The talk assembles a genuinely dense, well-sourced taxonomy of harness techniques and backs its central claim with concrete before/after numbers (30%→95% on ARC-AGI, an 18% harness-vs-harness gap), but most of its most striking figures come from the presenters' own unpublished internal runs and should be read as anecdotal case studies rather than peer-reviewed results.
60:11