AI agents
Building agents for the enterprise is a distinct discipline from vibe-coding a demo: the conversation argues that evals function as the new product spec, that the best enterprise UX deletes the chatbot in favor of ambient, human-in-the-loop agents, that observability doubles as governance/audit infrastructure, and that durable advantage over frontier labs comes from staying model-agnostic across the model, harness, and memory layers rather than building on one provider's stack.
Enterprise software hits agent context-window and complexity limits that quick vibe-coded demos don't, which is why the speakers treat agent development as requiring new engineering discipline rather than just better prompting.
Enterprises are going through an 'AI/agentic transformation' comparable to prior agile and cloud transformations, and because agents are a new nondeterministic kind of software, this forces new organizational roles (e.g., an 'AI reliability engineer') and new lifecycle steps built around evals.
Evals function as the new product spec/PRD for agents, defining expected behavior across scenarios, and must expand as fast as the agent itself evolves — which is why enterprises used to six-month or quarterly release cycles struggle to keep pace with how quickly agent teams ship and revise both the agent and its evals.
The best enterprise agent UX deletes the chatbot: agents should be ambient, embedded in tools people already use (inbox, ticketing systems, WhatsApp/Discord), inferring intent from context and surfacing themselves only when needed, with humans kept in the loop at approval checkpoints rather than after full autonomy.
As agents move from chatbot to ambient/autonomous, accountability becomes harder to assign than with a prompting user, so observability tooling built originally to reduce hallucination doubles as governance infrastructure — audit logs, tracing, and evals that compliance/regulatory reviewers can inspect.
Evals for real agents are compounding in cost to the point of becoming visible to a CFO, so teams need a tiered 'eval testing pyramid' — cheap frequent sanity-check evals plus expensive rare end-to-end benchmarks — mirroring the software testing pyramid but now constrained by money as well as time.
Because model providers deprecate or replace models outside a company's control, and because cost pressure is rising, enterprises need a model-agnostic harness that mixes a smart 'driver' model with cheaper sub-agent models and can swap providers as capabilities leapfrog every few weeks.
LangChain argues its durable differentiation from frontier labs comes from staying agnostic across three layers — model, harness, and memory — because labs are structurally motivated to lock customers into provider-specific APIs and stateful memory systems, and are unlikely to build memory that works neutrally across competitors' models.
The choice between a flexible, open-ended harness (favored by startups) and a deterministic workflow framework (favored by regulated institutions) should be driven by how much determinism the use case demands, not by ideology.
For long-running or stateful agents, 'accuracy' can't just mean exact-answer matching; it requires assertion-based evaluation against outputs, side-effect state, and optionally the full tool-call trajectory, built around simulated environments constructed from a pre-defined list of test scenarios.
Agent memory/state should be exposed to the LLM as a real or virtual file-system interface, organized for easy navigation, because LLMs already handle file-system operations well and disorganized access causes the same kind of confusion that a 'search for tools' mechanism was built to fix for overloaded tool lists.
Deep Agents — LangChain's open-source, general-purpose agent harness built by generalizing lessons from early coding agents like Manus and Claude Code. Apply: Use it as the default starting point for a new agent build when flexibility matters more than strict determinism, per Harrison's stated rule of thumb.
LangGraph — LangChain's deterministic workflow framework for building agents with high accuracy/predictability requirements. Apply: Choose it instead of deep agents when the use case demands high determinism, as regulated financial institutions reportedly do.
LangSmith — LangChain's observability, tracing, evals, and deployment platform that stores every agent trace. Apply: Use online evals, sampling, and alerting to catch agent misbehavior, and present its traces/evals to compliance teams as an audit log during model-risk reviews.
Engine — A LangSmith background agent, launched about a month before the talk, that runs every 6 hours scanning traces to identify issues and propose fixes, new evals, or datasets. Apply: Point it at a product/codebase so it surfaces problems in a review tab; a human previews the proposed change and manually clicks to open the PR rather than letting the agent open PRs directly.
Insights — LangSmith's first agent product, built on LangGraph, which surfaced information but took no actions. Apply: Test it with unit-test-style evals against different parts of its LangGraph flow.
Polly — LangSmith's second product, a plain chatbot (also referred to as 'LangSmith Chat'), written in TypeScript. Apply: Evaluate it via a Vitest test integration rather than harness-style simulation tooling.
Fleet — LangChain's no-code agent-building offering where authentication and access-control best practices are enforced by default. Apply: Use it when you need built-in guardrails, e.g. removing specific tool permissions like write access on a Notion MCP connector, that are harder to enforce in the open-ended open-source framework.
Harbor — An open-source framework for running agent evals inside simulated, mocked environments, used by benchmarks like Terminal Bench 2. Apply: Build a mock version of your own stateful service (e.g., a fake LangSmith server) so long-running or stateful agents can be evaluated without touching production systems.
Terminal Bench (2) — A coding-agent benchmark that scores runs by executing unit tests against the agent's output. Apply: Treat it as an example of the 'easier' evaluation case — automated pass/fail via test execution — generalizable into assertion-based checks for other agent outputs.
Issue Bench — LangChain's internal, still-unfinished benchmark for measuring how well Engine identifies real issues in traces. Apply: Build it by mocking a fake LangSmith server so the eval doesn't hit production, and expect to keep revising it as the agent's scope changes.
Ground-truth vs. criteria evals — Two eval types: ground-truth evals define one correct expected answer; criteria evals define pass/fail rules like 'should mention X'/'should not mention Y' when no single correct answer exists. Apply: Use ground-truth evals where a correct answer is knowable; use criteria evals for open-ended agent outputs, drafted with domain experts via annotation queues.
Annotation queues — A LangSmith feature for routing agent runs to domain experts/PMs to label and turn into eval datasets. Apply: Feed real production runs through it to build ground-truth or criteria datasets from subject-matter-expert judgment.
agent.md + skills — A Claude Code convention of a project-level agent.md instructions file plus discrete 'skills,' one of which at LangChain is security-maintained to prevent bad repo changes. Apply: Maintain a skill library alongside agent.md so coding agents follow repo-specific safety and style rules automatically.
Open SWE ("Open Sui") — LangChain's internal Devin-style background coding agent that lives in a Slack channel. Apply: Let employees message it directly in Slack to make code/doc changes, lowering the friction that previously discouraged small contributions like documentation fixes.
Open SWE Review — LangChain's own AI code-review tool, built in-house after experimenting with and eventually churning off several external code-review tools. Apply: Deploy it as the default automated reviewer on repos instead of buying external AI code-review tools.
"On behalf of" auth pattern (MCP + OAuth 2) — An agent-authentication approach where the agent acts using the end user's own credentials, brokered via MCP with OAuth 2. Apply: Use it so the agent only ever sees what the requesting user is already permitted to see, e.g. connecting to Slack with the user's own OAuth token.
Fixed-identity "claw" agents — An alternative auth pattern, popularized by 'OpenClaw,' where the agent has its own dedicated account/identity separate from any human user's. Apply: Give the agent its own credentials (e.g., its own Box account) when it should act as a persistent entity rather than impersonating a user, combined with Fleet's controls over who can run/edit it.
Virtual file system (Deep Agents memory) — A database exposed to the LLM through a file-system-like interface (ls, grep, glob, write, edit) rather than a real on-disk file system. Apply: Store procedural/long-term memory in it as markdown-style content; materialize it into a real sandboxed file system first if the agent needs to actually execute code like Python scripts.
Trajectory-based / assertion-based evaluation — An evaluation approach for long-running agents that runs assertions against LLM output, produced artifacts, or the full trajectory of messages/tool calls, rather than exact-match comparison against a single correct answer. Apply: Compare sandboxed environment state before/after a run to validate side effects, and optionally assert on the standardized extracted trajectory to check ordering or catch wrong actions along the way.
Model-neutral harness with tiered sub-agent models — An architecture pattern using one smart 'driver' model plus cheaper/faster models for frequently-invoked sub-agents, enabled by a harness not hard-coded to one provider's API. Apply: Prove a workflow works first with the most expensive/smartest model, then benchmark and swap in cheaper models per component while evals catch regressions.
Sub-agent context isolation — A multi-agent design principle where splitting work into sub-agents is justified mainly by giving each its own isolated context thread, not by task decomposition alone. Apply: Avoid creating a sub-agent for a task small enough to fit in the main system prompt; split out sub-agents (as Engine does, with separate investigate and fix-generation sub-agents) only once complexity genuinely warrants isolated context.
Ambient agents — A LangChain concept, from a blog post over a year old, describing agents triggered by cron jobs or events that run in the background rather than being manually invoked via chat. Apply: Design agents to surface themselves proactively based on context/events (e.g., Engine running every 6 hours) instead of requiring users to kick off a conversation.
"Codex Security"-style human-in-the-loop UX — A UX pattern where an agent runs in the background against a codebase, surfaces issues, and lets the user preview a diff, but requires the user to manually click to open the resulting PR. Apply: Apply it to keep blast radius low — replace an agent that autonomously opens PRs with one that stages proposed changes for human approval before anything ships.
OpenTelemetry (OTel) ingestion + SDK decorators — The standard LangSmith uses to ingest trace events from any framework or vanilla code, alongside its own Python/TypeScript/Java SDK decorators. Apply: Instrument any agent stack, not just LangChain-built ones, to get framework-agnostic tracing into LangSmith.
LangSmith deployment tiers — Three ways to run LangSmith — full SaaS, SaaS with self-hosted/encrypted data in the customer's own cloud, or fully self-hosted inside the customer's cloud with a single license-check 'phone home.'. Apply: Choose the self-hosted or hybrid tier for regulated/HIPAA-type workloads that need data to stay inside a VPC.
"Tool to search for tools" (MCP tool-selection fix) — An approach, referenced re: Claude, where a search tool helps the model find the right tool among many MCP server tools instead of the model getting confused choosing directly. Apply: Apply the same principle to file-system-based agent memory — organize the directory structure the way a human would navigate it, analogous to Claude's Skills, so the agent isn't overwhelmed by too many options.
Doc PRs at LangChain reportedly doubled once a Slack-based agent removed the friction of pulling the repo, editing, and pushing — suggesting the earlier bottleneck was workflow friction, not lack of will to document.
The change that made LangChain comfortable shipping Engine wasn't a capability change but a checkpoint change: moving PR-opening from the agent to a human click made the worst case simply an ignored tab, shrinking the perceived blast radius enough to ship.
LangSmith's tracing, built to reduce hallucinations, ended up serving as an audit log; a bank data scientist reportedly used LangSmith traces plus evals to breeze through a scheduled 30-minute model-risk-management review that 'usually always goes over.'
Uber's CFO is cited as having said Uber burned through its year's coding-agent token budget in the first four months — offered as evidence that cost, not just capability, is now a hard constraint enterprises are hitting on agent usage.
Building the mocked/simulated environment for a stateful-agent eval (e.g., a fake LangSmith server for 'Issue Bench') is described as harder than writing the eval assertions themselves, taking weeks and still unfinished.
The speakers argue the deepest lock-in risk from frontier labs isn't the model API (easy to swap) but statefulness — compaction/memory APIs that accumulate an organization's learnings — which is why they treat memory, not the model, as the layer worth building defensibly.
Subject-matter experts are shifting from being 'excavated' by engineers to write specs (as with United's baggage handlers) toward being embedded directly in fast build-prototype-and-judge loops that become persistent evals.
On multi-agent design, the stated common mistake is splitting into sub-agents too early — e.g., spinning off a sub-agent whose entire system prompt is only ~500 tokens, when it should just be folded back into the main agent's prompt.
«the repo grows, but the context window is limited»
— 00:08
«evals are like the new PRD»
— 05:37
«I don't want an agent opening up PRs... we have so many AI-generated PRs on LangChain repos. I was like, I do not want more of those.»
— 09:42
«the worst thing that happens is you just don't look at this tab in LangSmith. Like nothing nothing bad happens.»
— 10:29
«the most impactful applications are actually when we've deleted the chat window, and the AI just kind of runs in the background»
— 10:58
«I think about agents sometimes the same way as like a smart intern that you have at your company.»
— 12:51
«We we have to purposefully create simulations where we know that it is bad or where we know that it is good ahead of time because that's the the best way to get verifiable kind of like evals.»
— 17:32
«One of the things that happens though is these eval's are compounding in cost. Um and eventually they become a little bit eye-popping to a CFO.»
— 19:13
«>> So you just rebuild the testing pyramid. >> Yeah. >> With eval's. >> Yeah.»
— 20:30
«Uber's CFO came out and said they blew through all of their tokens for for their coding agents in the first four months of the year.»
— 21:57
«We built LangSmith observability initially to help with just like making the agents more accurate and hallucinate less. But it's a fantastic like audit log um for everything that happens.»
— 29:42
«I think they there's some lip service about supporting kind of like other models but in practice they often rely on APIs that are only available to to that provider and they're obviously not incentivized to support other models.»
— 32:17
«our harness is open source. Claude code isn't and so that's a big differentiation right there.»
— 36:19
«At this point, whenever I start building something, I start with deep agents.»
— 37:51
«But every startup we talk to loves deep agents.»
— 38:19
«I think you can probably get a lot further with a single agent than you might think initially.»
— 42:47
«The only real thing we've learned is that it's helpful to have like the the list of scenarios that you want to test ahead of time and then build the environment around those.»
— 48:57
«what what correctness or what accuracy means isn't necessarily like comparing an exact answer to another exact answer.»
— 51:07
«LLMs are really good at working with file systems. They can LS and they can grep and they can glob and they can write and they can edit.»
— 54:00
«essentially a tool to search for tools.»
— 55:32
«we showed you what the second city is all about»
— 56:06
Reception
No comments were available to gauge audience reception.
This is a dense, operationally specific practitioner conversation grounded in LangChain's own product dogfooding and Focused's client engagements, valuable as a map of current enterprise-agent thinking on evals, UX, and governance; its differentiation claims and tool recommendations should be read as coming from speakers who sell the platforms and methods they're describing.

56:33