Articles

The article articulates the emergence of 'Agent Engineering' as a crucial discipline for successfully deploying LLM-powered agents in production. It highlights the significant gap between development and production for non-deterministic LLM systems, where traditional software development approaches fall short due to unpredictable inputs and behaviors. Agent Engineering is defined as an iterative cycle of build, test, ship, observe, and refine, emphasizing continuous improvement based on real-world insights. The discipline integrates product thinking for scope definition and behavior shaping, engineering for infrastructure and production readiness, and data science for performance measurement and optimization. It's not a new job title but a set of responsibilities taken on by existing teams (engineers, PMs, data scientists) to harness the power of LLMs for complex, high-impact workflows while ensuring reliability.

The article from LangChain's blog details five crucial patterns for evaluating "deep agents," which are complex, stateful AI applications. It highlights that traditional LLM evaluation methods are often insufficient due to agents' dynamic nature and the need for specific, context-dependent success criteria per test case. To overcome these limitations and ensure robust testing, the article outlines five crucial patterns. The first emphasizes bespoke, code-based test logic for each datapoint, allowing specific assertions against an agent's trajectory, final response, and internal state. Secondly, it advocates for single-step evaluations as an efficient way to validate immediate decision-making and tool calls, akin to unit tests. Thirdly, full agent turns are presented as essential for providing a comprehensive view of end-to-end actions, useful for evaluating overall trajectory, final responses, and generated artifacts. Fourthly, multi-turn evaluations simulate realistic user interactions but require conditional logic to manage agent deviations and ensure test consistency. Finally, the importance of clean, reproducible test environments and mocking external API requests is stressed to ensure reliable and efficient evaluations. The article positions LangSmith's testing integrations as a flexible framework to implement these patterns, offering practical guidance for AI developers.

The article presents a robust evaluation workflow for Deep Agents, focusing on systematically optimizing their performance. It begins by outlining the Deep Agent architecture, which includes detailed system prompts, planning tools, filesystem management, and subagents. The core problem addressed is how to rigorously measure and improve agent performance. The solution involves three key steps: first, using Harbor for sandboxed, scalable, and isolated agent execution to ensure reproducibility and safety; second, leveraging Terminal Bench 2.0 as a comprehensive benchmark dataset with automated verification for diverse agent tasks; and third, integrating LangSmith for automatic tracing, detailed observability, and reward feedback, enabling developers to understand agent actions and identify failure patterns. A concrete example demonstrates how analyzing traces can lead to optimizations like reducing environment setup latency. The article emphasizes a data-driven improvement cycle, moving from building and running agents to analyzing performance and iterating on improvements, ultimately enabling systematic agent development.

The article introduces DeepAgents CLI, a model-agnostic, Python-based coding agent designed for interactive terminal use, featuring shell execution, filesystem tools, and memory. It addresses the critical challenge of evaluating coding agents in isolated environments due to their interactive nature and potential for side effects. To solve this, the authors leverage Harbor, a framework enabling scalable, containerized agent evaluations with automatic test execution and reward scoring across various sandbox providers. They specifically developed `deepagents-harbor` to integrate DeepAgents with Harbor. The evaluation was conducted on Terminal Bench 2.0, a benchmark comprising 89 tasks across domains like software engineering, biology, and security, designed to measure agent proficiency in terminal environments. DeepAgents CLI, powered by Claude Sonnet 4.5, achieved an average score of 42.65%, establishing it as a competitive baseline. The article concludes by highlighting future work on systematic trace analysis for performance optimization.