Evaluating Multi-Agent Systems Beyond the Final Answer

Learn how to evaluate multi-agent AI systems beyond just the final answer. This talk details a framework capturing agent collaboration, tool use, and failure points for better system design.

Python OpenAI API Anthropic API LangGraph CrewAI

Overview

I am building an evaluation framework for multi-agent AI systems that compares not only whether the system gets the right final answer, but how the agents collaborate to get there. In the live demo, I will show a small multi-agent workflow with a planner agent, worker agents, and a critic/evaluator agent, then walk through the code, traces, tool calls, handoffs, failure cases, and scoring logic used to evaluate each run.
The framework captures the full agent trajectory: which agent took control, what tools were called, where context was passed or lost, how much the run cost, how long it took, and whether the final outcome matched the task goal. I will demo how the same task behaves under different setups, such as a single-agent baseline, a manager-agent architecture, and a multi-agent handoff design.

Tech stack

Python

Python: The high-level, general-purpose language built for readability, powering everything from web backends to advanced machine learning models.

Python is the high-level, general-purpose language prioritizing clear, readable syntax (via significant indentation), ensuring rapid development for any team . Its ecosystem is massive: use it for robust web development with frameworks like Django and Flask, or leverage its power in data science with libraries such as Pandas and NumPy . The Python Package Index (PyPI) provides thousands of community-contributed modules, offering immediate solutions for tasks from network programming to GUI creation . The language is actively maintained by the Python Software Foundation (PSF), with the stable release currently at Python 3.14.0 (as of November 2025) .

https://python.org

View projects
OpenAI API

OpenAI API: Your direct gateway to cutting-edge AI models (GPT-4o, DALL-E 3, Whisper), enabling scalable, multimodal intelligence integration into any application.

The OpenAI API provides authenticated, programmatic access to a powerful suite of generative AI models. Developers leverage REST endpoints and official libraries (Python, Node.js) to integrate capabilities like advanced text generation (GPT-4o), image creation (DALL-E 3), and speech-to-text transcription (Whisper). This platform is engineered for scale, supporting millions of daily requests for tasks from complex reasoning to real-time customer support agents, ensuring your application gets reliable, state-of-the-art intelligence.

https://platform.openai.com/

View projects
Anthropic API

Programmatic access to Anthropic's Claude models (Opus, Sonnet, Haiku) for complex reasoning, vision, and tool-use applications.

The Anthropic API delivers programmatic access to the Claude model family (Opus, Sonnet, Haiku), enabling developers to integrate state-of-the-art AI into applications. Use the Messages API for conversational tasks, leveraging Claude 3.5 Sonnet for balanced performance or Claude 3 Opus for complex analysis. Key features include Tool Use (function calling), Vision capabilities for image analysis, and a large 200K token context window for extensive document processing. This API provides a powerful, reliable foundation for next-generation AI projects.

https://docs.anthropic.com

View projects
LangGraph

A low-level orchestration framework for building long-running, stateful, and cyclic multi-agent systems using a graph-based architecture.

LangGraph is the specialized, low-level runtime for developing complex AI agents, extending the LangChain ecosystem to handle intricate, stateful workflows. It models the agent's logic as a directed graph: nodes represent actions (LLM calls, tool use), and conditional edges dictate the flow, enabling critical features like cycles (loops) for iterative reasoning. This graph-based approach ensures durable execution, allowing agents to persist through failures and resume operations. Key capabilities include comprehensive memory management via a shared state object and built-in human-in-the-loop functionality (interrupts) for external oversight. This robust framework is trusted by production teams at companies like Klarna and Replit for deploying scalable, resilient agent architectures.

https://langchain.com/docs/langgraph/

View projects
CrewAI

The open-source Python framework for orchestrating autonomous, collaborative AI agent teams.

CrewAI is a lean, lightning-fast Python framework designed for building and deploying multi-agent systems. It empowers developers to define specialized Agents—each with a clear role, goal, and backstory—and organize them into a Crew to tackle complex tasks. The framework manages collaboration via context sharing and delegation, utilizing defined Processes like sequential or hierarchical execution to ensure efficient workflow. Agents gain specific capabilities through integrated features, including access to hundreds of open-source tools (e.g., internet search) and a sophisticated memory management system for short-term and long-term context. You define the agents and tasks; CrewAI handles the orchestration, delivering reliable, scalable AI automation.

https://crewai.com

View projects

Evaluating Multi-Agent Systems Beyond the Final Answer

Related talks