Traces, Not Outputs: A Nightly Improvement Routine for a Lead-Gen Agent

Discover a nightly routine to improve lead-gen agents. This talk details a two-month process of benchmarking, trace analysis, and engineer collaboration to identify and fix structural defects.

Claude Opus Langfuse LLM LinkedIn GitHub

Overview

An improvement routine for a production lead-gen agent, running nightly for two months.

A scheduled task pulls yesterday’s real user searches out of the production database and runs each genuinely new one through a bench: the same brief goes to two independent engines, every returned lead is verified against a live LinkedIn profile, and then AI agent reads the entire Langfuse trace of both runs — reasoning, tool calls, search arguments, the candidate arms that got discarded — not just the leads that came out the end.

Benchmarking agent triages both traces and writes up structural defects as numbered findings. But nothing becomes work until the engineer who owns that engine and I agree the finding is real (HITL), and nothing closes until a fix ships in the agent repo and passes a regression re-test on a later run. The reading is automated. The judgment is not.

Links

https://onsa.ai
Onsa.ai automates B2B sales workflows using CRM-integrated AI agents.

Tech stack

Claude Opus

Claude Opus: Anthropic's flagship large language model, delivering frontier intelligence for complex reasoning, advanced coding, and autonomous agentic workflows.

Claude Opus is Anthropic's most capable foundation model (LLM), setting the industry benchmark for complex reasoning, math, and coding. It achieves state-of-the-art results on key evaluations: Opus 4.1 scored 74.5% on SWE-bench Verified. The model features a massive 200,000-token context window (expandable to 1 million for specialized tasks), enabling deep, multi-file analysis and long-horizon agentic workflows. Deploy Opus for enterprise-grade automation, complex financial forecasting, or expediting R&D across critical sectors.

https://claude.ai/

View projects
Langfuse

Langfuse is the open-source LLM engineering platform: gain full observability, manage prompt versions, and run production-grade evaluations.

Langfuse delivers the essential LLM engineering stack. It's an open-source platform for full-lifecycle management of your AI applications (agents, chains, etc.). Use its comprehensive tracing (OpenTelemetry-based) to debug complex, non-deterministic interactions and track exact cost/latency metrics. The system provides robust prompt management (versioning, A/B testing) and flexible evaluation tools to measure output quality and monitor production health. Integrations are native: connect with Langchain, OpenAI, and LlamaIndex via Python/JS SDKs for immediate control and clarity over your LLM deployment.

https://langfuse.com/

View projects
LLM

Large Language Models (LLMs) are deep learning models, built on the Transformer architecture, that process and generate human-quality text and code at scale.

LLMs are a class of foundation models: massive, pre-trained neural networks (often with billions to trillions of parameters) that leverage the self-attention mechanism of the Transformer architecture (introduced in 2017) to predict the next token in a sequence. Trained on vast datasets (e.g., Common Crawl's 50 billion+ web pages), these models—like GPT-4, Gemini, and Claude—acquire predictive power over syntax and semantics. They function as general-purpose sequence models, enabling critical applications such as complex content generation, language translation, and automated code completion (e.g., GitHub Copilot). Their core value: generalizing across diverse tasks with minimal task-specific fine-tuning.

https://en.wikipedia.org/wiki/Large_language_model

View projects
LinkedIn

LinkedIn is the world's premier professional network: It connects over 1 billion members across 200+ countries, driving career development and recruitment for global businesses.

This platform, launched in 2003 and owned by Microsoft (since 2016), is the definitive hub for professional identity and opportunity. Users build digital CVs, network with peers, and follow 67 million company pages: It is a critical talent pipeline where, according to official statistics, six people are hired every minute. LinkedIn’s core function is facilitating job search (65 million weekly searches) and B2B connections, making it the essential digital resource for the global workforce.

https://www.linkedin.com

View projects
GitHub

Host Git repositories and enable massive-scale collaboration (pull requests, issue tracking) for over 100 million developers.

GitHub is the world's dominant web-based platform for Git repository hosting and collaborative software development. Built on Linus Torvalds' Git version control system, the platform facilitates 'social coding' by providing essential tools like pull requests, forking, and issue tracking. It currently serves over 100 million developers, managing a massive ecosystem of public and private codebases. Microsoft acquired the company in 2018 for $7.5 billion, solidifying its role as the central hub for open-source and enterprise-level version control.

https://github.com

View projects

Traces, Not Outputs: A Nightly Improvement Routine for a Lead-Gen Agent

Related talks