🚀 Best Practices in Evals & Observability Workshop 🎉

🔥 Due to popular demand, we’ve added a second night for this workshop: TUESDAY! Same content, additional day to accommodate folks from the waitlist. 🔥

Building LLM-based applications is exciting, but keeping them reliably performant in production is a challenge—even for the best teams. How do you track user interactions? Diagnose issues? Prevent regressions when prompts or models change? At Weights & Biases, we’ve worked with hundreds of companies to tackle questions like these.

Join us for this exclusive workshop, where we’ll guide you through building and evaluating an AI application using a proven, future-proof workflow, learning how to:

• Log and analyze system behavior.
• Collect and leverage user feedback.
• Design evaluations to identify and resolve issues.
• Confidently fine-tune and optimize performance.

No prior experience with evaluations? No problem. Just bring your laptop, and leave with practical skills to future-proof your AI applications.

📅 Event Details

• When: January 14, 2025 (Tuesday)
• Time: 5:00 PM - 9:00 PM
• Where: Downtown Seattle (Address available to confirmed attendees)

🚀 Agenda

• 5:00 PM - 5:30 PM: Arrival and Food 🥪
• 5:30 PM - 8:00 PM: Workshop: Best Practices in Evals & Observability
• Understanding evaluation metrics for AI models
• Hands-on Lab: Building evaluation pipelines
• Interactive Session: Implementing observability tools
• Real-world case studies and troubleshooting
• 8:00 PM - 8:30 PM: Q&A + Open Discussion
• 8:30 PM - 9:00 PM: Networking

💻 Bring your laptop: you’ll be building real evaluation pipelines and implementing observability tools throughout this hands-on workshop. Setup instructions and prep materials will be provided in advance.

🌟 Who Should Attend? 🤖

This session is curated for AI developers, data scientists, and engineers who:

Are actively building or managing AI pipelines

• Want to level up their evaluation practices, whether you’re currently going by vibes or using sophisticated metrics
• Are looking to improve how you track and analyze the results you’re getting from large models in production
• Are interested in learning about emerging tools and techniques in this rapidly evolving field

🌟 The reality is: many developers are still evaluating models based on ✨vibes✨ and tracking results in 📊 spreadsheets - we welcome you to this session to get a 🚀 boost and a level-up! Whether you’re just starting to formalize your evaluation process or already have robust systems in place, this session will help you take your next step forward. 🎯

Bring Your Laptop: Code along with live examples and leave with practical tools.

Whether you’re looking to improve your model evaluation processes or enhance your observability practices, this workshop is for you. Gain actionable insights, connect with like-minded peers, and explore the future of AI development.

👑 Sponsor

This event is sponsored by Weights & Biases Weave. When you’re ready to take your tinkering to production, you’ll need reliable logging and tracing, Weave will get you there with 3 lines of code, and you’ll get a beautiful dashboard you can share with even the non technical folks! Plus, once you’re ready to upgrade to o1 or gpt-next, Weave offers a robust LLM evaluation pipeline so you’ll always know which LLM works best for your use-case.

⬆️ Space is very limited for this hands-on session. Please register now! ⬆️

AI in Production Event Series

The AI in Production series by AI Tinkerers explores the realities of deploying AI systems at scale. By inviting experts and experienced practitioners to share their insights, the series emphasizes best practices, advanced tooling, and real-world applications—helping teams close the gap between experimentation and business-ready systems that create value.

About AI Tinkerers

AI Tinkerers is an exclusive global network of top AI talent hosting in-person events in over 60 cities worldwide. We bring together active builders, practitioners, and experts who are at the forefront of AI development, particularly in areas like large language models and generative AI. Our community is dedicated to fostering technical innovation and collaboration among highly skilled professionals actively shaping the future of artificial intelligence.

• Attendees: This exclusive community of 5,360 technical professionals comprises approximately 43% senior software engineers, 32% AI/ML scientists, and 25% technical founders. Members possess deep expertise in agentic workflows, LLM orchestration, and cloud infrastructure. Notably, the group bridges elite enterprise talent from Microsoft and Amazon with venture-backed startups, driving rapid, open-source AI integration and local-first hardware-software innovations.
• Companies Represented: Featuring industry giants like Microsoft, Amazon, Meta, Google, and Apple, alongside leading AI innovators such as OpenAI, Anthropic, and Together.ai, and rising startups like CopilotKit, OpenPipe, Fixie.ai, and Moondream, and more