Members-Only
Recent Talks & Demos are for members only
You must be an AI Tinkerers active member to view these talks and demos.
EvalForge: Automating LLM Judge
This talk explores automating custom LLM evaluation criteria using EvalForge and Weave, enabling users to create and run bespoke, human-aligned assessments.
LLM evals are all the rage now, with HumanEval, MMLU and others being shown by big labs on every release. But for your company specific use-case, those generic evals don’t mean much. It’s very clear to those who’ve built evals that custom, bespoke criteria and eval datasets are needed.
Building those evals are tricky, require a lot of data annotation and labeling, but what if we could automate this using human aligned criteria building with LLMS?
With EvalForge, we tried to do just that, take live data from Weave (Weights & BIases LLM observability framework) - let users annotate and then use LLMs to help come up with criteria, and then run evaluations.
Programmatically generates and executes traceable LLM evaluation judges via wandb Weave.
W&B Weave: LLM application framework for tracking, evaluating, and improving via SDKs.