Testing Probabilistic Agentic Systems

Speaker 0: And and BAML, BoundaryMail created a holy language, for both agents. And, and, actually, their language is great for building prompts. And we actually, I have used, used BoundaryMail almost from the beginning. We've used, BAML to to build prompts into our application. So, here we go, Craig. Speaker 0: Alright. Yeah. You can do it. It's Dave. I can code, but I can't do this. Speaker 0: Okay. Close to your mouth. A little closer, actually. Speaker 1: Hello. Thank you. So I've got 2 thoughts. We'll pick which 1 you guys wanna see. You guys wanna see more about how to understand code without beating it really quickly, or do you wanna talk about emails? Speaker 1: I have a favorite. Who are you? Emails? Number 1. Speaker 0: Chef's choice. Chef's choice. Alright. Wait. Wait. Speaker 0: Wait. Wait. I own Enter emails. Okay. Okay. Speaker 0: Hands for the other 1. Speaker 1: Emails are in. Emails are within the other bit. It's okay. So let's talk about emails. When we talk about emails, I just wanna really prime everyone really quickly because we talk about all these software cities and, like, gas down, whatever it is. Speaker 1: But What we really want is we want this world where we ship code at agent speed, not at human speed. To do that, we gotta go understand what the system does. So take face ID for example. If you wanna make face ID, I worked on face ID for a while. How do you do this? Speaker 1: You build metrics. You say, I want number of false positives to be down, and I want number of correct passes to be off. If you only pick 1 of them, you have a really good face ID system. Take the first 1. Number of false positives to be down. Speaker 1: The term false. It'll always work. Doesn't really work in, a zone. So you build both metrics. But then it gets trickier. Speaker 1: Because then you're like, okay. Well, how do we perform in different scenarios? Twins, people with glasses, when the face is front and center and face on the corner. So you actually start thinking the form of scenarios, and then you build different acceptance criteria for different scenarios. And then all of a sudden, COVID happens, and you're like, fuck. Speaker 1: Masks. Speaker 0: And then all Speaker 1: of a sudden, your acceptance criteria also goes down because it used to be a primary security feature. Like, okay. Loosen the bar on that so people can use their phones outside, and they'll accept that. And I think this this whole dilemma is anyone that's done machine learning is like, cool. I get it. Speaker 1: This is basic shit. For many of us that have really never built a probabilistic system before, this is probably why agents feel really flaky because the way to think about systems in this way with probabilistic outcomes is really not the same. The way that you write tests in traditional software is you build 1 metric, does the test pass? And you don't think of it any other way. So we've been thinking really, really hard about how you think about testing in this world where everything is agentic and probabilistic? Speaker 1: So part of it boils down, oh, I hate dark mode on projector, so we can fix that. Apologize to everyone that thinks it's a sin, but it is what we must do. So when you think about code and think about, how you wanna go build test cases, well, I built a sentiment classifier. The way I really wanna build an agentic sentiment classifier Speaker 0: over here Speaker 1: and this is just a function, sentiment classifier returns some sentiment. Nascimento is something. Sentiment is something that's in the feeling of the stream, conference, and reasoning, Speaker 0: and Speaker 1: that's what it returns. But what I really wanna say is I have a set of topics, happiness and sad talk, happy topics and sad topics. And I actually wanna load my test cases, not statically written, but actually from a database or a web query or somewhere that's really dynamic. Every time I run tests, I want to fetch from that data source in freshness. What what does this mean? Speaker 1: I have production data loaded somewhere. I wanna go pull the data out of prod and run the last 1% of traffic through my email speed regularly and watch the benchmark move. So what I really wanna do is write something that is actually able to understand that my tests are dynamically generated. They're static but dynamic at the same time, and you wanna be able to run arbitrary ones of them. So if I have the JSON file, for example, I have a happy dot JSON file I'm just serving from here, and I've removed 1 of these lines, and I refetched all my test cases, and I really this would be Speaker 0: best of my demo. Speaker 1: And I refetched all my test cases, you only get 3 now instead of 4. And what this lets you do is now you can run a CICD system that's actually fetching data from prod, and you can have an asynchronous process populate that data source through vetting. So what I would do is I would actually go ahead and go get all my data from prod, manually select ones that are difficult test cases, easy test cases, very similar to what we were talking about on face ID where we have different types of categories and different scenarios of test cases and build different acceptance criteria for them. You can go further. You might have some other acceptance criteria that's pure bias. Speaker 1: And when you do pure bias based acceptance criteria, you have a functional generate test. Generate test is literally just hitting an element every single time. It is building dynamic test cases. You likely want test cases like this as well, where some type there every single time you run them, it's completely dynamic. It could because they serve a different audience. Speaker 1: Now the next thing you wanna do is you likely wanna think about tests fundamentally differently, which is pass rates. Pass rates don't actually work in the traditional sense. So So I'll show you this if I can use this. 1 second. Artifacts. Speaker 1: What's the word font? Why is the dart sound? I apologize. Test don't actually work at the same way. The way that you wanna make pass rates work is almost slightly different. Speaker 1: You almost wanna say that, hey. I want a pass rate function that says if as long as 80% of test cases pass within here, this whole thing passes. Because, again, when we build Face ID, we didn't say every base of the pass every single time for the new model would be good. We said some percentages passed. And I think the lack of tooling like this is what makes stuff very hard in traditional software. Speaker 1: Because in iTest, how do you write something like 50% test passing is good? And this is really the question that we're thinking about. So any of you are thinking about emails or have thought about them and have had made major difficulty with them or perhaps eased with them, could chat with me. I'd be super curious to learn more. If you guys want beta access to this, let me know. Speaker 1: We have a few folks that have beta access. You guys would find me on Twitter. My username is vidcode, v a I d. I know 1 agrees. Love that. Speaker 1: Thank you, guys. That's fantastic. Speaker 0: Yeah. I think the way that the tests were presented concerns me a little bit. It seems like it's being rubber stamped because you're generating the code via, like, block code or something, and then you're generating your test using the same tool, how is any validation actually occurring that wouldn't just get caught by, the agent that wrote the code? Speaker 1: Because the way you actually build test is this is the test this is the test that's being written right over here. Okay. But the actual test case itself is simply the part oops. Speaker 0: It is simply the part that's actually Speaker 1: in the test block. So this is you can think Speaker 0: of it as metaprogramming for building Speaker 1: a test case. And then what you're running over here is this test that says, hey. I expect every result of viewing to be positive every single time. So you can build actual assertion or element is judged or whatever kind of workflows you want, because, fundamentally, tests are just running, tests are just running code. They're just calling functions. Speaker 1: And the last thing we have that I I don't have a live demo for is actually a way to collect metrics similar to what we talked about with face ID, where you can actually aggregate metrics over here. You can say, in this case, quality is the metric I'm tracking. And then we bubble up that metric to the aggregator, and then you can say that pass rate actually run the function that checks for the quality metric. And it says 80% of them have to have a quality of grid grid than true or something like that. So it's, like, built around aggregation with, like, functions as Speaker 0: the primary mechanism of doing that. Okay. And when you pair the percentage based testing with some, like, hard test that will always be there? Exactly. Speaker 1: So that's what we did in face IDs. Faces front and center, 100% of the test test should pass. Twins, not so much. Faces in the right hand corner had a higher pass rate than face in the left hand corner because the way the cameras include each other. Speaker 0: So we had the design for all of that. Awesome. Thank you, Cam. Great. Just before we go to the next question, can I have the next, presenter power come off the front? Speaker 0: Okay. Okay. Cool. We'll get you set up in a sec. Yep. Speaker 0: We got a question. Where was the question? Just there. Yeah. Speaker 2: Hi. I'm super curious about the system, and I think that I need it. Speaker 3: But I, I can't specifically, understand how exactly do Speaker 1: you repeat text? So it was, like, text, like, happy, sad. Is it manual like, do Speaker 2: you expect manual annotation of the data prior to any test generation or, like, how exactly did you mark your Speaker 1: So there's, like, 2 ways to do this. So I think, initially, what you should do to write tests is just plot code, write some tests for you. That's good. Step 2 is you have you can actually if everything is understood to be oops. If everything is understood to be a certain type system, you can actually just have an l o n go ahead and like, I learned it's not a function here called, like, generate test. Speaker 1: It takes an account, takes in a string, and then literally just fits out a test case in the form of that many test case. In this case, they input that function as a string, so it returns an array of strings out. So you're gonna have an l m generate test. Speaker 0: But then the most the Speaker 1: most advanced thing to do, and especially as you get more users, is you actually monitor your production data, and then you, basically flag certain ones to be test cases. And you can do that in automated ways or manually. So if a customer you can track, like, metrics. So let's say you're Amazon. You wanna try buy rate of a customer from a new element by line that Speaker 0: you build. Speaker 1: Well, you just track the times when a customer had a long conversation and didn't buy and automatically make the test cases. They might not have bought for various reasons, but you can promote them. And maybe then an element pipeline can go through that and select which 1 of these should have been buy intents, but the user didn't buy because they were frustrated by the agentic experience. Boom. You have test cases. Speaker 1: And then you can take that entire chat array, and that thing will return to you, like, an array of arrays, not because the chat message is an array of arrays. And you can say, let's write a new element pipeline that's gonna take in this array of arrays, and let's see if you would have produced a different message if we trimmed the number of messages by 3. So we're undo the last 3 messages and replay the calls back and see what happens. And what you wanna see is would the customer buy it again, and would the message quality have been better? Speaker 0: So you Speaker 1: can build all sorts of different benchmarks with this. Because, I don't know if that made sense, but, like, testing is just code. And if you model it like functions, it's very easy to not get stuck in, like, l m is judgment. They're everywhere. Everything is pure bytes. Speaker 0: Yeah. And, by both, what's what's the best way to reach the faculty which kind of The Speaker 1: best way is probably, if you can't find me on Twitter, our discord, foundry and all.com/discord. Discord. Speaker 0: We respond within, like, 5 minutes. They're really good on Discord. So I'm sure plenty of people have plenty of questions. Definitely get in touch with the Discord with Bammel. Big round of applause for my boss. Speaker 0: We're gonna tell him on stage. Tell us to talk about, orchestra, which is a great, Speaker 1: way