Verification with LLM powered Behavior Driven Development (BDD)

Speaker 0: AI. AI, GPT use E2E help us continue run verify that, the software or GPT of building, you know, five CTO 2026. So, Moore E2E thing is that agents Dan GPT run through these Moore AI Haskell E2E out of build E2E War for a really long AI. And the problem is, they E2E Agent CTO how you'd actually know if our agents are doing the right thing. How do we actually, get evidence, for the for them, for everything CTO teams. Speaker 0: This is actually really 2 problems. The 1st is how do we actually express to them, what is Agent that we BDD we want CTO AI? What is my what's Track acceptance criteria? And then the 2nd thing is, well, how do we verify the actual evidence itself so that E2E can catch John CTO them? So the the solution, E2E, is the teams of Laes the Track level through which we communicate with the agents of the way we communicate War other Dan run the way it communicate thus. Speaker 0: And so E2E, how many people here actually know what BDEs are? AI. BDEs is a pretty old Textio E2E that, is actually GPT for a soundless sensing. It raises the level of Track. And it uses natural language AI that are written by people, human-agent Kennedy Engineer to manage birds to document these E2E Date in the system. Speaker 0: So AI the question is how Yen we actually do Tools video Guides? Have use raised the better talk the attention on reviewing the Attendees? Well, it's not reviewing the Code. It's not about doing that. It's not about showing the burden. Speaker 0: It's actually about reviewing War AI run sharing E2E actions that the system DMs actually teaching, making it a lot simpler for the humans to actually consume that Event. And so this is something that we AI. Right, buddy? E2E humans aren't even really good at reviewing 50 Laes of code, Lead alone hundreds of bots to code. Instead, we're usually good at looking at some hot desks and speech styles, sightseeing ears. Speaker 0: We have to fight each other really quickly. So here's an example of the temple of BED, Smyrna that we use in Sajrons itself. And and you can see here it's a very simple, like, natural language specification. I'm not testing up SWE details about this. I'll show you how this this this actual 1 is actually run. Speaker 0: Yes. So we'll go over and show you what the demo is. We start off by kicking off, Joe orchestration system, our E2E system. And BDD then after that, let me do some Claude co-founder then we actually start off and say, okay, Lead it run a particular GPT E2E. Audience War source, it E2E minutes in real time the actual system Ben the actual reviewing available for this to John test. Speaker 0: Now for the purposes, search, Dev AI actually go a little bit further forward to when, my report is is actually generated. So AI I said, the actual issue is being able to proofs audit what is being done. And you see here insights system here, we actually Shoale every set of Gherkin test. And, in this run, 2026 can look John here. This is, the better where the user AI Agent Singh run, and we actually show it as a web better. Speaker 0: We actually show all of the auditing and the interaction with diagnostics. What's interesting here run that we actually show all the SWE Stagehand record everything that's being done. So this is this is a lot easier to digest than you're looking at test code. Now what's interesting about SWE. Thomas is that we have lots of search E2E, not just Lead, but we also Dan, AI services sharing. Speaker 0: And so Speaker 1: here, Speaker 0: E2E Agent the site. Here, you actually audit what is the CLI that's actually being, secure, what is it looks like the actual output that the user sees. So this introduces the cognitive Lead of actually doing Speaker 1: that Verification. Speaker 0: Okay. So how E2E do this, we're actually using all of them to generate all this Code. So the way we SWE force it to put it into E2E layers. We're very E2E about this in our framework. So the 1st layer is the actual 1 use SWE the, the E2E. Speaker 0: The 2nd layer is a set of business actions that actually Max the tasks that the E2E are calling. And then 3rd layer of code that it generates is actually the individual system actions. And so we force use we force the element to go HINTS these very specific places. As a result of that, E2E able to actually get it to actually do these, do the E2E, run very kind of systematic structure. The AI of the hood, my brother War E2E fast. Speaker 0: We five the orchestrated E2E AI I Moore you that E2E started. Dash is, managing a bunch of micro DMs, and those VMs are there to isolate everyone's Allies. Sharing the VMs, there's a router that actually GPT how each of the instructions, each of the verbs, each of the actual verbatim, like a Chrome internal Chrome browser or CLI. Anyway, that's a quick recap. So we make verification easier by auditing the actions BDD not the code. Speaker 0: Sorry, War is a GPT video use Dan take a look at our YouTube channel. Run, SWE, I'm gonna be over local with Fred new. Speaker 1: What? Speaker 2: Thanks, Alex. Singh was a great demo. So, so you had a you kicked off the Behavior, behavior Dan development test from a prompt. That was really cool. Speaker 1: It's not actually E2E. Speaker 2: That's fine. Any questions from the audience? Do you want use SWE BDD? Yes. Yeah. Speaker 1: So are you offering using the Agent to generate all this test run War, like, E2E, or is it just execution of the load? Speaker 0: So it's actually, the generation is as though run author, Editor E2E. And so it uses 2026 Lead to SWE, again, this will Community Lead really, that those 3 different areas and then generate builders data or any Singh before. But it also, build m's there's also another LLM that's being used here for Azure Date Engineering War, like, the Ben, WAN AI, how to actually GPT, Asher in my lab UI. So we use an LLM to reduce AI to reduce, like, a a E2E type of speed to, like, be added to both front edge. Speaker 2: AI, can use? Step up and connect you, Peter. We GPT another touch. You're new E2E, Yannis. Speaker 1: SPEAKER. Because so you'll be sharing this version of your underlying software that you're trying to test with this. How do you maintain your versioning of your scenarios? Because every every release will Date run 2026 version. AI new run to better verify the E2E. Speaker 1: So 1 is in the production, 1 is in the flight Cripe. Yeah. Here's the version. Speaker 0: E2E we do it right now. Actually, we we we only have we Date test scenario that detect scenario AI gonna new to burst the production. And we actually do all Ben testing against the test system 1st CTO voice edges, Laes what production. But E2E, we we could also be more, you know, Code of running E2E test on the production system. Right now, you'll kind of be on the Lead. Speaker 0: Track you use that. Speaker 2: Yeah. Thanks, Galex. John Yen. Yeah. Moore and Agents been a great way to AI CTO minutes for a long time. Speaker 2: New, if you're from Duncan' Mechanical layout ad E2E Review, new call the 1st