Code > Prompts: Why Your Agent Should Think in Code, Not English

Speaker 0: You plugged in there. I'll take that away so you can set your laptop down. So make sure you find our presenters afterwards because I'm sure there's tons more questions. It's great to have a variety of different presentations. There we go. Speaker 0: So many things we so many topics we cover, at the talks, so it's really exciting to connect afterwards. Speaker 1: Hi. I'm Rajkumar. I've been building, agents on code, as opposed to prompts. So let me show you how. So these are 2 agents side by side. Speaker 1: Same model, same data, same tools, and same question. This is an SCC sec, like a report, and then questions and answers are coming up. So on the left, we see the tool calling agent, which is basically based on React, which everyone is familiar with. And on the right is something new. Okay. Speaker 1: So this is how the tool calling agent looks like. Right? So you have, function, definitions. These are 8 function definitions here. And and a huge system prompt. Speaker 1: You're all familiar with that? It's called a behavior agent where it has primitives where, these are essentially tools and they are, Python code and strongly typed inputs and outputs. And and so this is a behavior. And behavior is essentially you know, create a piece of code using the primitives that is defined about and, you know, solve it then. So we are basically defining behaviors. Speaker 1: So so we tried a very simple query, before. So let's try a bit complex query. Okay. So, yeah. You can see the query here. Speaker 1: So behavior programming actually failed. The reason is because it gave a list of, companies, but it did not give the numbers. So we don't that is partially correct. And, React will eventually get there. There. Speaker 1: Let's let's try to fix this problem in the behavior programming paradigm. Alright. So what I did here is, I committed an example which says, okay, given a question like this, this is how you would solve this. In this case, it says there's a loop here and it explains, okay, you got to explain, get the text, get the revenue, blah blah blah and all of those things. It's an example. Speaker 1: It's not the same exact query we have. Okay. And the same exact, question. Yeah. It's, so it got numbers. Speaker 1: It's close. And, yeah, to call him and fix it. Yeah. All I wanna say is, like, hey. There there are 2, 3 different things. Speaker 1: 1 is that, like, primitives and decompositions, they don't occur. It's unit testable. It's compostable. All the good stuff that comes with the standard software engineering practices. So it's a different That's all. Speaker 0: That's great. Yeah. Don't put that microphone down, though. So, that's really interesting. So 2 ways 2 ways of harnessing the LLM. Speaker 0: We've got traditional tool calling and then behavior programming. Who knows about behavior programming here? Right. Where can people find this? It's open source. Speaker 1: You can find it on behalf. Speaker 0: Is it is it your concept, or did you do other people do this as well? Or Speaker 1: Just mine. Speaker 0: Interesting. Okay. You've got the URL. Yes. You are co. Speaker 0: Only 1 more tonight, folks. I promise. That's fascinating. How did you come up with that concept? Speaker 1: Well, I mean, language isn't precise. So the code is math. Math is precise. So you wanna go from if you wanna represent something, if you represent it in code, you as as precise as you can get. Speaker 0: But you're you're instead of kind of directing the model towards a tool call, you're directing it towards a function. Speaker 1: Yeah. Yeah. The input is a function and ask you to produce no function. In this case, you can have multiple functions as inputs to show that this is how you could do this. Speaker 0: Fascinating. Okay. Any questions? Yeah. Speaker 2: When you say that the input is a a function, are you saying that the prompt is being fed with your function definition as, like, an example? Speaker 1: Is that what you mean? Yes. Your code the code, shown here. This is essentially sent to the l m as, you know, input. Prompt. Speaker 3: And is it taking that and whatever intent you give it, it's just composing tool calls in a programmable way where you just execute it in a sandbox, get it back? Speaker 1: That's right. So, it's producing a similar piece of code. It's not tool calling. It's a similar piece of code which it produces. And it is run on the same process in a different name space, but it it will, basically, not allow any dangerous function or is protected. Speaker 3: When you say it's not tool calling, like, it is technically making a tool call with a composed function of, like, here's a class and things you can do. Speaker 1: Well, tool call technically, you know, you get the JSON blob and you extract the names and parameters and you make a method call. Right? So this essentially is a is a piece of code you get. You just dump it in Python to look it up. Gotcha. Speaker 2: In your emails, it showed that there is a higher output quality with the behavior programming. From the the survey earlier, like, output quality is kind of the number 1 concern of of people in this room. Can you expand on why that is, why why you think it is, and, like, what other work you're doing around that and, you know, how you evaluate it and that kind of thing? Speaker 1: Yeah. So I I think philosophically speaking, when we say, okay, these are the rule these are the things. It's a huge massive prompt in English. We're asking our students to do magic. We are not showing it examples, and this is how we solve a certain problem, it doesn't know how to do loops, it doesn't know how to do exceptions, conditions, and all those things. Speaker 1: While, if you're doing it by code, we know exactly what needs to happen. So by showing those examples, the code that LLM produces is much higher quality. So it's closer to what a human would want, in this defined encoding. I don't know if it quite answered the question, but we can check. What question is about? Speaker 1: Did you show the output quality? Oh, guys. Just a quick question. Here. Speaker 4: 2 questions. Have you tested this method on, like, a publicly available benchmark to see if, like, it beats, like, let's say, just, like, React. And then second question is, have you done, like, an evaluation analysis to see, let's say you just do you just give the ability to write code in a sandbox rather than giving any examples. Like, does that also, like, get very good results? Like, what's, like, the major IDENTIFICATION here? Speaker 4: Is it the examples or is it sandbox coding to answer the question? Speaker 1: Yeah. I've I've done both. I've done like 3 benchmarks so Carsten I can share. And, so, yes, it can produce code. Elements produce code. Speaker 1: But it do produce random code. We don't want to import random libraries and execute those. So that's why the primitives say basically say, these are the only things that you can you're allowed to use and lend and simple things like that. Speaker 0: Alright. Well, that's all the time we've got for questions, but how fascinating. Thank you so much.