LMStack - Development stack using local LLMs

Speaker 0: Hello. I'm Devashish. I'm gonna talk about a project I've been working on. It's called LMStack, and it is a way for you to use your GPUs, that are setting items probably to AI real code. So why is it so hard for local models to write code? Speaker 0: I mean, it's not a successful problem for 2 big reasons. So let's look at what a cloud model usage looks like. Like, this is probably your setup. You are a user, you enough. With local models, things become very complex. Speaker 0: And, again, QuickShow has run into a similar problem like this. Okay. A few Speaker 1: of them. Speaker 0: So for for for for folks who have not run into this, right, so look working with local models need you to look into the infrastructure, the harness, fine tuning for for picking out the right parameters for the models, AI can serve tokens predictably over your network with, some, like, decent tokens per 2nd improvement. So I I started solving 2 fundamental problems, that I ran into while using the local models for coding. The 1st 1 is infrastructure, and then the 2nd 1 is task management. I'm gonna talk briefly about both of these. So infrastructure means, engine, the gateway, v ramp, budget allocation, and how to get a real model to work on your system. Speaker 0: And the word management is how to assign the right kind of tasks to that works to that system. Working with local models maybe cannot throw the most complex task at it, but there needs a certain category of task that it is actually good at. So the 1st thing that LMS tag does is it builds the infrastructure for you. So this is an open source project. You can drop in, or you can install it and then, the SKIM gets So at a high level, it installs vLLM as the inference engine if it's supported or if it's, AMD GPU or something else. Speaker 0: But it pretty much sets it up for you. The 2nd suggestion or 2nd thing that the stack element or element stack does is it Jake. And this is an important bit. Right? So what I struggled with with for the longest period of time was how to get the tasks to run on my local system predictably. Speaker 0: Right? If I threw a bigger task, it kind of failed. Smaller tasks were fine, but they were not useful enough, so I could not run a predictable repeatable system on my local setup. So the task classifier does not match it. Right? Speaker 0: So on 1 side, it has think of it like it has an it has the access to your entire git, git issue list. And on the on the other side, it has a stack on which it can execute those tasks. So the classifier will decide dynamically which stuff are right for that stack and will kind of start queuing it 1 by 1. So it's like a if software factories does, task management at scale, I'm kind of building software forge that does it at a much smaller scale and on your system. So this is the entire system end to end. Speaker 0: The 1st 2 things, the 1st 2 steps do the infrastructure management. And on a day to day basis, you'll limit 2 loops, harvest phase finding or harvest step is finding the task to run on, the hardware, and then execution stage just execute those tasks repeatedly and then does it over and Speaker 1: over again. Is this visible? No. Done. That was a good clarifier. Speaker 0: Is this better? By the way. Yeah. So I just, at a high level, I'm sure what it does. So as a user, I go into .org, I say, harvest task for an open source repo. Speaker 0: It starts it starts harvesting all the tasks. At the end of, the process, this will give me, a list of tasks that are or that can be executed on the stack. Running short of time, so I'll The 2nd stage is the exec stage. This is again, as a user, you just go, oh, these are the tasks that it has identified. As a user, you can just go, say, setup stack, execute this list. Speaker 0: It goes 1 by 1 through the entire list and starts queuing it on the stack. Eventually, what you get is, like a dashboard that, has all the tasks. You can visualize which task are the execution and, other details. This is a real example. Like, this entire queue was flushed through my local setup. Speaker 0: Like, this is open source project. I pretty much Speaker 1: AI the moment. There's so many more devices you can use. Do you have any questions? Yes. Speaker 0: Yes. Speaker 1: This just seems like a lot of fun. How do you get data from your users? Like, do you new any metrics for guiding how you orchestrate which models to use? Speaker 0: Yes. Speaker 1: So Speaker 0: so there are 2 ways I'm trying to track this. So 1 is the, the VLLM sorry, LiteLLM dashboard. This is just for the token usage. This LLC, like, last few days, I've been, 109,000,000 tokens. So this is just, token utilizations. Speaker 0: But from a task perspective, I also built a dashboard here, which Joe this gives a lot of traces and see, what I get to see is, where the model is struggling. I use that data to AI the problem that is finally going to it. So the data from local usage comes back and feeds the classifier to get it more and more accurate for the repo I'm working on. Speaker 1: Okay. And that would be for me. Do you get it as well so that your project improves? Speaker 0: Oh, no. I just a 2nd. But this is for just not, like, your views. I don't do automatic data formats. Speaker 1: Yeah. Yeah. Any other questions? Yes. I was curious how the task task fire works. Speaker 1: Is it, like, deterministically running on metadata or is it actually using, like, a like, is it, like, parsing the semantics of the query? Yes. Speaker 0: It's not deterministic yet, but that's I have a few rules that I want to identify how what the complexity of the task versus the complex or the, capacity of the hardware and try to merge the 2 together. There are still heuristics built in, but it's not super great. It's because it's built for 1 project that I know really well. I don't know how to make it more generic, so that's the thing I'm working on now. Speaker 1: Cool. That was great.