WIBTD/W/M: Routing AI responses between local runners

I’ve been fascinated with local models more and more recently. These are the open models you can download and run on your laptop. It’s been really cool starting to use these with things like LMStudio on my Mac, but I’ve been thinking more also about how you can run and distribute LLM requests to these models across different machines and fleets, and it’s a little different than just running a cluster or service.

So my goal this past week was to build a service that could take requests centrally, route them out to local models, and pull them back to the user. Why? There’s a ton of examples out there calling a local model directly, but when you have a fleet of models and agents / users that can call them, you need some way to manage the chaos and deployments. Smartrouter provides logging and monitoring for these calls, and we can extend it to support ‘local AI’.

So how did I build it? First, research about how I should run local AI.

Considerations for running local AI

This is by no means an exhaustive list but incorporate some of the things I’ve thought about and read up on as I’m getting up to speed on local AI, keeping in mind the goal is concurrency and increasing the number of requests a cluster could serve for a particular model type.

First thing I’ve been reading up on is how to run local AI in clusters. This isn’t so much an activity like database sharding, more like how you can run multiple services. And there’s complexity here where if you try to run a large model, you need to take into account how to shard a model across things like different GPU’s based on vRAM size. In the example of concurrency, we can scale up here with faster GPU’s or storage like Lustre for model loading. But if we’re constrained by something like cost (so no GPU usage), or limited to the devices we have (I have 2 AMD Ryzen 5 NUC’s), that doesn’t really work. I’m looking to scale out.

So, in this example, what I really want is a fleet or pool of workers that take items, work them, and put them back. And there are a number of great existing examples of using queues and workers to complete tasks. So I decided to implement this pattern in smartrouter for local routing.

How the local cluster works in smartrouter

Smartrouter allows you to register machine runners into a ‘local cluster’, which is a logical service representation. This contains a queue for worker requests, the models available to serve, and any additional queueing or execution limitations.

Next, you can use the existing router architecture to route requests through smartrouter to the local clusters. This includes dynamic routing.

Here’s an example of a request that does a few things: first, it routes to a local model. Next, the router actually changes which model the request gets sent to based on the routing rules. Then the runner picks up the request from the queue, works it, and publishes it.

A runner in smartrouter is like a machine agent. It registers itself with smartrouter and uses llama.cpp to pull down the model to run. I’ve created an example using Kubernetes pods that also allow you to run pure CPU or GPU inference, and I’m working on a Mac and Linux version next (the Mac code exists but I haven’t been able to give it a thorough working through yet).

We can also test smartrouter local agents by creating a key just like any other app registered and routing to the appropriate ‘model’:

curl -s -X POST "https://gemini-smart-router-your-url-hash-uc.a.run.app/v1/models/gemma2:2b:generateContent" \
      -H "x-goog-api-key: gr_local_cluster_verify_key" \
      -H "X-Client-App-ID: app-local-verify" \
      -H "Content-Type: application/json" \
      -d '{"contents": [{"parts": [{"text": "Write a 3-word quote about coding."}]}], "generationConfig": {"maxOutputTokens": 10}}'

And get a result back:

Target Endpoint: https://gemini-smart-router-your-url-hash-uc.a.run.app
------------------------------------------------------------
💬 Gemma: "Code, debug, deploy." 
💬 Gemma: **Code, create, solve.** 
💬 Gemma: "Code it, fix it, thrive." 
💬 Gemma: **"Code, Create, Connect."** 
💬 Gemma: "Code makes reality." 
💬 Gemma: **Lines, logic, build.** 
💬 Gemma: **Code. Create. Conquer.** 
💬 Gemma: **Write code, build dreams.** 

Feats and pain using AI for development

Feat: Once we defined the desired architecture, LLM’s are pretty good at writing kubernetes specs in particular and running a playbook. When we got to this point, the process was a lot easier.

Feat: The rework cycle is dramatically shorter. I explored some earlier concepts about clusters and runners, and while they worked, the data model needed some heavy revisions to support scaling. LLM’s make this work significantly faster, to the point that you sometimes have to make sure you didn’t throw too much out the window.

Pain: Definitions really matter. In the first iteration of my local implementation, I called the machine agent an ‘agent’. And that created a number of comprehension problems for the LLM where changes would be made both on the frontend of smartrouter and in the machine agent, which I obviously did not want. After a few rounds of this, I went back to the drawing board and defined very clear and different terms in the application’s terminology. This helped the LLM generate better context for each of my reuqests going forward.

Pain: LLM’s still like to tell you it worked even though it didn’t. When I do iterative development, I like to use test driven design, but even that has flaws with the human. The tests look fine, but LLM’s I’ve found have a tendency to try to mock services to pass tests rather than run the request to the ground. I also ran into a few instances where a ‘200’ was not in fact a success. My learning here is to continue to build out my agent skills on testing so its clear what my expectations are, in this case no mocks in this code and real logs to prove it.

Next steps

Download and try smartrouter here: https://github.com/davenportjw/smartrouter

Let me know what you find / excites you / frustrates you!

I’m planning to extend smartrouter next week to support more dynamic routing requests and implement more analytic capabilities for routing. Cost in particular is a vector I hear a lot about, and mining current information for future learning is top of mind in a number of areas. Look for more on this in the next few weeks.

Also, I’m exploring cool concepts others have in this space, for example Matthew Tracy’s implementation of a router here: https://github.com/MatthewTracy/yagami

There’s a lot of good stuff here.