Making AI Faster: The Story Behind Bhumi

Hey everyone, Rach here. Welcome back to the blog! Today, I want to share the journey behind Bhumi, a tool I built to make inference clients faster and more efficient.

Why I Built Bhumi

A while back, while working on finbro.ai, a project where we built AI-powered agents, I ran into a frustrating problem: latency. Every time we asked an AI to do something, it took forever to respond. And since we had multiple agents working together, the delays stacked up, making everything painfully slow.

I knew AI could be faster, but the existing solutions weren't cutting it. So I built Bhumi, an AI inference client focused on client-side performance and efficient streaming.

Why Is AI Slow in the First Place?

Think of streaming a movie. You don't want to wait for the entire movie to download before you start watching, right? You just want it to start playing instantly while the rest loads in the background.

Most AI models don't work that way. Instead of "streaming" small chunks of information as they become available, they often wait to generate everything at once before showing you results. That's like downloading a whole movie before you can watch the first scene. Inefficient and frustrating.

Another problem? The tools that manage AI requests (like LiteLLM) weren't handling multiple requests well, leading to even more delays.

How Bhumi Makes AI Faster

Bhumi fixes these issues with three key optimizations:

1. Optimized Request Handling with MAP-Elites

Instead of traditional HTTP request handling, Bhumi uses an adaptive optimization approach. Using the MAP-Elites algorithm, it dynamically adjusts buffer sizes and processing patterns based on:

Provider-Specific Optimization: Different buffer sizes for different AI providers
Adaptive Processing: Buffer management that evolves based on performance data
Quality-Diversity Balance: Maintaining both speed and reliability
Continuous Improvement: Learning from each request to optimize future ones

Our testing showed throughput improvements from 600 to over 1400 characters per second after just 15 iterations.

2. Rust + Python Architecture

Bhumi's core is built in Rust for maximum performance, with a Python interface for ease of use. This hybrid approach delivers:

Native-speed processing with PyO3
Developer-friendly API
Minimal overhead

3. Optimized Validation with Satya

We replaced the standard Pydantic validation with Satya, our custom validation library that:

Reduces memory overhead
Processes types faster
Maintains full type safety

Results and Impact

These optimizations deliver significant performance improvements:

Response Time Improvements

OpenAI: 2.5x faster than raw implementation, 1.9x faster than native
Gemini: 1.5x faster than raw, 1.6x faster than native
Anthropic: 1.8x faster than raw, 1.4x faster than native

Memory Efficiency

Only 1.1x memory overhead vs native implementations
Stable performance under load
Efficient resource utilization

Real-World Impact

These improvements translate to real-world benefits:

Faster response times for user interactions
More efficient resource utilization
Better scaling for multi-agent systems
Reduced operational costs

MAP-Elites optimization, Rust-based streaming, and careful buffer management improved the client in different parts of the request path. There is still room to measure and optimize.

Supported AI Providers & Structured Outputs

Bhumi supports multiple AI providers, allowing seamless switching between them. Currently supported providers include:

OpenAI (openai/{model_name})
Anthropic (anthropic/{model_name})
Gemini (gemini/{model_name})
Groq (groq/{model_name})
SambaNova (sambanova/{model_name})

Bhumi also supports structured outputs and tool use, making it easy to integrate external functions into AI responses.

Using Bhumi for Tool Use & Structured Outputs

Bhumi allows AI models to call external tools for better interactivity. Here's an example that registers a weather tool and lets AI call it dynamically:

import asyncio
from bhumi.base_client import BaseLLMClient, LLMConfig
import os
import json
from dotenv import load_dotenv

load_dotenv()

# Example weather tool function
async def get_weather(location: str, unit: str = "f") -> str:
    result = f"The weather in {location} is 75°{unit}"
    print(f"\nTool executed: get_weather({location}, {unit}) -> {result}")
    return result

async def main():
    config = LLMConfig(
        api_key=os.getenv("OPENAI_API_KEY"),
        model="openai/gpt-4o-mini"
    )
    
    client = BaseLLMClient(config)
    
    # Register the weather tool
    client.register_tool(
        name="get_weather",
        func=get_weather,
        description="Get the current weather for a location",
        parameters={
            "type": "object",
            "properties": {
                "location": {"type": "string", "description": "The city and state e.g. San Francisco, CA"},
                "unit": {"type": "string", "enum": ["c", "f"], "description": "Temperature unit (c for Celsius, f for Fahrenheit)"}
            },
            "required": ["location", "unit"],
            "additionalProperties": False
        }
    )
    
    print("\nStarting weather query test...")
    messages = [{"role": "user", "content": "What's the weather like in San Francisco?"}]
    
    print(f"\nSending messages: {json.dumps(messages, indent=2)}")
    
    try:
        response = await client.completion(messages)
        print(f"\nFinal Response: {response['text']}")
    except Exception as e:
        print(f"\nError during completion: {e}")

if __name__ == "__main__":
    asyncio.run(main())

With Bhumi, AI models can generate structured responses and interact with external tools effortlessly.

Final Thoughts

Bhumi is also about flexibility and efficiency. It can switch AI providers, return structured outputs, and support tool use through one client.

Drop a comment below. I'd love to hear your thoughts! 🚀

Hey everyone, Rach here. Welcome back to the blog! Today, I want to share the journey behind Bhumi, a tool I built to make inference clients faster and more efficient.

If you've ever waited for an app to load, a chatbot to respond, or an AI tool to generate something, you know the pain of slow inference. Bhumi fixes that by optimizing the client side of AI interactions. In this post, I'll break down how it works, including the technical details.

Why I Built Bhumi

I knew AI could be faster, but the existing solutions weren't cutting it. So I built Bhumi, an AI inference client focused on client-side performance and efficient streaming.

Bhumi's client-side optimization layer in the LLM request pipeline

Bhumi speeds up the client side of the LLM pipeline. Nothing changes at the provider. It optimizes buffers and makes the client more efficient [^1].

Why Is AI Slow in the First Place?

Another problem? The tools that manage AI requests (like LiteLLM) weren't handling multiple requests well, leading to even more delays.

Hypotheses for speeding up the LLM client pipeline

Hypothesis 1: A Rust-based streaming layer could improve performance through PyO3

Bhumi's Rust streaming path connected to Python through PyO3

Well if you can see the diagram above, it all seems to work fine! Right? Not quite, everything broke loose when I introduced types.

Typed Bhumi request flow showing Pydantic as the validation bottleneck

Specifically Pydantic was the bottleneck, as it was using a lot of memory and was not able to handle the requests as fast as I would have liked.

Hypothesis 1.5: A similar streaming pattern could speed up validation

The question then was, was it worthwhile to implement a similar streaming pattern on a validation library? Maybe we could also use Rust and map it to Python with PyO3? And I opened an issue and started working on it.

Plan to replace Pydantic with a Rust-backed validation layer

It took about an hour, and a week's worth of refining the dev experience for myself, but I got it working! And boy was it fast!

Validation benchmark after replacing Pydantic in Bhumi

And with that the 1.5 hypothesis was proven to be true!

End-to-end Bhumi performance after the validation change

Why the faster library was still not fast enough

Buffer sizes had become the next bottleneck, which led to a second hypothesis.

Hypothesis 2: Some buffer sizes may be optimal for LLM output

Response-time results across tested LLM buffer sizes Throughput results across tested LLM buffer sizes Buffer-size benchmark showing no consistently optimal fixed setting

I ran a few benchmarks and found what appeared to be optimal buffer sizes for LLM output. That conclusion did not hold up.

The result turned out to be wrong, so I scrapped the fixed-buffer approach.

I had an idea

Could buffer conditions adapt to different providers over time? I knew an algorithm that prioritized both quality and diversity.

Hypothesis 3: Can quality-diversity algorithms find better chunk sizes for each buffer?

Quality-diversity algorithms explore a search space while seeking a diverse set of high-performing solutions across several dimensions. If we could map better chunk sizes for each buffer, we could potentially speed up the client side of the pipeline.

The algorithm I chose was Map-Elites because it's a quality-diversity algorithm that explores a search space but also seeks to discover a diverse set of high-performing solutions across multiple dimensions. Furthermore, it's a very performant algorithm, and I could use it to map out the optimal chunk sizes for the type of buffer that comes thru.(and again it was just a hypothesis)

So I started working on it, and I got it working! After 15 iterations, the throughput had improved from 600 characters per second to over 1400 characters per second.

Here was the grid after 15 iterations:

MAP-Elites buffer-configuration grid after 15 iterations

Furthermore, as more iterations are run and more of the grid is explored, we observed an interesting phenomenon in the MAP-Elites algorithm's performance. The rate of improvement begins to plateau, and in some cases, we see a decrease in overall performance. This behavior stems from several key factors:

Search Space Saturation
- Initially, the algorithm easily finds high-performing solutions because the behavior space is largely unexplored
- As the grid fills up, finding better solutions becomes exponentially harder
- The algorithm must work harder to discover solutions that outperform existing elites
Local Optima Traps
- The algorithm can get stuck in local optima within certain regions of the behavior space
- Mutations and crossovers start producing increasingly similar solutions
- Breaking out of these local optima requires larger, potentially disruptive variations
Exploration-Exploitation Balance
- Early iterations benefit from broad exploration of the behavior space
- Later iterations tend to focus more on exploitation (refining existing solutions)
- This shift can lead to decreased diversity in the candidate pool

In our specific case with buffer optimization, we observed:

Peak performance around iteration 15 (~1400 characters/second)
Gradual decline in improvement rate after iteration 20
Increased computational cost per improvement as the grid filled up

Throughput over MAP-Elites iterations, peaking near iteration 15 at about 1,400 characters per second

This pattern is actually expected in quality-diversity algorithms like MAP-Elites, and it helped us identify the optimal point to stop training and deploy the solution.

How Bhumi Makes AI Faster

Bhumi fixes these issues with three key optimizations:

1. Optimized Request Handling with MAP-Elites

Instead of traditional HTTP request handling, Bhumi uses an adaptive optimization approach. Using the MAP-Elites algorithm, it dynamically adjusts buffer sizes and processing patterns based on:

Provider-Specific Optimization: Different buffer sizes for different AI providers
Adaptive Processing: Buffer management that evolves based on performance data
Quality-Diversity Balance: Maintaining both speed and reliability
Continuous Improvement: Learning from each request to optimize future ones

Our testing showed throughput improvements from 600 to over 1400 characters per second after just 15 iterations.