How to Build Your Own LLM Aggregation Gateway: Manage 40+ Providers, Virtual Keys & Budgets

How to Build Your Own LLM Aggregation Gateway: Manage 40+ Providers, Virtual Keys & Budgets

Last year, our team integrated three large language model providers. Each came with its own API key and a different billing structure. The real headache started when one service unexpectedly blew past its limit, causing our end-of-month bill to skyrocket. We spent an entire afternoon digging through logs just to figure out who used what. If you've ever dealt with scattered API keys across config files, lost visibility into per-app costs, or wanted a centralized way to enforce rate limits and allocate budgets—this tutorial is exactly what you need.

I'll walk you through building an LLM aggregation gateway step-by-step. You'll use a single unified endpoint to access 40+ model providers, issue virtual keys, set budgets, and monitor usage—all deployed locally. The project is called Otari, open-sourced by the Mozilla AI team. It's fully OpenAI-compatible, meaning code written with the OpenAI SDK works out of the box. Just change the base_url, and you're done.

By the end of this guide, you'll have a production-ready gateway instance running, completely eliminating the need to hardcode API keys in your applications.

Prerequisites

Before we begin, ensure your environment meets these requirements:

• Docker and Docker Compose installed (we'll use Compose for the easiest setup)
• At least one model provider API key (e.g., OpenAI. Anthropic or Google Gemini keys are great for later comparisons)
• Basic familiarity with curl and terminal commands
• Python 3.13+ (only needed for local development; not required to run the gateway itself)

Why Docker? Otari relies on PostgreSQL for persistent storage. Using docker-compose spins up both the gateway and the database in one command, saving you from manual database setup. This architecture is also strongly recommended for production.

Step 1: Quick Start with Docker (Ephemeral Instance)

Let's spin up a temporary instance to experience how Otari works. Open your terminal and run:

docker run --rm -p 8000:8000 \
  -e OTARI_MASTER_KEY=SET_A_MASTER_KEY \
  -e OPENAI_API_KEY=YOUR_OPENAI_KEY \
  -e OTARI_CONFIG_YAML='
default_pricing: true
providers:
  openai:
    api_key: ${OPENAI_API_KEY}
' \
  mzdotai/otari:latest \
  otari serve

What's happening here?

• -p 8000:8000: Maps the gateway's port 8000 to your host machine for easy local debugging.
• OTARI_MASTER_KEY: The management secret for the gateway. You'll need it later to create virtual keys or access the dashboard.
• OTARI_CONFIG_YAML: Inline configuration telling Otari which providers to enable. We're only configuring OpenAI for now.
• default_pricing: true: Enables default cost estimation so Otari can automatically calculate the price of each request.

Once started, you'll see output similar to:

No API keys found. Created bootstrap key for first run. Save this now:
tk-abc123...

Write down this key. It's the "bootstrap key" auto-generated for your first run, and all API calls will depend on it.

Step 2: Send Your First Request

Let's verify the gateway is working. Open a new terminal tab and send a message via curl:

curl http://localhost:8000/api/v1/chat/completions \
  -H "Authorization: Bearer tk-YOUR_BOOTSTRAP_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai:gpt-6-luna",
    "messages": [{"role": "user", "content": "Introduce Otari in one sentence"}]
  }'

Notice the model field format: provider_name:model_name. Here we use openai:gpt-6-luna. If you had Anthropic configured, it would be anthropic:claude-sonnet-5-5.

If successful, you'll receive a standard OpenAI-compatible JSON response. This is Otari's core value: your application code requires zero changes. Just point your base_url to the gateway, and all model calls route through it.

Here's the equivalent call using the OpenAI Python SDK:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/api/v1",
    api_key="tk-YOUR_BOOTSTRAP_KEY"
)

response = client.chat.completions.create(
    model="openai:gpt-6-luna",
    messages=[{"role": "user", "content": "Hello, Otari!"}]
)
print(response.choices[0].message.content)

Changing just the base_url means your existing client.chat.completions.create() calls remain untouched. This is exactly why I recommend teams adopt a gateway—the migration overhead is virtually zero.

Step 3: Deploy a Persistent Production Instance

The Docker container above is temporary (--rm). Stop it, and your data vanishes. For real-world use, you need persistent storage. Otari's official Docker Compose template includes a PostgreSQL database.

## Clone the repository
git clone https://github.com/mozilla-ai/otari
cd otari

## Copy the configuration template
cp config.example.yml config.yml

## Edit config.yml to set your master key and provider credentials

Open config.yml, locate the providers section, and input your actual API keys. For example, configuring both OpenAI and Anthropic:

providers:
  openai:
    api_key: sk-openai-xxx
  anthropic:
    api_key: sk-ant-xxx

Next, generate an encryption key (used to securely store provider credentials in the database):

grep -qs OTARI_PROVIDER_ACCOUNT_PEPPER .env || echo "OTARI_PROVIDER_ACCOUNT_PEPPER=$(openssl rand -base64 32)" >> .env

Finally, start the services:

docker compose pull
docker compose up -d

Once running, navigate to http://localhost:8000/ in your browser. You'll see the Otari management dashboard. From here, you can:

• Manage providers and models
• Create virtual API keys (scoped by user, project, or model)
• Set budget limits
• Monitor real-time usage and costs
• Configure routing policies (e.g., automatic fallback to Anthropic if OpenAI goes down)

If you prefer entering provider keys directly via the Dashboard instead of editing YAML files, you must also set OTARI_SECRET_KEY in your .env file. Generate a Fernet key using the otari gen-secret-key command.

Hands-On Example: Distribute Keys to Two Projects & Set Budgets

This is where gateways truly shine. Suppose you have two internal projects: a "Customer Support Bot" and a "Data Analytics Pipeline". You want to issue separate keys for each and cap their monthly spend at $50 total.

Step 1: Generate Virtual Keys in the Dashboard
Navigate to Dashboard → Keys → Create Key.

• Create a key for the Support Bot, scoping it to openai:gpt-6-luna (restricting it to the cheaper model)
• Create a key for Data Analytics, scoping it to openai:gpt-6.1-sol (allowing the more capable model)

Step 2: Configure Budgets
Go to the Budgets page and set a monthly spend limit for each key, e.g., $25 each. Once usage exceeds the limit, Otari will automatically block further requests to prevent runaway costs.

Step 3: Verify Budget Enforcement
Send a request using the Support Bot's key:

curl http://localhost:8000/api/v1/chat/completions \
  -H "Authorization: Bearer tk-SUPPORT_BOT_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai:gpt-6-luna",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

If the budget is exhausted, you'll receive a 403 HTTP status with the error code budget_exceeded. The gateway intercepts the request before forwarding it to the provider, saving you from writing budget logic in your application layer.

FAQ & Troubleshooting

Q1: Getting a 401 error with the OpenAI SDK?
Ensure your api_key isn't missing the key string itself. In curl, you manually add -H "Authorization: Bearer tk-xxx", but the OpenAI SDK handles this automatically. Just pass api_key="tk-xxx".

Q2: How do I add other model providers?
Simply add them to the providers section in config.yml. Otari supports 40+ providers, including Anthropic, Google Gemini, Mistral, Cohere, and more. Refer to Otari's official configuration documentation for exact field names.

Q3: I don't want to install PostgreSQL for local development?
No problem. Change the database_url in your config to sqlite+aiosqlite:///./otari.db. This switches the backend to SQLite, perfect for local debugging. Stick to PostgreSQL for production, though.

Q4: Does the gateway itself cost anything?
Otari is open-sourced under Apache 2.0 and completely free. You only pay your model providers' rates. The gateway's value lies in eliminating multi-key management overhead and preventing cost overruns.

Q5: How can I auto-retry failed requests with another provider?
Use the Dashboard's Routing page to configure a fallback strategy. For example: try OpenAI first; if a 5xx error or timeout occurs, automatically route to Anthropic. This is classic failover routing.

Summary

Today, we accomplished four key milestones:

1. Launched an Otari gateway instantly with Docker.
2. Sent our first API requests via curl and the OpenAI Python SDK to verify seamless compatibility.
3. Deployed a persistent, production-ready instance with PostgreSQL.
4. Implemented virtual key distribution and budget caps for project-level cost isolation.

Next Steps:

• Explore the Routing features in the Dashboard to configure automatic failover for mission-critical apps.
• Check out extended capabilities like --profile code-exec and --profile web-search to enable code sandboxes and live web search.
• Point your production app's base_url to your server, secure it with Nginx + HTTPS, and go live.

If someone on your team is struggling with scattered model keys and unpredictable cloud bills, send them this guide. A 30-minute setup can save months of debugging and cost headaches down the line.