Open-Source LLM Gateways: An Honest Review

An open-source LLM gateway can give a team control over deployment, routing rules, logging and provider credentials. It can also hand that team a new production service to secure, upgrade and support. Neither outcome is automatic. The right question is not “is open source better?” but “which operating responsibility are we willing to own?”

This review is intentionally balanced. Self-hosting can suit data-location requirements, bespoke routing or a platform team with mature operations. A managed gateway can be the safer choice when the team needs reliable delivery, support and a smaller attack surface today.

What you gain by self-hosting

The clearest advantage is deployment control. You choose the region, network path, database, observability stack, egress rules and upgrade timing. You can inspect the code, integrate a bespoke identity system and implement a routing rule that fits your workload rather than a vendor's menu.

Open source does not remove provider dependencies, prompt security, model evaluation or data governance. It changes who operates the layer between your application and those services.

What a managed service can do better

A managed service may provide reliable upgrades, incident response, credential handling, documented support and a polished console without requiring your team to maintain a multi-tenant proxy. Those are real advantages, especially where the gateway is not the product differentiator.

Ask for evidence, not assurances: service status history, security documentation, data controls, support terms, exported receipts, migration path and failure behaviour. A managed service that cannot emit request-level evidence can make cost and incident investigations harder. LLM observability tools provides a useful lens for the tracing and attribution part of that assessment.

The operational checklist people skip

Before self-hosting, identify the on-call owner, deployment process, secrets rotation, rate limiting, DDoS protection, database backup, metrics, alerts, incident runbook and upgrade cadence. Run a restore or failover drill. A gateway that merely starts in Docker is not an operated service.

Authentication deserves special care. Keep provider keys server-side, isolate projects or tenants, and make it impossible for a caller to choose arbitrary upstream URLs. Log request IDs and policy verdicts without defaulting to raw prompt retention. The LLM data privacy guide explains why a gateway's telemetry decisions are part of the deployment decision.

Compare with your actual workload

Trial both paths against the same staging workloads. Measure ordinary and tail latency, provider error handling, cache-hit preservation, model selection, receipts, tool safety and operator time. Include a failure exercise: take one provider down, revoke a credential and inspect what the application tells a user.

The LLM gateway overview explains the category and the project's own boundary. Whatever you choose, retain an explicit failure state. A gateway should not silently claim a fallback succeeded if it has no confirmed upstream response.

Decide with a written operating model

Set down who deploys the gateway, who holds production access, how emergency changes are approved, and how upstream credentials are rotated. Include a budget for dependency updates and security fixes. This clarifies the decision better than a licence debate: a self-hosted gateway is a service your team runs, while a managed gateway is a supplier relationship your team governs. Both require ownership; the work simply lands in different places.

Ask the same owner to describe a bad Tuesday: an upstream provider is slow, a newly deployed route is returning invalid output, and a customer needs an answer. If the response depends on a person who is neither on call nor documented, the operating model is not ready.

Next step

Make a one-page responsibility map. If your team cannot name the owner for upgrades, incident response and data retention, a managed option may be the safer immediate fit. If it can, run a focused staging trial before committing to a migration.

Frequently asked questions

Is an open-source LLM gateway cheaper?

It may reduce licence spend, but infrastructure, operations, security and engineering time remain costs. Compare total operating responsibility, not only a subscription price.

Does self-hosting solve data privacy automatically?

No. It controls one layer. Model providers, application logs, retrieval systems, access controls and retention still need their own documented decisions.

Who should self-host a gateway?

Teams with a genuine deployment or integration need and the ability to operate the service reliably. It is a poor fit when no one owns on-call, upgrades or security response.

What should we test before switching?

Test request compatibility, failure and fallback behaviour, identity isolation, telemetry, cache effects, provider receipts and the operator workflow during an outage.

Operating exercise

Revisit the decision record after the first incident drill. The gap between the planned operating model and the one people can execute is the most useful result of the exercise, and it should drive the next investment.

Decision record

Document the chosen operating model, the responsible service owner, the accepted risks, the data-handling configuration and the next review date. This lets procurement, security and engineering refer to one accountable decision instead of three different impressions of why the gateway was selected.