Sovereign AI gateway: one API, all your models, costs centralized

Two approaches cost a lot. Sending everything to a US API exposes your personal data and inflates a per-token bill. Hosting everything in-house means sizing the GPU for the most demanding task, while most of the volume needs little intelligence.

So the gateway picks, request by request, the model that fits the task and the sensitivity of the data.

• Private route, flat fee: requests carrying personal or sensitive data go to an internal open-weight model on a dedicated GPU in Europe. These are high-volume tasks that need little intelligence, such as OCR of handwritten documents, correcting learners' work, rewording or classification. This is how our AI hosted in Europe works, billed as a flat monthly fee.
• Cloud route, per token: content generation without personal data goes to Mistral La Plateforme, in the EU cloud, on your own Mistral account.
• Blocked request: a request that contains a disallowed type of personal data is stopped before any cloud call.

The control detects personal data and blocks the request, without trying to mask it. By default, only email addresses are allowed to the cloud, and any other type of personal data detected stops the request. This filter strongly reduces the risk without being an absolute guarantee.

Under the hood, the gateway assembles open source components (vLLM Semantic Router behind an Envoy proxy) and exposes an OpenAI-compatible API.

The gateway exposes one endpoint, in the OpenAI API format. Your applications talk to it as they would to OpenAI; behind it, the gateway distributes requests between open-weight models hosted in Europe and, for generation without personal data, Mistral La Plateforme in the EU.

• Mistral: an open-weight European model, capable and commercially licensed.
• Llama, Qwen, GLM, DeepSeek: to cover other task profiles such as reasoning, code, long context and multilingual.
• One API contract for all of them: you switch models by changing a parameter, without rewriting your code.

Your existing tools (OpenAI SDK, libraries, agents) work without adaptation: the base URL and the key change, while your code stays as is.

Not every request deserves the biggest model. The gateway applies cost-aware routing: a simple task (classification, extraction, short summary) goes to a small, fast, cheap model; a demanding task goes to a more capable one.

• Weighted distribution: you split traffic across several models by weight, for example to test a new model on a fraction of calls.
• Two routes, two rules: a request carrying personal data stays on the private route, which never falls back to the cloud and returns an explicit error if the internal model is unavailable. A request without personal data can be served by a model hosted in the EU, such as Mistral La Plateforme.
• A gain you measure: tuned well, this routing can reduce latency and the volume of billed tokens, depending on your request mix. The gain depends on your real usage and shows up in measurement, with no figure promised up front.

Without guardrails, a generative-AI bill drifts fast, and no one knows which team consumes what. The gateway is the right place to centralize that governance.

• Virtual keys: keys per team, per project or per application can be set up on request, at deployment, without exposing the credentials of the underlying models.
• Caps and budgets: a monthly budget or a spend cap per key is set up the same way, to keep the bill in check.
• Consolidated costs: once those keys are in place, consumption across all models is broken down by team and project. The CIO and CFO then see a single cost line instead of ten scattered invoices.

Per-team keys and per-key spend caps are not included as standard in the current version. They are set up on request, at deployment. As standard, consumption is tracked per route and per model.

That turns "we don't know what AI costs us" into a metric you can steer, the way FinOps does for cloud.

Per token, cost follows volume, and every document processed adds to the bill. High-volume, low-intelligence tasks (OCR of handwritten documents, correcting learners' work, rewording, classification) are exactly the ones that drive that meter up. On the private route, they move to a flat monthly fee.

The AI Instance starts at €160 excl. VAT per month, and that amount does not depend on the number of requests as long as the machine absorbs the load. Beyond that, the next tier is quoted.

Content generation without personal data stays per token, with Mistral, on your own account. The budget becomes predictable and can be expressed in euros per user (or per learner) per year, all the more since usage is tracked route by route.

Costing starts from real measurements. During the prototype, we measure the number of tokens consumed per document, route by route, then derive the annual cost per user. Until then, any performance or cost figure is indicative.

At low volume, a per-token API can be cheaper on paper, and we would rather say so. The value of the flat fee lies in a predictable budget, control over your data and the volume that moves to a fixed cost.

The gateway sits in the path of every request. That position lets you apply controls before the prompt reaches a model.

• In-path inspection: calls go through a single point where you can log, filter and apply rules.
• Fail-closed on personal data: the default posture is to refuse rather than let through. If a routing rule cannot be satisfied while staying sovereign, the request is blocked instead of being redirected outside Europe.

This is a security posture, a set of configured rules; it is not an absolute guarantee. It reduces the margin for error; it does not replace your own risk analysis.

Many organizations already have an AI charter or usage policy. It often remains a document nobody rereads at the moment of calling a model. The gateway turns each rule of that policy into a setting applied to every request, without relying on everyone's vigilance.

• “Sensitive data stays in-house” becomes a fail-closed private route. If the internal model is unavailable, the request fails with an explicit error and never goes to the cloud.
• “No personal data at a cloud provider” becomes blocking of detected personal data, with an allowlist of permitted types (by default, email addresses only).
• “Every use has an owner” translates into one API key per feature, per application or per team, set up on request at deployment, on top of the standard tracking per route and per model.
• “No dependency on a single provider” becomes a parameter. Switching model or provider happens in the gateway configuration, and your applications stay unchanged.

Cost governance is covered above, as is request control in the part on in-path security. For implementation, the Private GPT technical guide covers these settings and the recommendations for IT and security teams.

A gateway is only worth what it routes to. Here, the gateway and the models it calls stay in Europe.

• The entry point is hosted on European infrastructure.
• The internal model runs on a dedicated GPU, hosted in Europe by a European company. Your data is never used by us to train a model, and the list of our subprocessors is part of the GDPR Article 28 contract.
• The cloud route calls Mistral La Plateforme, in the EU, with no switch to an API outside the EU. The private route never falls back to the cloud.

A sovereign AI gateway controls the data path end to end, where a generic gateway stops at the entrance.

The gateway applies technical rules. It replaces neither your compliance work nor your own risk analysis.

• Controller obligations: the legal basis, retention periods and data-subject rights remain your responsibility, and the gateway does not cover them.
• Impact assessment: for sensitive data or vulnerable people, such as minors, a data protection impact assessment (DPIA, GDPR Article 35) will likely be required. It is carried out before going to production.
• AI instance without SLA: the internal model runs on a single machine, with no redundancy. It is monitored, but with no availability commitment, as Bunker's SLAs cover application hosting. Because the private route is fail-closed, an outage results in an explicit error and no request goes to the cloud.
• Filter without absolute guarantee: personal-data detection strongly reduces the risk, but unusual or malformed data can slip through.

This gateway is the routing and governance layer of our Private GPT, our chat and RAG app, ready for your teams.

Private GPT provides the interface (chat, RAG over your documents, access management); the gateway brings the multi-model routing, cost-aware routing and cost governance underneath. You can also call the gateway directly from your own applications.