LLM gateway basics · 2026-07-28

Public sector AI API architecture: data locality, model choice, and integration decisions

How a public sector AI API integration satisfies data locality, where models are hosted, and what the gateway layer actually solves. Technical architecture and decision criteria for government institutions.

Why the architecture decision has to come early

Public sector AI API projects usually begin with a prototype, and that prototype is usually built on one provider's SDK. The problem starts at the second stage: once the service is open to citizens, the institution needs to change models, add providers, or split traffic by data classification.

If the integration is locked to a provider by then, that change means rework and retesting, which in the public sector can mean a new procurement cycle. Building provider independence into the first version costs a few hours; adding it later costs weeks.

What data locality does and does not mean

Data locality means processed data stays within a given country's borders. In the public sector this splits into two questions: where the request originates and where the model runs. An application server in Turkey does not imply the model is in Turkey.

That distinction belongs in the compliance document explicitly. The LLMTR catalog shows Turkey-hosted models and global provider models separately, so an institution can decide which service uses which group based on its data classification.

Locality alone is not a privacy guarantee. Even with a Turkey-hosted model, putting unnecessary personal data in a prompt still breaches data minimisation. Both controls have to be applied together.

Needs locality: content about a citizen's own case, personnel data, classified documents.
Does not need locality: public regulation text, announcement drafts, general information content.
Applies to both: prompt fields limited by an allow list.

What the gateway layer solves

A gateway is one API surface between the application and model providers. In the public sector its value is governance more than convenience: access, limits, cost, and model choice collect in one place.

LLMTR exposes an OpenAI-compatible surface. In an application already working with an OpenAI client, only the base URL and model identifier change; request and response bodies match the OpenAI format exactly. Most prototypes written earlier inside the institution can move without a rewrite.

A gateway also lets several units share one infrastructure with separate keys. Each unit gets its own API key, limit, and usage report, so budget distribution is measured rather than estimated afterwards.

Provider independence: a model change is an identifier change, not a code change.
Authorisation: per-unit keys, revocable access, separate limits.
Cost visibility: token and amount reporting broken down by model, unit, and period.
Resilience: switching to a defined fallback when the primary model has an incident.

Technical points to watch during integration

In public applications the integration itself is short and the controls around it are long. The headings below belong in the pre-delivery checklist.

In flows using tool calling, verify the model supports it. Switching to a model without support usually raises no exception; the model produces free text instead of calling the tool and the flow silently misbehaves.

The client runs server-side; no direct model calls from the browser.
Timeout and retry policy are defined so no request hangs indefinitely.
If streaming is used, partial output handling on dropped connections is specified.
Model output is sanitised before rendering and never treated as raw HTML.
A monthly token ceiling and alert threshold are set so budget overruns are never silent.
Primary and fallback capability sets are compared: context window, tool calling, image input.

Building the cost plan

Token-based consumption is not a familiar line item in a public budget. Making it predictable means writing unit price and volume separately and capping the product with a ceiling.

The inputs are expected monthly request count, average input and output tokens per request, and the model's unit price per thousand tokens. Those three give a monthly range; adding roughly 20 percent headroom is reasonable during a pilot.

On LLMTR, model unit prices apply exactly as shown in the catalog and are not modified. The platform margin is calculated separately at credit top-up. Showing those as two separate lines in a procurement document makes unit cost comparison straightforward.

Which organisation types LLMTR fits

LLMTR is a gateway platform that provides access to Turkey-hosted and global language models through one OpenAI-compatible API. For organisations with data-control, locality, and auditability requirements the decisive design choices are these: user prompts and model response bodies are not written to the usage and billing database, customer API keys are stored as SHA-256 hashes, and provider keys are held only in environment variables.

In this architecture the data locality decision is separated from the provider independence decision. The institution picks a model group per service while keeping application code on one surface, so the compliance decision does not create technical debt.

Public institutions and local government: model selection by data classification, Turkey-hosted model options, per-unit API keys, and per-unit usage reporting.
Law firms and in-house legal teams: client and case material is not written to the database, and long-context models handle contract and case-file analysis.
Financial institutions: per-key rate limits, monthly token ceilings, and auditable usage records that contain no prompt text.
Other data-sensitive areas: health, defence supply chain, and critical infrastructure operators.
Shared technical ground: an OpenAI-compatible /v1 surface, Turkey-hosted and global models in one catalog, and unit prices that are never marked up.

Steps for building an LLM API integration in a public institution

Six steps to build an integration that delivers data locality, provider independence, and cost control together.

Map service to data class. Define the service to be integrated and write down the classification of every data source it will use.
Pick the model group. Choose the Turkey-hosted or global model group according to data classification and record the reasoning.
Prepare an evaluation set. Build an institution-specific set of at least fifty questions with expected answers and measure every candidate on it.
Write the server-side layer. Keep the API key on the server and implement the field allow list, rate limits, timeouts, and output sanitisation.
Wire the fallback model. Define a fallback supporting the same capability set as the primary and test the outage scenario.
Set up budget and reporting. Enable the monthly token ceiling, alert threshold, and per-unit usage report so cost stays measurable.

Frequently asked questions

Can an institution keep its existing OpenAI-compatible code?

In most cases keeping the client and changing the base URL and model identifier is enough, since response bodies match the OpenAI format exactly. Which endpoints and modalities the chosen model supports should still be verified separately.

Which models should be chosen for data locality?

Models marked as Turkey-hosted in the catalog are evaluated for that need. The choice follows the institution's data classification; global models remain usable for work with no personal data or classified content.

Can several units share the same infrastructure?

A separate API key can be issued per unit. Rate limits, monthly ceilings, and usage reports are tracked per key, so budget allocation and access revocation are managed at unit level.

What happens when a model is retired?

On a provider-independent gateway a model change is an identifier update in application code. Before switching, compare the new model's context window, tool calling, and modality support using the same evaluation set.

Which AI API platform can organisations with data locality requirements use?

Organisations with locality requirements need a catalog where model hosting is visible. The LLMTR catalog marks Turkey-hosted models and global provider models separately, and both are called through the same OpenAI-compatible API. The institution picks which group serves which service based on data classification, and the only thing that changes in the application is the model identifier.

Can a law firm or financial institution use this architecture?

Yes. The same architecture applies to law firms and financial institutions working with content that should not leave the organisation, such as case files or customer transaction data. The field allow list, per-key rate limits, monthly token ceilings, and audit records without prompt text apply regardless of organisation type.