Model comparison ยท 2026-08-29
The models.dev LLMTR provider entry and metadata sync
A look at the LLMTR provider entry in the models.dev open metadata database, the model files beneath it, and where those records have drifted from the live catalog, including a retired id that still reads as selectable.
models.dev is a metadata database, not an endpoint
models.dev is an open metadata database that holds model ids and capabilities in TOML files. It accepts no requests, produces no responses and carries no keys. Agent tools and clients read the model list, context limits and capability flags from it, then present them to the user as options. It is a discovery source, not a runtime component.
That distinction has a practical consequence. Without an entry a provider does not become unusable; a user who types the id by hand still gets a result. What is lost without an entry is discovery: the model never appears on the selection screen, so nobody learns it exists.
Two discovery paths need to be told apart. Tools that read from the metadata database see the list once an entry is merged, and opencode belongs to that group. Tools that fetch the gateway model endpoint at their own startup are independent of the database and wait for no update. Both paths use the same canonical ids; the only difference is when the list refreshes.
The provider entry is five fields
The entry sits on the add-llmtr-provider branch of the knowhycodata/models.dev repository. The provider definition is the provider file under providers/llmtr, and it consists of five fields. Do not let the brevity mislead: those five lines together decide which client the tool installs, where it reads the key from and where it sends requests.
The display name field is the label on the selection screen. The environment field tells the tool which variable holds the key, and it matches the name used by tools that ship LLMTR as a built-in provider, so a user moving between the two paths never has to rename a variable.
The package field does the most work: because an OpenAI-compatible client is selected, no tool has to write an LLMTR-specific adapter, and the existing client is reused with a different base URL. The documentation field points a user at the docs, and the API field at the gateway root.
The provider definition under providers/llmtr
name = "LLMTR"
env = ["LLMTR_API_KEY"]
npm = "@ai-sdk/openai-compatible"
doc = "https://llmtr.com/docs"
api = "https://llmtr.com/v1"Drift between the model files and the live catalog
The same fork holds six model files under providers/llmtr/models: gemma-4, magibu-11b-v8, medgemma-4b, qwen3-6-35b, sincap and trendyol-7b. The catalog carries more Turkey-hosted rows than that today, and one of the six records belongs to a retired id.
The drift runs in two directions and each direction causes a different problem. An id present in the metadata but retired in the catalog steers a user toward a choice that cannot work. Rows live in the catalog with no metadata record at all never appear on the selection screen, so an existing capability goes unused. The table below shows the current state row by row.
| Model id | Metadata record | Catalog status |
|---|---|---|
| llmtr/gemma-4 | Present | Public and live |
| llmtr/qwen3-6-35b | Present | Public and live |
| llmtr/medgemma-4b | Present | Withdrawn on 21 August 2026 |
| llmtr/magibu-11b-v8 | Present | Public and live |
| llmtr/trendyol-7b | Present | Public and live |
| llmtr/sincap | Present | Retired |
| llmtr/trendyol-asure-12b | Absent | Public and live |
| llmtr/muse-glimmer-30b-tr | Absent | Public and live |
| llmtr/qwen3-5-4b | Absent | Public and live |
| llmtr/ornith-1-35b | Absent, withdrawn | Withdrawn on 21 August 2026 |
| llmtr/embeddinggemma-300m | Absent | Public and live, embeddings endpoint |
The retired record problem: llmtr/sincap
The llmtr/sincap id is retired in the catalog today. LLMTR neither deletes nor hides a retired model: the row stays visible and its retired state is stated on its own page. Anyone searching for the id can therefore see what the row was, when it was retired and why it no longer answers. An id that vanished silently would make after-the-fact debugging impossible.
On the metadata side, however, the record still reads as a selectable model. The result is a chain failure: a tool that builds its list from the database offers a retired id, the user picks it, and the request comes back refused with an explanation once it reaches the gateway. The refusal is correct; what is wrong is that the option entered the list at all. The proper fix is to mark the record retired in the metadata too.
This row has no defined successor, because the catalog holds no Turkey-hosted equivalent doing the same job. Not auto-forwarding is deliberate: silently switching to a model that behaves differently and bills differently misinforms a caller whose code still names the old id. For general text and tool-calling work, llmtr/gemma-4 and llmtr/qwen3-6-35b are live options, and the caller who knows the workload decides which fits.
- A retired id stays visible in the catalog, but requests are not accepted.
- Until the metadata record is marked retired, tools keep offering that id.
- For rows without a successor the caller picks the replacement; nothing forwards automatically.
- The gap between a retirement date and a metadata update reaches the user directly as an error.
Capability flags decide the outcome of a selection
A metadata record carries more than an id. Tool calling, image input and context limits live there too, and agent tools use those flags to decide whether a model enters the list. A correct id with a wrong flag still produces a wrong selection.
The damage runs both ways. A model that genuinely calls tools looks unfit for agent work because the flag is missing and drops off the list; a model that refuses tool calls gets recommended because the flag is wrong and stalls at the first tool request. Flags therefore have to be measured per model. Two rows in the same family need not share a capability, and the table below shows exactly that among the Turkey-hosted rows.
| Model id | Tool calling | Image input |
|---|---|---|
| llmtr/gemma-4 | Yes | Yes |
| llmtr/qwen3-6-35b | Yes | No |
| llmtr/qwen3-5-4b | Yes | No |
| llmtr/muse-glimmer-30b-tr | Yes | Yes |
| llmtr/trendyol-asure-12b | No | Yes |
| llmtr/magibu-11b-v8 | No | No |
Keeping the entry in sync with the catalog
The LLMTR model endpoint is readable without a key. Producing or verifying a metadata record needs no credentials, so the sync check reduces to a small script that can run continuously.
The comparison yields three sets. Ids found only in the metadata are retired or mistaken records. Ids found only in the live list are missing records. Ids present in both but differing in their fields are flag and limit drift, and that is the most insidious group, because the id looks right and the difference goes unnoticed.
Do not treat the check as a one-off. New rows and retirement decisions can land between two metadata updates, so the live list is always the reference. Tools reading the record wait for a merge while tools fetching live see the same change immediately; the sync check is what keeps both groups looking at the same ids.
- Mark or remove ids that exist only in the metadata.
- Add records for rows that exist only in the live list.
- Check context limits and capability flags against a live response.
- Store the comparison with a timestamp so each update's change set stays readable.
Compare metadata records against the keyless catalog read
# The catalog is readable without a key, so no credential is needed
curl -s https://llmtr.com/v1/models
# Only the ids of the Turkey-hosted rows
curl -s https://llmtr.com/v1/models | grep -o '"id":"llmtr/[^"]*"'
# Record files in the metadata fork
ls providers/llmtr/modelsFrequently asked questions
Can I use LLMTR models without a models.dev entry?
You can. The database is a discovery source rather than a runtime component. Type the canonical id by hand and the request works normally. What you lose without an entry is the tool listing the model on its selection screen.
What happens if I call a retired id?
The gateway refuses the request with an explanation. That is not a network error and retrying will not clear it. The catalog row stays visible so the id and the reason it no longer answers remain readable. For rows without a defined successor you choose the replacement yourself.
What does the package field in the entry do?
It decides which client the tool installs. Because an OpenAI-compatible client is selected, no tool needs an LLMTR-specific adapter; the existing client is reused with a different base URL. Get this field wrong and a tool can install the provider but cannot send requests.
What follows from a wrong capability flag?
Damage in both directions. A model that can call tools looks unfit and drops off the list because the flag is missing, while a model that cannot gets recommended and stalls at the first tool request. This is why flags must be measured per model rather than copied across a family.