Timeouts and slow models

Most timeout errors do not originate in the gateway: the model is still producing a response when the client exhausts its own read budget and closes the connection. This page explains how long to wait per model and which settings keep the connection open.

Why a request times out

On a non-streaming call (stream: false), the first byte of the body is only sent once the response is fully generated. If the model spends 90 seconds generating, the client sees no bytes on the socket for 90 seconds. Most HTTP clients default to a 30-60 second read timeout, and the request is cancelled client-side when it expires.

When that happens:

The request counts as failed and is not billed.
It is recorded server-side as provider_error or provider_timeout.
Retrying the same request usually produces the same result, because it waits the same amount of time again.

Two fixes

1. Use streaming

With stream: true the gateway sends response headers immediately and keeps the connection alive while the model thinks. The read timeout never fires because bytes keep arriving. This is the recommended path for every model that produces long answers.

import openai

client = openai.OpenAI(
    base_url="https://llmtr.com/v1",
    api_key="llmtr-your_key",
)

stream = client.chat.completions.create(
    model="llmtr/qwen3-6-35b",
    messages=[{"role": "user", "content": "Write a function"}],
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

See /docs/en/gateway/streaming for the streaming format.

2. Raise the read timeout

If you cannot stream, set the client timeout from the model's real duration. For Turkey-hosted models, 300 seconds is a safe starting value.

client = openai.OpenAI(
    base_url="https://llmtr.com/v1",
    api_key="llmtr-your_key",
    timeout=300.0,
)

curl --max-time 300 "https://llmtr.com/v1/chat/completions" \
  -H "Authorization: Bearer llmtr-your_key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llmtr/gemma-4",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Expected generation speeds

The values below are measured from production records, not theoretical figures. Estimate duration as expected tokens ÷ speed.

Model	Generation speed	Typical response time
`llmtr/gemma-4`	30-45 tokens/sec	15-20 sec for a 512-token output
`llmtr/qwen3-6-35b`	~8 tokens/sec	Over 2 minutes for 1000+ token outputs
`llmtr/medgemma-4b`	5-7 tokens/sec (warm)	The first request can take minutes while the model loads
`llmtr/trendyol-asure-12b`	4-5 tokens/sec	Close to 2 minutes for a 512-token output
`llmtr/ornith-1-35b`	~3.6 tokens/sec	~18 minutes for a 4000-token output

Models from global providers (OpenAI, Anthropic, Google, Qwen and others) are outside this table and typically answer within seconds.

Prompt processing time on long inputs

On Turkey-hosted models, generation speed is not the only cost: processing the prompt for the first time takes time too, and how long that takes differs noticeably per model. llmtr/gemma-4 reads long prompts quickly — a 20,000-token prompt is processed in seconds — while on the other Turkey-hosted models the same prompt can take minutes before the model produces a single token. When working with long contexts:

Scale your client read timeout with the prompt size instead of leaving it at a fixed 60 seconds.
Resend the same prefix (system instructions, document, conversation history). The prefix is cached: in measurement a 1,603-token prompt took 3,985 ms on the first call and 714 ms on a second call with the same prefix, and cached tokens are billed at a lower rate.
For very large prompts, make the first request separately and then continue on top of the same prefix.

Cap the response length

Duration scales directly with the number of tokens generated, so capping with max_tokens is the fastest improvement when a short answer is enough.

{
  "model": "llmtr/qwen3-6-35b",
  "messages": [{"role": "user", "content": "Summarize in one sentence"}],
  "max_tokens": 256
}

Batch jobs

For batch jobs that send many sequential requests:

Stream every request, so one stuck request does not stall the whole job.
Retry failed requests with backoff, not immediately and not indefinitely.
If you hit a concurrency ceiling, honor the Retry-After header on 429 or 503 model_busy responses.

/docs/en/gateway/streaming — streaming format and event structure
/docs/en/gateway/errors — error types and HTTP codes
/docs/en/usage — viewing status codes and latency for your requests