Ling-3.0-Tiny local API: vLLM environment and model identity

What does a local Tiny API change?

Ling-3.0-Tiny can be served on your own hardware using its official open weights and a vLLM environment with Ling support. This does not reactivate inclusionai/ling-3.0-tiny, retired on LLMTR on August 15, 2026. The local server is a separate service whose setup, capacity, and access controls you manage.

This guide uses sources checked on August 28, 2026. These examples were not executed on hardware here; no memory, speed, or successful model response measurements are claimed.

Prepare the matching vLLM environment

The InclusionAI model card specifies the ling_3_0 branch of vllm-ling-v3 and installation using precompiled components. Do not assume any vLLM package has equivalent model support. Record the branch commit, weights revision, and dependency versions.

The Bash example assumes a prepared Linux GPU environment and an existing BF16 weights directory. The general vLLM GPU documentation states that native Windows is unsupported; this Linux recipe does not establish Tiny compatibility on Mac or WSL.

• Check that your driver, GPU, and runtime match.
• Use weights, tokenizer, and configuration files from the same revision.
• Set MODEL_PATH to the local directory and LOCAL_LING_API_KEY to a separate secret for this server.

Keep four different names separate

The weights source, disk location, and API model name are different fields. In vLLM, --served-model-name defines the name clients submit. This example retains auto from the official recipe. It does not request automatic LLMTR model selection or routing to another service.

Listen locally before accepting remote traffic

The example adds a local listening address and key authentication to the model card’s vLLM settings. --trust-remote-code requires trusting model code; inspect its source and revision first. Checking the directory also helps prevent treating a repository name as a local path.

127.0.0.1 only restricts the HTTP listener. Apply firewall controls to internal communication ports too. --api-key does not protect every server endpoint; this command is not a complete public production setup.

Local vLLM server in a prepared Linux environment

set -eu
: "${MODEL_PATH:?Set MODEL_PATH to the reviewed local BF16 directory}"
: "${LOCAL_LING_API_KEY:?Set an independent local API secret}"
test -d "$MODEL_PATH"

vllm serve "$MODEL_PATH" \
  --host 127.0.0.1 \
  --port 8000 \
  --api-key "$LOCAL_LING_API_KEY" \
  --trust-remote-code \
  --served-model-name auto \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.85 \
  --enable-prefix-caching \
  --mamba-cache-mode align \
  --enable-auto-tool-choice \
  --tool-call-parser ling3 \
  --reasoning-parser ling3

Send the same model name from the client

After startup, send this Python request from the same Linux environment. It uses only the standard library. The address points directly to local vLLM; LLMTR_API_KEY is not its credential. Changing the serving name requires changing the request model field too.

The 1024-token limit and 120-second client timeout are example choices, not model limits or speed guarantees. Thinking is configured inside chat_template_kwargs. Do not append LLMTR-specific suffixes to the local name. If the output budget expires, inspect finish_reason; that alone does not indicate a broken connection.

Sends a synthetic message without printing prompt or response content

import json
import os
from urllib.request import Request, urlopen

payload = {
    "model": "auto",
    "messages": [{"role": "user", "content": "Compute 17 times 23."}],
    "chat_template_kwargs": {"enable_thinking": True},
    "temperature": 1.0,
    "top_p": 0.95,
    "top_k": 20,
    "max_tokens": 1024,
    "stream": False,
}
request = Request(
    "http://127.0.0.1:8000/v1/chat/completions",
    data=json.dumps(payload).encode("utf-8"),
    headers={
        "Content-Type": "application/json",
        "Authorization": "Bearer " + os.environ["LOCAL_LING_API_KEY"],
    },
    method="POST",
)
with urlopen(request, timeout=120) as response:
    result = json.load(response)
print({
    "model": result.get("model"),
    "finish_reason": result["choices"][0].get("finish_reason"),
})

Do not mix parser and NEXTN recipes

This example follows the ling3 tool and reasoning parser pair in the Hugging Face card’s vLLM section. The card’s SGLang example and SGLang’s own Tiny cookbook diverge: the cookbook uses deepseek-r1 and glm45 parsers and excludes a Tiny NEXTN recipe.

Do not transfer SGLang flags into this vLLM command. For an unknown-parser error, check which environment supplies the vLLM executable before changing options arbitrarily. This guide adds no NEXTN, YaRN, or other extended-context settings to the vLLM example.

Diagnose the first failure at the right layer

For a refused connection, check readiness and port. For authentication failures, compare local keys; for an unknown model, compare the serving name with the request. Receiving model_retired from LLMTR means the request reached the old service, not your local server.

Start with a short synthetic message, then increase context and concurrency separately. Do not log production prompts or model outputs. Local caching settings are not an LLMTR billed cache discount. The official LLMTR successor is the paid inclusionai/ling-3.0-flash; switching to it is a separate decision.

Frequently asked questions

Must the local API model be named inclusionai/ling-3.0-tiny?

No. This recipe uses --served-model-name auto and the client sends auto. You may choose another name, provided the server and client agree.

Does a single-GPU recipe mean it fits every GPU?

No. Weights, runtime memory, context, and concurrency must be considered together. This guide makes no minimum-memory or performance claim for any GPU.

Why can Tiny run locally if it is retired?

Retirement concerns LLMTR API access. Official weights and running them in your own environment are separate matters; local inference sends no request to LLMTR.

Related posts

• Ling-3.0-Tiny download guide: choosing BF16, FP8 or INT4
• Ling 3.0 Tiny: migrating the retired LLMTR API route to Flash

• inclusionai/ling-3.0-tiny
• Model catalog
• Chat Completions guide
• OpenAI / OpenRouter