How to evaluate the Ember-1 research preview

Short answer

Ember-1 is a research model Fireworks built on Kimi K3 and trained to reach its answer with a shorter reasoning trace. Fireworks publishes research releases as two weeks of serverless access and makes them permanent based on community demand. On LLMTR this row runs until 00:00 on 6 October 2026 (Istanbul); after that the fireworks/ember-1 identifier returns 410 model_retired and suggests moonshot/kimi-k3 at the same price.

So the evaluation comes down to two questions: does the shorter reasoning trace make a difference on your own tasks, and is the move to Kimi K3 ready when the date arrives. Both can be answered in a short time with the same task set.

• Source: Fireworks Ember-1 announcement (checked 26 September 2026)
• LLMTR model page: Ember-1

What Fireworks claims

According to Fireworks' announcement, Ember-1 delivers Kimi K3's quality with 40 percent fewer tokens. The same announcement says that across seven benchmarks and two customers' production traffic, Kimi K3's reasoning could be shortened by 35 to 50 percent. Because reasoning tokens are billed as output, that shortening can show up directly in cost and waiting time.

These are Fireworks' own evaluations. Whether the same ratio appears on your tasks becomes clear when the same prompts are sent to both models and the output tokens are compared. Whether the shortening preserves quality is measured separately, by looking at the answers themselves.

• Source: Fireworks Ember-1 announcement (checked 26 September 2026)

Differences between Ember-1 and Kimi K3

The two models have the same price and the same context window. The differences appear in input format and structured output; if a flow depends on either capability, the move needs a code change.

• Source: Fireworks Ember-1 model page (checked 26 September 2026)
• LLMTR model page: Kimi K3

A meaningful comparison in a short time

Because the preview window is short, the evaluation uses a small but representative task set: a few dozen examples taken from your real work, with known expected answers. The same set runs on both models, and quality, output tokens and cost are recorded for each answer. Scoring the answers with the model name hidden keeps expectations from colouring the scores.

Since both models cost the same, any cost difference comes only from token counts. The example below calculates the cost of each call from the usage field of the response.

Running the same tasks on Ember-1 and Kimi K3

import os
from openai import OpenAI

client = OpenAI(base_url=os.environ["LLMTR_BASE_URL"] + "/v1", api_key=os.environ["LLMTR_API_KEY"])
PRICE = {"input": 3.00, "cached": 0.30, "output": 15.00}  # per 1M tokens, same on both models

def run(model: str, prompt: str) -> dict:
    res = client.chat.completions.create(model=model, messages=[{"role": "user", "content": prompt}])
    u = res.usage.model_dump()
    cached = (u.get("prompt_tokens_details") or {}).get("cached_tokens") or 0
    cost = ((u["prompt_tokens"] - cached) * PRICE["input"] + cached * PRICE["cached"]
            + u["completion_tokens"] * PRICE["output"]) / 1_000_000
    return {"model": model, "answer": res.choices[0].message.content,
            "completion_tokens": u["completion_tokens"], "cost_usd": round(cost, 6)}

# The same task set on both models; answers are scored with the model name hidden.
for prompt in test_prompts:
    for model in ("fireworks/ember-1", "moonshot/kimi-k3"):
        record(run(model, prompt))

What to look at in the evaluation

In a short evaluation, recording several things together for each answer means the result rests on comparable numbers rather than a single impression:

• Correctness: whether the answer matches the expected answer known in advance.
• Completeness: whether every requested point appears in the answer.
• Output tokens: completion_tokens including the reasoning trace; this is where Fireworks' claim shows its equivalent in your own work.
• Response time: as the reasoning trace gets shorter, the time to the first visible word gets shorter too.
• Format compliance: if structured output or tool calling is used, whether the answer arrives in the expected form.

Preparing for the move

After 6 October 2026, requests to fireworks/ember-1 return 410 model_retired and the response names the successor; the move does not happen on its own. Keeping the model identifier in configuration rather than buried in code turns the move into a one-line change. An error handler that treats a 410 response separately also keeps the date from surfacing as a generic "model not responding" error.

The two capabilities that do not carry over can be handled in advance. If images are sent as https addresses, converting them to base64 data URLs now avoids image errors at the move. If strict json_schema is used, validating the same schema on your side gives the same assurance with json_object on Kimi K3.

• LLMTR model page: Kimi K3

Data policy

Ember-1 runs on third-party infrastructure abroad, and prompts and completions may be logged by the provider; it is not a Turkey-hosted LLMTR model. Building the evaluation set from examples without personal data or customer secrets is the most comfortable way to try a preview model.

• LLMTR model page: Ember-1

Frequently asked questions

What does a request to Ember-1 return after 6 October 2026?

It returns 410 model_retired, and the response names moonshot/kimi-k3, at the same price, as the successor. The request is not redirected to Kimi K3 automatically; the model identifier has to be changed in code.

What happens if Fireworks makes the model permanent?

Fireworks says it may make research releases permanent based on community demand. In that case the date on LLMTR is reconsidered; the current status appears on Ember-1's model page.

Are 40 percent fewer tokens guaranteed?

No. That is Fireworks' own evaluation. You can see the difference on your tasks by sending the same prompts to both models and comparing output tokens.

What changes when moving to Kimi K3?

Price and context stay the same. Images are sent only as base64 data URLs, and json_object is used instead of strict json_schema; schema validation moves to your side.

Can a preview model be used in production?

It can be used until 6 October 2026. Because the end date is known, keeping the model identifier in configuration and trying the move to moonshot/kimi-k3 in advance keeps the date from turning into an outage.

Related posts

• Ember-1 reasoning tokens, quality, and cost
• Tool calling and JSON Schema with Ember-1
• Screenshot and error analysis with Ember-1
• Ember-1 context, cache, and usage cost

• Ember-1 model
• Kimi K3 model
• Model catalog
• Chat Completions guide
• Pricing