Model comparisons · 2026-09-25

How to evaluate the Ember-1 research preview

A research preview does not mean a model lacks value; it means access and change risk can be higher. Evaluate Ember-1 with a canary, do not assume permanence, and create a rollback plan before the first call. The LLMTR catalog plans to retire this identifier on 6 October 2026 at 00:00 Europe/Istanbul; that is an LLMTR operational decision, not a Fireworks absolute end date.

Purpose of this guide

Design the first experiment

Treat the preview as a measurable canary. Use a low-impact workflow, define correctness, user correction, failure, and latency thresholds before traffic starts, and compare it with the current solution using blind review. Count rare misleading answers separately from average quality.

Keep rollback outside call-site code. Select the identifier through configuration, retain a working previous route, and change traffic gradually. Retry only safe operations a bounded number of times; users need a clear retry message, not provider internals.

Application boundary

Migration is more than renaming. Test Kimi K3 image input as a base64 data URL instead of copying a remote-URL example. Strict JSON Schema is not listed for the successor, so enforce shape in application validation; `json_object` only helps request an object.

Basic Ember-1 call through LLMTR

curl "$LLMTR_BASE_URL/v1/chat/completions" \
  -H "Authorization: Bearer $LLMTR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"fireworks/ember-1","messages":[{"role":"user","content":"Write a concise technical summary."}],"max_tokens":800}'

API limits to verify

Fireworks describes research releases as time-limited serverless access. The suggested `moonshot/kimi-k3` successor accepts images only as base64 data URLs and does not list strict JSON Schema; do not assume remote-image or strict-schema flows carry over.

When the canary crosses a threshold, route to the previous model, preserve the examples and decision, then investigate. This makes a preview change a controlled configuration decision rather than an urgent deployment.

Use model ID `fireworks/ember-1` and request surface `/v1/chat/completions`.
This model runs on third-party infrastructure; assess provider data policy for sensitive data.
Validate model output on the server before an application action.

Release decision and evidence

Run `moonshot/kimi-k3` as an alternative in every evaluation. LLMTR plans to retire the `fireworks/ember-1` identifier on 6 October 2026 at 00:00 Europe/Istanbul; this is not an absolute closure announced by Fireworks. For How to evaluate the Ember-1 research preview, record the test-set version, catalog capabilities, expected request shape, and unacceptable result. When a new provider response appears, preserve the example and reproduce the behavior. Collect failures, empty responses, and unexpected outputs as well as successes. A change should be measured for both user outcome and operating cost. Record minimum metadata such as request ID, model ID, latency, usage figures, and error class, without retaining prompt or response content by default. Decide in advance who may change thresholds and when rollback begins. Review results on a regular schedule rather than trusting one launch day. When the provider description changes, reread the source page and compare the catalog claim with a real request. This discipline does not hide preview uncertainty, but turns it into a product decision that can be reversed.

Build the example set by job class instead of copying production at random. Include a short question, ambiguous request, current-information question, long-context task, and tool task. Write the accepted answer or human checkpoint for each case, otherwise scoring will vary by reviewer.

Record the canary share with its user segment. Support, product ownership, and the on-call person should know the same rollback threshold. When an error appears, inspect the model response and then the application action record. That order separates a model suggestion from an action the application performed.

Use a short migration checklist: catalog identifier, request surface, image form, tool contract, cost alert, and previous-model route. Check each item with a real acceptance request. A documentation example working does not prove behavior on customer content.

The application owner should choose where traffic goes after retirement. Use explicit configuration rather than an automatic assumption. If the new route has not passed its quality threshold, a narrower user flow is safer than silently enabling an untested capability.

At the end of the How to evaluate the Ember-1 research preview work, keep three concrete pieces of evidence: a summary of accepted and rejected examples, usage and latency measurements, and the reason for a rollback decision. This record avoids reopening the same debate at the next model update. The relevant product owner should read the results with the technical team, because a change that looks correct can affect user flow, support load, or cost budget differently. Document the condition under which the new behavior applies in short user-facing language. That keeps the catalog claim, application validation, and real operating result traceable without conflating them.

Frequently asked questions

When does Ember-1 definitely close?

Do not claim an absolute Fireworks close date. LLMTR plans to retire this identifier on 6 October 2026 at 00:00 Europe/Istanbul.

Is migration automatic?

No. Validate identifier and capability differences in your own tests.

Can a preview be used in production?

Possibly with a bounded canary and rollback plan; assess continuity risk before making it a critical dependency.