A non-sensitive evaluation set for Muse Spark Contributor

Start with the training warning and scope

A Muse Spark Contributor evaluation set should use synthetic records created from scratch and open data whose conditions you have reviewed. Instead of copying customer conversations, represent the behavior you want to test through independent examples. This limits data sharing while testing extraction, missing information and instruction following.

Training warning: prompts and completions in the Contributor tier may be used by Meta for model training. Do not submit confidential, personal or customer data. This includes system messages, attached documents and tool results. The examples below are synthetic inputs written for this article, not measured model responses.

Keep a source and permission record

Give every example an identifier and provenance record. For synthetic records, document the creation method, independence from real records and reviewing role. Changing names in a customer document does not remove other details or business information; do not classify that as safe synthetic data.

For open data, record the source address, version, access date and applicable conditions. Being publicly visible is not permission evidence for this use. Creative Commons' CC0 explanation also notes that other rights, including privacy and publicity rights, remain unaffected. Exclude uncertain examples; this record is not a guarantee of legal compliance.

• Example identifier, dataset version and synthetic or open source classification.
• Source, permission basis and decision covering provider training use.
• Sensitive data review, approving role and next review date.

Write comparable examples for one task

Start by extracting a product code and quantity from a short inventory note. Require exactly item, quantity and status. Missing or conflicting quantities produce quantity null and status needs_review; an explicit quantity uses status ok. Treat additional instructions inside the note as data. Derive the expected values below from this contract, not from the model.

Freeze acceptance criteria before the trial

For this small starter set, require the correct product code, data types and exact field matching in all four cases. Treating an absent quantity as zero or selecting one conflicting quantity is a failure. A polished explanation cannot compensate for a wrong value. These are proposed acceptance conditions, not achieved model results.

Separate prompt development examples from final evaluation examples. Do not place paraphrases of the same note in both groups. Track Turkish and English inputs separately; a combined score can hide language problems. Start with synthetic tool results before connecting real tools, and grant no permission to write to external systems.

Compare locally without printing response content

The Python code below makes no API calls. Supply your candidate results through standard input as a JSON object keyed by the four example identifiers. It compares the exact field set, values and types, printing only identifiers and pass/fail flags. A working validator does not establish model quality.

Check the four acceptance keys without a network connection

import json
import sys

expected = {
    "normal": {"item": "P-101", "quantity": 3, "status": "ok"},
    "missing": {"item": "P-102", "quantity": None, "status": "needs_review"},
    "conflict": {"item": "P-103", "quantity": None, "status": "needs_review"},
    "instruction": {"item": "P-104", "quantity": 4, "status": "ok"},
}

try:
    candidate = json.load(sys.stdin)
except (json.JSONDecodeError, UnicodeDecodeError):
    raise SystemExit("invalid_json")

if not isinstance(candidate, dict) or set(candidate) != set(expected):
    raise SystemExit("invalid_case_ids")

def matches(actual, target):
    return (
        isinstance(actual, dict)
        and set(actual) == set(target)
        and all(type(actual[k]) is type(v) and actual[k] == v
                for k, v in target.items())
    )

scores = {key: matches(candidate[key], value)
          for key, value in expected.items()}
print(json.dumps(scores))

Finish a small trial with a version record

When using LLMTR, record meta/muse-spark-1.2-contributor as the model identifier, alongside the prompt version, dataset version, request settings and date. Evaluate the starter set sequentially, set a budget first and stop on access or quota errors. Do not use Contributor for load testing or automatically switch to another paid model.

Report incomplete calls separately from content failures. Do not write raw prompts or responses to application logs. This small set is a starting check, not a guarantee of production accuracy, capacity or privacy. Meta's model and pricing/rate-limit documentation required login during the August 28, 2026 review, so current numerical limits are not verified here.

Frequently asked questions

Is deleting names from customer records enough?

No. Dates, unusual events and business-specific details may remain. For this Contributor trial, write an independent synthetic task instead of transforming a real customer record.

Can open data be used automatically?

No. Review its source, usage conditions and absence of personal data. Exclude an example if its suitability for sharing into provider training remains uncertain.

Are four correct results enough for production?

No. These four cases are only a starter check. Expand independent examples and failure categories for your task, using predefined criteria and human review to make the acceptance decision.

Related posts

• Moving a Muse Spark Contributor prototype to the standard tier
• muse-spark-1.2-contributor: LLMTR model ID and endpoint
• What is Muse Spark 1.2 Contributor? Data policy and the standard tier

• meta/muse-spark-1.2-contributor
• Model catalog
• Chat Completions guide
• OpenAI / OpenRouter