Model comparison · 2026-09-11
Evaluating language models in Spanish: ODESIA, IberBench and ALIA
A global average says nothing about how a model behaves in Spanish. Which public evaluation resources exist, how to build your own evaluation set, and why we do not publish a ranking.
A global average does not describe your case
Most of the figures circulating about models are aggregate averages, and an aggregate average blends languages, domains and task formats. A model can lead that average and still do worse than another on your specific Spanish-language case.
The mismatch grows when the real job is not open conversation but something bounded: classifying tickets, extracting fields from an invoice, rewriting legal text without changing its meaning. Variation between models there is wider than the distance between their published averages.
The public resources you can use
Spanish has, unlike many languages, public evaluation infrastructure. Three starting points, with links, instead of a list of opinions.
ODESIA, coordinated from UNED, publishes leaderboards over discriminative Spanish-language tasks with undisclosed test sets, which reduces the risk of training contamination.
IberBench extends the approach to Iberian languages and to Ibero-American varieties, grouping dozens of industry-relevant tasks alongside the academic ones.
ALIA is the public infrastructure of models in Spanish and the co-official languages, coordinated by the Barcelona Supercomputing Center; its open model family, Salamandra included, is a useful comparison point even when it is not the model you end up running in production.
How to build your own evaluation set
None of the above replaces measuring on your own data. The good news is that a useful set is smaller than people assume: fifty to two hundred real cases, with the correct answer written by somebody who knows the domain, already separates models.
- Take real cases, including the ones that fail today; a set of easy cases distinguishes nothing.
- Write the success criterion before you look at any model output.
- Include dialect variety if your audience has it: peninsular and Latin American Spanish do not always behave alike.
- Run every candidate with the same prompt and the same temperature, through a single API surface so the comparison stays clean.
- Store the result with a date and the exact model identifier; without that you cannot repeat the measurement in six months.
What we are not going to claim
We do not publish a ranking of which model is best in Spanish. We have not run that measurement rigorously enough to defend it, and an invented table would be worse than no table.
What the platform does offer is the substrate for you to run it: several multilingual models reachable with one key and one API surface, at prices that stay exactly as listed in the catalog, so swapping a candidate does not mean rewriting the evaluation harness.
Frequently asked questions
Do open models work well in Spanish?
It depends on the task, which is precisely the reason to measure. There are open models trained with explicit attention to Spanish and generalist models that do well without it; the comparison has to be made on the specific case.
Can I use ALIA through LLMTR?
The ALIA models are a public resource of the project coordinated by the Barcelona Supercomputing Center. This article cites them as a comparison point; it does not claim they are available in the LLMTR catalog.
How many cases does a useful evaluation need?
Fifty well-chosen real cases already show differences; at two hundred, small differences stop being noise. The quality of the cases matters more than the count.