Choosing a language model to work in Spanish is not solved with a scoreboard. Most public comparisons focus on benchmarks in English, and when you transfer those figures to a real case in Spanish, you discover that the model that won on MMLU loses the thread in a conversation with Argentine slang or mixes Catalan terminology with Spanish. This guide does not give you a closed ranking: it gives you a reproducible configuration procedure so that you can measure for yourself which model performs best in your specific case, with concrete tests and decision thresholds.
Step 0: define what «performance in Spanish» means for your project
Before touching an API or downloading weights, write down on a sheet of paper which tasks you are going to delegate to the model. A support ticket classifier is not the same as a legal summary generator. The axes that truly discriminate between models in Spanish are four:
- Reading comprehension: extracting answers from a long text without inventing data.
- Fluency and register: ensuring the text does not sound like machine translation from English.
- Dialectal robustness: behavior with Rioplatense, Mexican, Peninsular, or Andean variants.
- Instructional fidelity: respecting formats (JSON, lists, lengths) in Spanish.
A model can be excellent in the first and disastrous in the third. According to official documentation from several providers, their multilingual evaluations aggregate results by language, which hides regional differences. That is why step 0 is essential.
Step 1: build the test bench with your own real texts
Gather between 20 and 40 of your own examples. Customer emails, excerpts from your knowledge base, product pages, or call transcripts all work. Manually label the expected answer in a separate column. This detail makes the difference: public benchmarks use templates that models have seen in training; your texts have not.
What to include mandatorily
- At least five examples with code-switching (Spanish with English terms, common in technology).
- Two or three with figures and units (dates, amounts, percentages) to detect formatting errors.
- One example with a negative instruction: «do not mention prices» or «respond only with the table.»
- Texts with accents, eñes, and opening punctuation marks (¿ ¡). It is a cheap test that rules out models with poor tokenizers for Spanish.
Step 2: configure the comparison environment
Do not compare models from different chat interfaces: they change the default system prompt, the temperature, and the truncation. Use a single script that calls each endpoint with identical parameters. A skeleton in Python with each provider’s standard library is enough.
params = {
«temperature»: 0.2,
«max_tokens»: 800,
«top_p»: 1.0
}
Set a low temperature (0.1–0.3) for extraction tasks. If your case is creative, repeat the battery with temperature 0.8 and compare stability across runs. A model that varies a lot with the same seed is difficult to integrate into production.
The mistake almost everyone makes here
Comparing only the first response. Large models tend to get it right on the first turn and degrade as the context grows. Add a two- and three-turn test with accumulated context of 3,000–6,000 tokens in Spanish. In my experience with customer service assistants, the sharpest drop appears right there: the model starts responding in English or loses the format instruction when the history gets long. That limitation does not show up in any public comparison table.
Step 3: define metrics you can calculate without a team of annotators
You do not need a laboratory. With three simple metrics you get enough signal:
- Exact or overlap accuracy: for extraction, compare with the labeled answer using keyword matching or string similarity.
- Format violation rate: percentage of responses that do not respect the requested schema (invalid JSON, bulleted lists when prose was requested).
- Anglicism penalty: count of English terms not justified by the domain. A reasonable threshold is more than two per 200 words in informative texts.
If you need to evaluate fluency, ask two colleagues to rank pairs of responses blindly. With 30 pairs you get a fairly stable preference. That is enough to decide.
Step 4: run the battery and read the results with skepticism
Run each model three times per example and record the variance. A model with 85% accuracy and high deviation is worse for production than one with 80% stable. Also document latency and cost per 1,000 tokens: in Spanish, the average length of responses is usually 10–20% longer than in English for the same content, and that makes real use more expensive even if the price per token seems low.
What to expect depending on the type of model
Medium-sized open models tend to hold up well in classification and extraction with clear instructions, but they are weak in dialectal nuances and in maintaining formal register over long texts. Large proprietary models tend to produce more natural Spanish, although they sometimes «overcorrect» and generate an artificial neutral Spanish that sounds strange to a local reader. None wins at everything. If your product is for Spain, test explicitly with examples that use «vosotros» and Peninsular vocabulary; if it is for LatAm, prevent the model from imposing «ordenador» where your user says «computadora.»
Step 5: adjust the prompt before changing models
Many differences between models disappear with a well-written system prompt in Spanish. Include the output language explicitly, the desired register, and an example response. Models interpret instructions better when the prompt itself is in the target language, even though they understand English perfectly.
Try this sequence: first optimize the prompt with model A; then apply the same prompt to model B. If B improves a lot, the problem was the prompt and not the model. If B still fails on the same examples, there you have a real limitation that justifies the change.
Step 6: decide with a matrix, not with a hunch
Score each candidate from 1 to 5 on four criteria weighted according to your case: accuracy on your battery, stability across runs, total cost, and latency. Multiply by the weight and add up. A model that scores 4 in accuracy but 2 in stability is rarely the right choice for an automated workflow.
Add a «risk» column: if the provider does not document how it handles data in Spanish or does not offer zero retention, that factor can weigh more than two points of accuracy. Provider transparency about its training data is a quality signal, not an ornament.
Limitations that this methodology does not solve
It does not systematically evaluate cultural biases, nor does it cover co-official languages (Catalan, Galician, Basque) unless you add your own examples. Nor does it predict behavior after a provider update: models change without notice and your battery must be repeated periodically. And it does not replace human review in regulated domains. If your case is healthcare or legal, expert validation is mandatory regardless of the score.
Finally, keep in mind that performance in Spanish is not a fixed property of the model: it depends on the tokenizer, the context length, and the domain. A model that shines in general conversation may fail on technical terminology in your sector. The only comparison that matters is the one you run with your data, your parameters, and your acceptance criteria. The rest are hints, not verdicts.