Comparing LLMs without getting lost in the noise
When someone asks what LLMs are and how they work, the quick answer is usually «language models that predict the next word.» It’s correct, but insufficient for deciding which one to use. The practical difference between one model and another isn’t in that definition, but in how they handle context, how much they cost per million tokens, how well they follow long instructions, and how far their attention window reaches before they forget what was said at the beginning of the conversation.
This article doesn’t repeat the textbook explanation. It focuses on what changes when comparing the most widely used families —GPT, Claude, Gemini, Llama, and Mistral— and on the criteria that truly make a difference in professional use: reasoning with extensive documents, code generation, operating cost, and control over data.
How an LLM works internally (just enough to compare)
An LLM is a neural network trained with a transformer architecture. It processes text by dividing it into tokens (word fragments) and calculates probabilities for the next token. That mechanism, repeated, produces coherent responses. What’s relevant for a comparison is that there are two clearly distinct phases:
- Pre-training: the model absorbs patterns from enormous text corpora. This is where its general knowledge and base reasoning capacity are defined.
- Alignment (fine-tuning and RLHF): it is adjusted to follow instructions, reject harmful requests, and adopt a specific tone. This is where its «character» and obedience are defined.
Two models of the same size can behave radically differently depending on how that second phase was carried out. That’s why comparing «parameters» without looking at alignment leads to wrong conclusions.
Context isn’t just a number
Technical specs boast context windows of 128,000, 200,000, or even a million tokens. In practice, retrieval quality within that window decays. A pattern that repeats when testing models with long contracts: information located in the middle of the document is retrieved worse than information at the beginning or the end. It’s the well-known «lost in the middle» effect, documented in industry studies on long-context evaluation.
Practical consequence: if you’re going to analyze a 300-page report, it’s not enough for the model to accept the document. It’s advisable to chunk it, index it, and perform semantic retrieval (RAG) instead of trusting that the model will read everything with the same attention.
Comparison of the most widely used families
GPT (OpenAI)
It’s the reference for its ecosystem: mature API, tools, functions, and a huge community. It stands out in code generation and in following structured instructions (JSON outputs, function calls). Its historical weak point has been verbosity and a certain tendency to please the user even when the premise is wrong. For data extraction tasks with strict formatting, it tends to be the most predictable option.
Claude (Anthropic)
It has positioned itself strongly in long-form writing and analysis of extensive documents. It tends to produce texts with fewer filler phrases and accepts nuanced corrections better. In editorial review or contract summarization tasks, many teams prefer it for its tone. Real limitation: in some very specific code workflows it performs below GPT, and its integration with external tools arrived later.
Gemini (Google)
Its advantage is integration with the Google ecosystem: search, Workspace, Android. The large context window and access to up-to-date information via search make it useful for tasks requiring recent data. The nuance: variability between versions (Flash, Pro, Ultra) is high, and choosing the wrong model for a task can drive up costs without improving the result.
Llama (Meta)
It’s the reference open family. It allows local execution or execution on your own servers, which addresses privacy requirements that cloud models don’t cover. Trade-off: it requires infrastructure (GPU), maintenance, and technical knowledge. For an SME without an ML team, the licensing savings are eaten up by operating costs.
Mistral
Efficient models, good in terms of quality/size ratio, with both open and commercial options. They fit well when you need a lightweight model for classification, extraction, or customer service with low latency. It’s not the best option for complex multi-step reasoning.
Quick decision table
| Need | Reasonable option | Why |
|---|---|---|
| Generate code and automate with APIs | GPT | Ecosystem and reliable structured outputs |
| Analyze long documents and write | Claude | Careful tone and good handling of extensive context |
| Up-to-date data and Google integration | Gemini | Integrated search and connection with Workspace |
| Privacy and local execution | Llama | Open model, full control of the data |
| Low latency and contained cost | Mistral | Efficient small models |
A real case that changes the decision
A legal team tried uploading a 180-page framework agreement to a model with a 200,000-token window. The surprise: the model correctly cited the clauses at the beginning and the end, but omitted a penalty clause located around page 90. It wasn’t a capacity failure, but one of distributed attention. The solution wasn’t to change models, but to chunk by sections, index with embeddings, and ask about specific clauses. This is the kind of limitation that doesn’t appear in benchmark comparisons and that you only discover when using it with real documents.
What no LLM does well (yet)
It’s worth saying clearly: no current model guarantees truthfulness. They can invent references, figures, or quotes with complete confidence. Nor are they aware of their limits: if you ask them about a fact they don’t know, they tend to fill in the gap. And they don’t reason in a strictly symbolic way: for financial calculations or formal logic, a spreadsheet or a rules engine remains more reliable.
Moreover, the cost per token may seem low until you scale. A workflow that processes thousands of documents daily can turn a modest monthly bill into a significant figure if the largest model is used for tasks that a small model would solve just as well.
How to choose without getting it wrong
- Define the task: classify, write, extract, reason. Each one has a model that stands out.
- Measure with your data: public benchmarks provide guidance, but they don’t replace a test with a hundred real examples from your domain.
- Calculate total cost: not just price per token, but also latency, retries, and human supervision.
- Review privacy: if the data can’t leave your infrastructure, the decision is reduced to open models.
- Prepare for substitution: using an abstraction layer (such as LangChain-type libraries or unified clients) prevents you from being tied to a single provider.
The useful comparison isn’t «which is the best model,» but «which is the best for this task, with this budget and these constraints.» That question changes every few months, and that’s why it’s advisable to build the workflow so that you can switch engines without redoing everything.