The question is no longer whether an open model can compete with a proprietary one, but on which specific tasks it does so and at what real cost. During 2025 many teams migrated part of their AI stack to downloadable weights and discovered two things: that savings on API costs do not always offset infrastructure costs, and that the license matters more than it seems when the product goes into production.
Here you will not find a flat ranking. You will find which families of open models are delivering defensible results, where they fall short against paid ones, and how to decide using criteria that will survive the next release.
The real map: what «open source» means in language models
Before comparing, it is worth dismantling an ambiguity that leads to wrong decisions. Almost no model called open meets the strict OSI definition. What is common is to find downloadable weights with community-use licenses that impose restrictions: monthly user limits, prohibition of competitive use, the obligation to display the model name in the interface, or commercial scaling clauses.
This has a practical consequence: a model can be free to prototype with and expensive — legally — as it grows. The Llama family is the classic example of a license with a user threshold; Qwen and DeepSeek publish much of their weights under Apache 2.0, which facilitates commercial integration without legal friction; Mistral combines Apache licenses on its small models with research licenses on the large ones. Before choosing based on benchmarks, review the license of the specific model you are going to deploy, not that of the repository in general.
Open models worth testing in 2025
Llama (Meta) remains the de facto standard of the ecosystem. Its advantage is not raw performance but the maturity of the tooling: ready-made quantizations, support in Ollama, vLLM and llama.cpp, and a community that resolves questions quickly. The 8B variant is a workhorse for classification, extraction and summarization; the larger sizes compete in reasoning but demand hardware that is rarely on a developer’s desktop.
Qwen has gained ground in multilingual and code tasks. Its small versions surprise in SQL generation and in understanding instructions with strict formatting, and the Apache license on many of its weights eliminates the legal debate. In tests of extracting structured data from messy text, a medium-sized Qwen model is often enough where a large proprietary model would be overkill.
DeepSeek has shown that a lab with fewer resources can publish competitive models in mathematical reasoning and code. The trade-off is opacity regarding training data and the need to validate biases in regulated domains.
Mistral maintains a European offering with good results in instruction tasks and an active community around its quantization formats.
Gemma (Google) offers open weights with a conservative approach to safety, useful when you need a model that does not generate problematic content by default, although its license also has conditions of use.
Where the open model wins without discussion
There are scenarios where the comparison does not even make sense because the proprietary option cannot offer what you need.
Data that cannot leave your infrastructure
If you process clinical histories, legal files or personal data under strict regulations, sending prompts to an external API requires a risk analysis that many legal teams do not approve. An open model run locally eliminates data transfer. It does not eliminate responsibility: you still need to record which model you used, with which version and which prompts, because traceability is also an obligation.
High and predictable volume
When your load is constant and the model fits on a GPU you already have, the marginal cost per token tends toward zero. A typical case: classifying thousands of daily tickets into fixed categories. A 7-8B model quantized to 4 bits runs on a consumer GPU and performs that task with an accuracy that, after light fine-tuning with your own examples, matches much larger proprietary models in that narrow domain. The key is that the task be bounded; if you need open-ended reasoning, the calculation changes.
Controlled latency and no dependence on third parties
For interactive applications where 300 ms more ruins the experience, or for products that cannot afford a provider outage, self-hosting gives control. It also gives you the obligation to monitor it, something many teams underestimate.
Where the paid model remains the sensible option
Honesty requires saying it: in complex multi-step reasoning, in following long instructions and in tasks that require context of hundreds of thousands of tokens, frontier proprietary models maintain an advantage. A medium-sized open model can write a correct function; asking it to reason about an entire repository and propose a coherent refactoring usually produces fragile results.
There is also a factor that rarely appears in comparisons: the cost of maintenance. Choosing an open model means versioning weights, managing updates, setting up continuous evaluation and accepting that one day the community will stop maintaining the quantization you use. That work has a price in engineering hours that, in small teams, exceeds API spending for months.
A common mistake I have seen repeated
Teams that migrate to an open model because it scores well on a public benchmark and then discover that their specific use case — for example, extracting fields from scanned invoices with heterogeneous templates — fails systematically. The benchmark measured general reasoning, not the ability to respect a JSON schema with nested fields and null values. The lesson: build an evaluation set with 50-100 real examples from your domain before deciding. If the open model does not pass that test, the savings are fictitious.
How to compare without falling into ranking noise
Leaderboards change every week and mix tasks that are not alike. A more useful approach is to build your own decision matrix with four axes:
- License and compliance: does it allow commercial use without thresholds? Does it require attribution? What happens if you exceed a certain volume?
- Hardware fit: does it fit on your GPU with the quantization you need? What real throughput do you get with vLLM or llama.cpp?
- Quality in your domain: results on your evaluation set, not on MMLU.
- Total cost: API versus infrastructure plus maintenance hours.
A technical detail that makes a difference and is often overlooked: quantization degrades some tasks more than others. Creative generation holds up well at 4 bits; mathematical reasoning and precise data extraction suffer more. Testing the same task with the model in fp16 and in quantized form will tell you whether the VRAM savings justify the loss.
Hybrid strategy: what works in practice
The free-versus-paid dichotomy is misleading. The architecture that delivers the most performance in real projects combines both: a small open model for the bulk of the traffic (classification, filtering, short summaries, FAQ responses) and a proprietary model as a fallback for queries that the small one escalates due to low confidence. This pattern reduces cost without sacrificing quality in difficult cases, and it is easier to justify to the business than a full migration.
It is also worth remembering that «free» does not exist in inference: someone pays for the electricity, the GPU and the time of whoever maintains the deployment. The right question is not which one is free, but which one has a lower total cost for your volume and your team.
What to watch in the coming months
The pace of publishing open weights is not slowing, and the gap with frontier models is closing in some tasks while holding in others. Two fronts deserve attention: licenses, which tend to harden when a model becomes popular, and evaluation tools, which remain the weak link. A model without a good evaluation system of your own is a black box with good press.
If I had to summarize the criterion: test open models first on bounded, measurable tasks, reserve proprietary ones for what requires broad reasoning, and do not make any decision without an evaluation set built with your own data. That is the only comparison that does not expire the following month.