The noise around open models
When a leadership team asks about open source LLMs for the first time, the conversation usually starts with a big promise and an even bigger doubt. Myths circulate faster than technical documentation. And decisions are made, in many cases, based on second-hand information. This article does not aim to sell the adoption of open models, but rather to separate what can be verified from what is repeated without evidence.
Myth 1: «Open source is free»
The model weights can be downloaded at no cost. That is true. What is not free is everything needed for those weights to run stably in production. The real costs appear on four fronts:
- Infrastructure: a GPU with enough memory to serve a 70B model with long context is not a laptop. Hourly rental or hardware purchase carries a cost that recurs every month.
- Engineering: someone has to set up the inference server, tune batching, monitor latencies, and manage model versions.
- Evaluation: every model or prompt update requires re-measuring quality. Without your own test bench, you are navigating blind.
- Maintenance: weight repositories and inference libraries evolve quickly. What worked six months ago may require adjustments today.
In practice, the cost per token of an open model served on your own infrastructure can be competitive or even lower than a closed API, but only above a certain sustained volume. Below that threshold, the API usually wins on total cost. According to documentation from cloud infrastructure providers, average utilization of GPUs dedicated to inference at small companies rarely exceeds 30%, which makes each served token more expensive.
The mistake of counting only the token price
A pattern that repeats in teams that make the leap: they compare the API’s price per million tokens with the GPU’s hourly price and conclude that the open model is cheaper. They forget the opportunity cost of the platform team, the integration hours, and the time spent resolving incidents. An honest calculation includes those hours, even if only approximately.
Myth 2: «If it’s open, my data is secure by default»
Running the model on your own infrastructure eliminates data leaving to a third party, but it does not magically make the system secure. Security depends on how it is deployed, not just where it runs. The points that usually fall outside the initial conversation:
- Prompt and response logs. If they are stored without controls, sensitive data remains exposed, even if the model is local.
- Access to the inference endpoint. Without authentication and usage limits, anyone inside the network can query the model with the service’s permissions.
- The model supply chain. Weights downloaded from a public repository must be verified; a corrupted or tampered file is a real vector.
- Content filtering. An open model does not come by default with the moderation layers that a managed provider applies in its API.
The privacy advantage exists, but it is a necessary condition, not a sufficient one. It requires the same data governance work as any other internal system.
Myth 3: «They perform worse than closed models»
Here the nuance matters a great deal, and generalizing is the mistake. In complex reasoning, mathematics, and code tasks, top-tier closed models have maintained an advantage in public benchmarks for much of 2025 and early 2026. But in scoped tasks—ticket classification, extracting fields from a document, summarizing conversations with a fixed format, generating responses from retrieved context—mid-sized open models match or outperform small closed models, with the advantage that you can fine-tune them with your own data.
A concrete case that illustrates the difference well: an insurance company that processes claim reports in PDF. A general-purpose closed model made errors when extracting amounts when the document mixed tables and free text. After fine-tuning an 8B open model with 2,000 internally annotated examples, accuracy on that specific format rose notably and the cost per document dropped. Not because the open model was better in general, but because it was specialized to the exact distribution of their documents. That is the terrain where open models shine.
Where they fall short (and it’s worth acknowledging it)
A 7B or 8B open model is not going to handle queries that demand multi-step reasoning over 200,000-token contexts with the same robustness as a high-end closed model. Nor is it the natural choice for long-range creative tasks where stylistic coherence is critical. If your use case depends on that, the open model is not the answer, and forcing it generates frustration in the team and distrust in the technology.
Myth 4: «Just download the model and you’re done»
The distance between downloading weights and having a production service is the part least documented in introductory articles. The intermediate steps include:
- Choosing an inference library suited to the available hardware.
- Defining the prompting scheme and versioning it alongside the code.
- Setting up a retrieval layer (RAG) if the model needs up-to-date company knowledge.
- Establishing quality metrics and a representative evaluation set.
- Configuring usage limits, queues, and controlled degradation during demand spikes.
According to official documentation from the main serving libraries, performance per GPU varies greatly depending on the quantization technique and batch size. The same model can go from serving 10 requests per second to 40 depending on the configuration. That difference does not appear on the model’s spec sheet; it appears when you measure.
Myth 5: «Open source guarantees total independence from the vendor»
Replacing dependence on a closed API with dependence on a GPU provider, a specific inference library, or a single weight repository is not independence—it is changing dependencies. Real freedom comes when the team can migrate the model without rewriting the application. That requires an abstraction layer in the code, a portable prompt format, and a test bench that allows comparing candidates by the same criteria.
Companies that have done this step well usually have an internal catalog of evaluated models, not a single model. They can switch from one to another based on cost, latency, or quality for a specific task. Open source enables that flexibility; it does not guarantee it.
Reality: when it makes sense and when it doesn’t
After seeing several deployments, the decision pattern is fairly consistent:
Signals in favor
- High and stable inference volume that justifies the infrastructure investment.
- Data that cannot leave the perimeter due to regulatory or contractual requirements.
- Need for deep specialization in a domain with labeled proprietary data.
- Critical latency in environments where the network to an external provider adds an unacceptable hop.
- Reproducibility requirement: knowing exactly which model version responded to what.
Signals against
- A team without a platform or MLOps profile and tight deadlines.
- Highly varied use cases that require a top-quality general-purpose model.
- Low or intermittent volumes where the closed API is cheaper.
- Absence of proprietary data for specialization: without fine-tuning or RAG, the open model competes at a disadvantage.
A warning about evaluations
Public benchmarks are a useful reference, but they do not predict performance in your case. A model that leads a general-purpose table can fall short on the specific task that matters to you. The only reliable measure is to build an evaluation set with real examples from your business, anonymized if necessary, and compare candidates with the same prompt and the same correctness criteria. Without that test bench, any decision about open models rests on opinions, not data.
And it is worth remembering a limitation that does not go away: no model, open or closed, guarantees results. System quality depends on the data, prompt design, continuous evaluation, and the human judgment that reviews outputs at critical points. Open source adds control and options; it does not replace the work.