The noise around fine-tuning
Few artificial intelligence techniques have accumulated as many disproportionate expectations as the fine-tuning of language models. It is often presented as a magic wand that turns a generic model into an expert assistant just by uploading a CSV of examples. The reality is far more prosaic: fine-tuning is a surgical tool with a real cost, clear limitations and, in many cases, cheaper alternatives that work better.
This article is not a command tutorial, but a contrast between what is said in forums and what happens when the process is run with real data. If you are looking for an infallible recipe, you won’t find it here. If you want to know when it is worth it and when you are throwing money away, keep reading.
Myth 1: «With 100 examples you already have an expert model»
It is probably the most repeated claim in discussion threads. The official documentation of the main providers of fine-tunable models usually recommends datasets ranging from several hundred to thousands of high-quality examples, depending on the complexity of the task. One hundred examples may work for a very specific binary classification, but not for an assistant that must reason about internal regulations.
There is a detail that is almost never mentioned: quality weighs more than quantity, but not linearly. A dataset of 500 examples reviewed by a person with good judgment can outperform one of 5,000 generated automatically with the model itself. The phenomenon is known as model collapse when training iteratively with synthetic data without human filtering.
A specific case that illustrates the problem
In a support ticket classification project, a team generated 3,000 labeled examples with a large model and fine-tuned a small model. Initial accuracy looked excellent in validation, but when deployed to production, the ambiguous categories (complaints that mixed billing and technical support) degraded. The cause: the automatic labeler inherited its own biases and the fine-tuned model learned to imitate those errors with great confidence. The solution was not more data, but manually reviewing 200 ambiguous cases and adding explicit rules to the system prompt.
Myth 2: «Fine-tuning replaces RAG»
This confusion appears constantly. They are different techniques that solve different problems:
- Fine-tuning: modifies the model’s behavior (style, format, tone, decision criteria). It is useful when you want the model to always respond in a specific way.
- RAG (Retrieval-Augmented Generation): injects external knowledge at inference time. It is useful when the information changes frequently or is too extensive to fit in training.
If your problem is that the model does not know your product catalog updated every week, fine-tuning is a bad choice: you would have to retrain every time. If your problem is that the model responds with a tone that is too informal for customer service, RAG won’t fix it. According to official documentation from platforms such as OpenAI or Anthropic, the usual recommendation is to start with advanced prompting, then RAG, and only afterwards consider fine-tuning.
Myth 3: «It’s cheap and fast»
The visible cost of training has gone down, but the total cost has not. There are three items that are usually ignored:
- Data preparation: cleaning, deduplication, label review and formatting. In practice, it consumes more time than the training itself.
- Evaluation: you need an independent validation set and metrics that reflect real use. Without this, you are flying blind.
- Maintenance: every time the domain changes (new regulations, new product, new policy), the fine-tuned model may become outdated and the cycle must be repeated.
In many cases, a good prompt with examples (few-shot) and a low temperature offers 80% of the benefit with 5% of the effort. Fine-tuning is justified when that remaining 20% is critical for the business.
Reality: when fine-tuning makes the difference
There are scenarios where fine-tuning is clearly superior:
- Reduction of latency and cost per token: a fine-tuned small model can replace a large one in very narrow tasks, with lower latency and lower inference cost.
- Strict formats: when the output must follow a complex JSON schema or an internal report format, fine-tuning reduces the need for subsequent validations.
- Very specific domain with its own jargon: for example, incident classification in a regulated sector where the vocabulary does not appear in the general corpus.
- Consistent brand style: when tone is part of the product and cannot be delegated to instructions that the model sometimes ignores.
Outside these cases, it is worth asking whether the problem is really about the model’s behavior or about the input data.
The step-by-step that does matter (and that is usually not told)
Beyond running a training script, the real process has phases that determine the result:
1. Define the success criterion before touching data
«That it responds well» is not the same as «that it reduces escalations to a human by 20% in billing tickets.» Without an operational metric, any improvement is anecdotal.
2. Build an independent evaluation set
It must come from the same distribution as the production data, but never from the same training batch. A frequent mistake is evaluating with examples that the model has already seen, which inflates the results.
3. Format the data according to the provider
Each platform expects a different format (instruction-response pairs, conversations with roles, etc.). An incorrect format can go unnoticed and produce a model that ignores instructions.
4. Train with a suitable base model
Not all models support fine-tuning, and those that do have size and token limits. Choosing a model that is too small for a complex task leads to a fine-tuning that does not generalize.
5. Evaluate with metrics and with human review
Automatic metrics (accuracy, F1, BLEU) do not capture nuances of usefulness. A blind review by someone who did not participate in the labeling detects problems that the metrics hide.
6. Deploy with monitoring
The fine-tuned model can degrade if the input distribution changes. Without drift monitoring, the problem is detected when users complain.
Limitations that should be accepted
Fine-tuning does not eliminate hallucinations. A model fine-tuned with correct data can still invent information when asked something outside its distribution. Nor does it guarantee policy compliance: if the dataset contains ambiguous examples, the model will learn that ambiguity.
Another practical limit: catastrophic forgetting. When fine-tuning a model for a specific task, it can lose general capabilities it previously had. If you need a versatile assistant, aggressive fine-tuning can worsen performance on unforeseen tasks.
And a transparency notice: there is no formula that guarantees that fine-tuning will improve your case. The decision must be based on a controlled test with clear metrics, not on third-party promises.
Alternatives that sometimes win
Before fine-tuning, it is worth exploring:
- Prompt engineering with dynamic examples: selecting the most relevant examples for each query at inference time.
- RAG with good embeddings: for factual and up-to-date knowledge.
- External tools: delegating calculations or queries to APIs instead of expecting the model to do them from memory.
- Models already fine-tuned by the provider: in some cases, a commercially available specialized model covers the need without your own training.
The combination of several techniques usually gives better results than betting everything on fine-tuning.
Conclusion without promises
Fine-tuning LLMs is a powerful technique, but it is not a shortcut. It works when the problem is about behavior and there is enough quality data to define it. It fails when it is used to inject changing knowledge, when the data is noisy or when a small model is expected to match a large one in complex reasoning. The difference between a successful project and one that stays in the drawer lies less in the training and more in the definition of the problem, honest evaluation and the willingness to accept that, sometimes, the right answer is not to fine-tune anything.