Training a language model from scratch is not an exercise in computational brute force. It is an engineering project with dozens of chained decisions where an early mistake can invalidate weeks of computation and thousands of euros of GPU. Most guides explain the ideal pipeline (tokenizer, dataset, pretraining, fine-tuning), but almost none dwell on the failures that actually happen when someone tries to put it into practice. This article covers precisely that: the most frequent mistakes and how to detect them before it’s too late.
Mistakes in the data phase: the source of almost everything
1. Measuring the dataset in gigabytes instead of useful tokens
A common mistake is boasting about a corpus by its size on disk. A compressed 50 GB file may contain less useful text than it suggests, and after deduplicating and filtering HTML boilerplate it may be reduced to a fraction. What matters is the number of tokens after actual tokenization, not the weight of the file.
How to avoid it: tokenize a representative sample (for example, 1 GB uncompressed) with the exact tokenizer you are going to use, calculate the tokens/character ratio, and extrapolate. Repeat the calculation after each cleaning phase. If the final corpus does not reach a reasonable volume for the model size, rethink the objective before launching training.
2. Deduplicating only with exact matching
Exact hash deduplication removes identical copies, but lets paraphrases, repeated templates, and near-duplicate fragments through that artificially inflate certain distributions. The model ends up memorizing recurring phrases and generating repetitive outputs.
How to avoid it: apply fuzzy deduplication by n-grams or by MinHash similarity over document blocks. You don’t need a perfect system: removing near-duplicates at the paragraph level already reduces a lot of noise.
3. Ignoring domain mixing and quality filtering
Mixing generic web, code, scientific articles, and forums without weighting produces models that imitate the tone of a forum in responses that should be formal. It is also common not to filter toxic content, spam, or automatically generated text.
How to avoid it: define explicit proportions per source, document where each block comes from, and apply lightweight quality classifiers. According to official documentation from several open projects, quality filtering has more impact on the final result than adding more low-quality data.
4. Contamination of the evaluation set
This is probably the most silent failure. If your validation set or your internal benchmarks appear, even partially, in the training data, the metrics lie. A model that «improves» on every evaluation may simply be memorizing.
How to avoid it: separate the evaluation sets before building the corpus, keep fingerprints (n-gram hashes) of those sets, and check for overlap against the training dataset. If there are matches, remove them from the corpus, not from the test set.
Mistakes in tokenization and architecture
5. Training the tokenizer with a corpus that does not represent the final domain
If the tokenizer is trained only on English and then the model is used in Spanish or code, the number of tokens per word skyrockets, the effective context window shrinks, and the cost of training and inference increases without it showing in the initial metrics.
How to avoid it: train the tokenizer with a stratified sample that reflects the real languages and domains of use. Verify with out-of-sample text how many tokens the same sentence consumes in different languages; if there are abysmal differences, review the vocabulary.
6. Choosing the model size without a clear compute budget
A larger model is not always better for a specific objective, and it also imposes memory and time requirements that many projects underestimate. Launching a pretraining without having calculated the total cost (GPU hours, checkpoints, storage, retries) is a fast track to ending up halfway.
How to avoid it: do a scaling test with a small model for a few hours, measure real throughput on your hardware, and extrapolate. If the budget does not cover the full training plus at least one restart from checkpoint, reduce the size.
7. Not defining a realistic context length
Announcing enormous contexts without data that takes advantage of them produces models that perform worse on long windows than the architecture suggests. Attention over long sequences makes training more expensive and, if the corpus is made up of short documents, the model never learns to use it.
How to avoid it: align the sequence length with the real distribution of your documents. If you need long context, include data with long structure and specifically evaluate tasks that require it.
Mistakes during pretraining
8. Not monitoring loss by domain
The overall loss can go down while a specific domain gets worse. If the model is overfitting to the generic web and losing capacity in code or in a specific language, the aggregate metric hides it.
How to avoid it: log validation loss broken down by source. A dashboard with separate curves per domain detects imbalances much earlier than the global average.
9. Insufficient or poorly managed checkpoints
Saving only at the end is a gamble. Hardware failures, cluster restarts, or numerical instability can force you to repeat days of computation. It is also common to save checkpoints without recording the exact associated configuration, which makes it impossible to reproduce the state.
How to avoid it: save periodic checkpoints with their configuration, seed, and optimizer state. Verify that resuming from checkpoint works before launching the long training run.
10. Ignoring numerical instability
Loss spikes, exploding gradients, or intermittent NaNs are signs that the training configuration is not stable. Many projects treat them as noise and keep going, and the final model ends up degraded in a way that is difficult to diagnose.
How to avoid it: use gradient clipping, review the initialization scale, and test with shorter sequences before scaling up. If NaNs appear, stop and analyze; do not restart blindly.
Mistakes in fine-tuning and evaluation
11. Confusing fine-tuning with fixing everything
Fine-tuning does not correct deficient pretraining. If the base model does not have sufficient capacity or data, fine-tuning only glosses over the result on the evaluation set and generalizes poorly.
How to avoid it: evaluate the base model before fine-tuning. If the base metrics are poor, invest in data and pretraining, not in more fine-tuning epochs.
12. Evaluating only with automatic metrics
Perplexity and accuracy on synthetic tasks do not describe real behavior. A model can improve on a metric and become more repetitive, less safe, or less useful in conversation.
How to avoid it: combine automatic metrics with human evaluations on a small but well-defined sample. Document the criteria and periodically review whether they still measure what matters to you.
13. Not documenting the process
Without a record of data versions, hyperparameters, and results, every experiment is irreproducible. This is not an academic detail: in production, when something fails, traceability is what allows you to diagnose it.
How to avoid it: version the dataset, the configuration, and the checkpoints with clear identifiers. A minimal but consistent record saves weeks of confusion.
A concrete case that illustrates the problem
In a real project fine-tuning a small model for technical answers in Spanish, the team detected that the model generated correct answers but with a strangely promotional tone. The cause was not in the fine-tuning: the pretraining corpus included a high proportion of marketing pages and product blogs that dominated certain topics. The overall loss was good and the automatic metrics did not detect it. It was corrected by rebalancing the domain mix and filtering promotional content, not by adding more data.
This type of problem is only detected by reading real model outputs regularly and comparing them with what is expected, not just by looking at loss curves.
What this article does not cover and why
Training an LLM from scratch at industrial scale requires distributed infrastructure, parallelism strategies, and memory optimizations that depend on the specific hardware and the chosen framework. What is described here focuses on judgment mistakes that affect any scale, from a small model on a single GPU to a larger project. None of these recommendations guarantees results: the final behavior depends on the interaction between data, architecture, and configuration, and there is always uncertainty.
If anything characterizes those who have successfully trained models, it is prudence: they assume that the dataset has problems, that the metrics can deceive, and that the first training run will not be the definitive one. Planning for that reality saves more time than any premature optimization.