Before touching code: what you really need to train an LLM
Training a language model from scratch isn’t about downloading a script and pressing enter. It’s an engineering project with three fronts you must resolve before the first line: data, compute, and evaluation. According to the official documentation of frameworks like PyTorch and Hugging Face, most failures in training projects don’t come from the code, but from poor corpus planning and poorly sized infrastructure.
This article describes a step-by-step setup assuming you want to build a small or medium model (between 100M and 1.5B parameters) for a specific domain, not a competitor to the large generalist models. If your goal is a 70B parameter model with massive multilingual data, the budget is measured in millions of dollars and this tutorial falls short: acknowledge that from the start.
Step 1: defining scope and the compute budget
Before writing the tokenizer, decide the model size based on the available GPUs. The rule of thumb used in the industry relates training tokens to parameters: a well-trained model usually consumes between 20 and 100 times more tokens than parameters. For a 125M parameter model, a corpus of 2.5B to 10B tokens is reasonable.
Estimating VRAM and time
The memory needed during training doesn’t depend only on the model size. The AdamW optimizer states multiply the model weight by four in mixed precision. A 125M parameter model can consume between 2 and 3 GB just in optimizer states, plus activations and gradients. On a 24 GB GPU you can train models of up to 300-400M parameters with a small batch and gradient checkpointing.
A common mistake I’ve seen in novice teams: assuming that a consumer RTX 4090 allows training a 1B parameter model without memory-saving techniques. The reality is that training stops due to OOM (out of memory) after a few steps, and debugging time exceeds training time.
Step 2: building and cleaning the corpus
The dataset is the most valuable asset and the most neglected. For a domain model (legal, medical, technical, customer service), you need between 1B and 10B tokens of quality text. If you start from scratch, the cleaning pipeline must include:
- Document-level and paragraph-level deduplication with MinHash or SimHash. Without this, the model memorizes and overfits to repeated phrases.
- Language filtering with a lightweight classifier (fastText works well) to discard content that isn’t Spanish if that’s your domain.
- Boilerplate removal (navigation menus, cookie notices, footers) that poison the training signal.
- Heuristic quality control: ratio of alphabetic characters, average sentence length, presence of n-gram repetitions.
According to industry studies on public datasets like C4 and The Pile, between 40% and 60% of raw text is discarded after cleaning. Plan storage accordingly: a corpus of 10B tokens in plain text takes up approximately 40 GB.
Step 3: domain-specific tokenizer
Don’t reuse the GPT-2 or Llama tokenizer if your domain has its own vocabulary. A BPE tokenizer trained on your corpus reduces the number of tokens per document and speeds up training. The setup with Hugging Face’s tokenizers library is straightforward:
from tokenizers import Tokenizer, models, trainers, pre_tokenizers
tokenizer = Tokenizer(models.BPE(unk_token=»[UNK]»))
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False)
trainer = trainers.BpeTrainer(vocab_size=32000, special_tokens=[«[PAD]»,»[UNK]»,»[BOS]»,»[EOS]»])
tokenizer.train_from_iterator(corpus_iter, trainer=trainer)
tokenizer.save(«tokenizer_dominio.json»)
A vocabulary between 32,000 and 50,000 tokens covers general Spanish well. For technical domains with many acronyms, consider 48,000. Check the compression rate: if your tokenizer generates more than 1.4 tokens per word on average, you probably need more vocabulary or a more representative training corpus.
Step 4: model architecture and initial configuration
For a model from scratch, a GPT-style decoder-only architecture is the standard. The configuration must be coherent: hidden dimension (d_model), number of layers, attention heads, and feed-forward dimension. A rule that works: feed-forward = 4 × d_model, and number of heads such that the dimension per head is 64 or 128.
Example configuration for a 125M model
{
«vocab_size»: 32000,
«hidden_size»: 768,
«num_hidden_layers»: 12,
«num_attention_heads»: 12,
«intermediate_size»: 3072,
«max_position_embeddings»: 1024,
«hidden_act»: «gelu»,
«dropout»: 0.1
}
With this configuration, the model has approximately 125M parameters. If your GPU is 16 GB, reduce the per-device batch to 8 and use gradient accumulation of 4 for an effective batch of 32. The typical initial learning rate is 3e-4 with a cosine scheduler and warmup of 2,000 steps.
Step 5: training loop with PyTorch and Accelerate
The minimum viable setup uses transformers with Trainer or a manual loop with accelerate. For full control, the manual loop is preferable. The critical points:
- Mixed precision (fp16 or bf16) with torch.cuda.amp. On Ampere or newer GPUs, bf16 is more stable.
- Gradient checkpointing enabled if the model doesn’t fit in memory.
- Saving checkpoints every 500 steps, not every epoch. In large models, one epoch may be unfeasible.
- Perplexity evaluation on a validation set of 5% of the corpus.
A detail you only learn through practice: the loss may seem stable but validation perplexity may diverge if the validation corpus has a different distribution. Make sure the validation set is randomly extracted from the same clean corpus, not from a different source.
Step 6: evaluation and warning signs
Perplexity isn’t enough. You need to evaluate the model on concrete tasks: text generation, classification, or question answering. For a small model, use lightweight benchmarks like a subset of the reasoning task or manual generation tests with prompts from your domain.
Signs that something is wrong:
- Loss that drops but incoherent generation: probably overfitting due to duplicates in the corpus.
- Validation perplexity that rises after a minimum: learning rate too high or corpus too small for the model size.
- Repetitions in generation: lack of diversity in the corpus or a poorly trained tokenizer.
Step 7: fine-tuning and deployment
A base model trained from scratch doesn’t follow instructions. For practical use, you need an instruction phase or supervised fine-tuning with question-answer pairs from your domain. This phase is lighter: between 10,000 and 100,000 quality examples are usually enough for a small model.
Deployment with vLLM or llama.cpp allows serving the model with 8 or 4-bit quantization, reducing VRAM requirements. A 125M model quantized to 4 bits takes up less than 100 MB and can run on CPU, although latency will be higher.
Limitations you must accept
Training from scratch doesn’t guarantee quality. A 125M model trained with 5B tokens in Spanish will have limited factual knowledge and will hallucinate frequently. It doesn’t compete with larger pre-trained models. If your goal is a general conversational assistant, fine-tuning an existing model is more cost- and time-efficient. Training from scratch makes sense when you need full control over the vocabulary, the license, or the architecture, or when you’re researching training dynamics.
According to the official PyTorch documentation and the Hugging Face guides, the cost of training 125M parameters with 5B tokens on a single 24 GB GPU can range from several days to a week, depending on throughput. There are no shortcuts: the data pipeline consumes as much time as training.
If you decide to move forward, start with a 500M token corpus and a 50M parameter model. Validate the entire pipeline before scaling. Experience shows that 80% of errors appear in the first 24 hours of training and are easier to fix with a small model.