Choosing between a large language model (LLM) and a classic natural language processing (NLP) pipeline is not a matter of fashion, but of technical fit. This checklist is designed for you to use as a filter before committing budget, infrastructure, or an entire team to the wrong direction. Each point includes the nuance we usually learn the hard way: when the obvious answer fails.
How to use this checklist
It is not a yes-or-no questionnaire. Check each box with evidence (a test, a document, a measurement), not with intuition. If you cannot fill in a box with data, that box is precisely where your project will break in production.
1. Define the task with surgical precision
1.1 Is it a closed or open task?
Classifying 40,000 tickets into 12 fixed categories is a closed task. Summarizing heterogeneous conversations is an open task. Traditional models (logistic regression on TF-IDF, SVM, CRF for sequence labeling, topic models such as LDA) remain competitive on closed tasks with stable vocabulary and well-defined boundaries. LLMs shine when the output cannot be enumerated in advance.
1.2 Is the correct label unambiguous?
If two expert annotators do not agree when labeling the same example, an LLM will tend to produce plausible but inconsistent answers across runs, and a classic model will simply learn the noise. Before choosing technology, measure inter-annotator agreement. Without that metric, any subsequent comparison is smoke.
1.3 Do you need structured extraction or generation?
Extracting entities, relationships, and dates from contracts is an extraction problem. An LLM can do it with forced JSON format, but an entity recognition model trained on your own data is usually cheaper per document, faster, and more stable against format variations. Free generation, on the other hand, is almost exclusively the territory of LLMs.
2. Audit your data before auditing models
2.1 Volume and imbalance
With 300 labeled examples per class, a well-tuned classic model may be enough. With 30, neither will work well, and the LLM will need examples in the prompt (in-context learning) or fine-tuning, which is not free. Check for imbalance: in fraud or critical incident classification, the minority class is the one that matters and the one with the least data.
2.2 Domain and jargon
A general-purpose LLM knows standard Spanish, but not necessarily your company’s internal jargon, product codes, or industry abbreviations. Classic models do not either, but their training is cheaper and you can inject controlled vocabulary without retraining a model with billions of parameters.
2.3 Data sensitivity and residency
If the text contains personal data, clinical records, or information under contractual secrecy, sending each document to an external API requires a risk assessment that many organizations have not done. Classic models run on your own infrastructure with a smaller exposure surface. Self-hosted LLMs exist, but they completely change the cost and staffing calculation.
3. Calculate the real cost, not the cost per token
3.1 Cost per inference versus cost per maintenance
The price per token of an LLM seems low until you multiply by millions of documents and add retries, validations, and context tokens. A classic model has an almost negligible inference cost, but it carries a maintenance cost: periodic retraining, feature management, model drift (data drift). The right question is not which is cheaper today, but which will be cheaper in two years with your projected volume.
3.2 Latency and user experience
A linear classifier responds in milliseconds; a call to an LLM with long context can take seconds. If the task is on the critical path of an interface (autocomplete, real-time moderation, chat routing), that difference is decisive. I have seen teams migrate an intent classifier to an LLM and discover that response time multiplied by ten with no perceptible improvement in accuracy.
3.3 Evaluation cost
Evaluating an LLM is more expensive and more diffuse than evaluating a classifier: you need reference sets, sometimes subjective quality criteria, and human review. Budget that effort before you start; according to industry documentation, evaluation usually consumes more time than expected.
4. Check reproducibility and auditability
4.1 Do you need to explain every decision?
In regulated sectors (insurance, banking, healthcare, public administration) you often have to justify why a system rejected an application. A regression model with interpretable coefficients or a decision tree is relatively easy to explain. An LLM generates a justification in natural language that can sound convincing and not correspond to the actual reasoning. Apparent explainability is a risk, not an advantage.
4.2 Must the output be identical for the same input?
LLMs with stochastic sampling do not guarantee this. You can set temperature to zero, but that does not eliminate all sources of variation (model versions, provider changes, context). Classic models are deterministic by design. If your use case requires repeatable auditing, this weighs more than a few points of accuracy.
4.3 Versioning and vendor dependency
A hosted LLM can change behavior without notice when the provider updates the model. Your classic pipeline only changes when you change it. Document how you will pin versions and what you will do if the provider retires the one you use.
5. Assess the engineering effort and the team
5.1 Do you have someone to maintain prompts or someone to maintain features?
An LLM shifts the work from feature engineering toward prompt design, output validation, and regression testing. A classic model requires profiles more oriented toward data and statistics. Neither path is free; choose the one your team can sustain over time.
5.2 Unstructured outputs and validation
If the LLM returns free text, you will need validation layers, retries, and error handling. If it returns forced JSON, you will need strict schemas and tolerance for format failures. Count those pieces in the design; they are not a detail.
5.3 Regression testing
With classic models, a test suite with fixed metrics is usually enough. With LLMs, it is advisable to maintain a golden set of cases and compare responses between versions, accepting some variability. If you do not have that set, you have no way of knowing whether an update improved or worsened the system.
6. Review the limitations nobody mentions in demos
6.1 Hallucinations in extraction tasks
An LLM can invent a date that does not appear in the document if the format suggests it. In data extraction, this is a silent and dangerous failure. Classic sequence labeling models have their own errors, but they tend to be errors of omission or boundary, not inventions.
6.2 Degradation with long or noisy inputs
LLM performance drops when the text is very long, contains poorly converted tables, or mixes languages. In a real case of support email classification, the classic pipeline with specific cleaning outperformed the LLM on messages with nested quotes and signatures, because those parts confused the generative model. The lesson: prior cleaning matters in both worlds, and often more than the chosen model.
6.3 Languages and dialectal variants
A multilingual LLM can work acceptably in many languages, but a classic model trained on local data can outperform it on a specific variant. If your corpus is very specific, test before assuming the large model wins.
6.4 What does not work well with LLMs
Very high-precision tasks with legal consequences, decisions that must be identical and auditable, massive processing with minimal latency and a tight budget, and classification of very closed vocabulary with abundant labeled data. In those cases, the LLM adds cost and opacity without a clear advantage.
6.5 What does not work well with classic models
Coherent text generation, abstractive summarization, open dialogue, adaptation to new instructions without retraining, and understanding of pragmatic nuances (irony, implicit intent). Forcing a classic model to do this produces rigid and fragile results.
7. Design the experiment before deciding
7.1 Representative evaluation set
Gather real examples, including rare and ambiguous cases. An evaluation set biased toward easy cases favors the LLM because it impresses in general terms; difficult cases are the ones that reveal differences.
7.2 Metrics aligned with the business
Precision, recall, and F1 are the starting point, but ask which error costs more. In content moderation, a false negative can be serious; in lead classification, a false positive annoys the sales team. The metric must reflect that asymmetry.
7.3 Hybrid test
The dichotomy is false in many cases. A common pattern is to use a classic model to filter the 80% of easy cases and reserve the LLM for the ambiguous ones. Or use embeddings and semantic search (heirs of classic techniques) alongside an LLM to generate the final response. Document that architecture as one more option, not as a failure of the comparison.
8. Final decision checklist
- I have defined the task and I know whether it is closed or open.
- I have measured agreement between human annotators.
- I know the volume, the imbalance, and the quality of my data.
- I have calculated cost per inference and maintenance cost over two years.
- I have checked the admissible latency in my use case.
- I know whether I need explainability and strict reproducibility.
- I have a versioning and vendor dependency plan.
- My team can sustain the chosen approach (prompts or features).
- I have identified the dangerous silent failures in my domain.
- I have tested both approaches with a representative evaluation set.
- I have considered a hybrid architecture before discarding it.
If you have checked everything with evidence and not with intuition, the decision between LLM and classic NLP stops being a gamble and becomes a documented choice. And if you still cannot check some box, that is your next task, not the choice of model.