Why almost everything you were told about LLMs in production is only half the story
I have seen teams throw away six months of work by accepting two opposing ideas at face value: that an LLM worked like a REST endpoint and that launching it was enough, or that it was so unpredictable that it wasn’t worth taking out of the lab. Neither holds up to contact with real traffic. According to the official documentation of the main model providers, latency, cost, and quality vary non-linearly with context, temperature, and response length, and that breaks many of the promises that circulate in talks and posts.
This article reviews the myths that do the most damage when taking language models to production and contrasts them with what happens when the system serves real users: retries, malformed prompts, sensitive data, and load spikes. There are no guarantees of results: there are patterns that work in certain contexts and others that fail spectacularly.
Myth 1: «The perfect prompt eliminates the need for engineering»
The prompt matters, but it does not replace architecture. A team I collaborated with built a support assistant that responded accurately in manual tests. In production, the same prompt failed as soon as users started writing in all caps, with typos, or pasting screenshots transcribed by a faulty OCR. The rate of useful responses dropped without the model having changed.
What does work
- Input validation: normalize text, limit length, and detect language before calling the model.
- Output contracts: require JSON with a schema and validate it with a library; if it fails, retry with a bounded correction prompt.
- Deterministic fallback: for high-frequency queries, a template or rules can respond better and more cheaply than the model.
The reality is that the prompt is one piece of the system, not the system. Mature teams version prompts like code, evaluate them with case sets, and measure regressions before deploying.
Myth 2: «More context always improves the response»
A large context window is useful, but it is not free. According to provider documentation, cost and latency grow with input tokens, and in practice quality does not improve monotonically: inserting ten irrelevant documents dilutes the signal and increases the likelihood that the model cites tangential information.
A concrete failure example
In an internal search engine over technical documentation, going from 3 to 15 retrieved fragments improved coverage of correct answers for broad questions, but worsened answers to detail questions because the correct fragment got buried among other similar ones. The solution was not to expand the context, but to improve retrieval ranking and limit to the highest-scoring fragments, with a cutoff threshold.
Limitations worth acknowledging
There is no magic number of fragments. It depends on the domain, the quality of the index, and the type of question. Anyone who claims «always use top-k=5» is oversimplifying. You have to measure it with your own evaluation set.
Myth 3: «The model is deterministic if I set temperature to 0»
Temperature 0 reduces variability, but it does not eliminate it across all providers or all models. Moreover, apparent determinism hides another source of variation: silent changes to the hosted model. The same identifier can be updated and alter responses, something that the documentation of several providers explicitly warns about.
Practical consequences
- Pin specific versions when the provider allows it.
- Maintain a regression test suite that runs after every version change.
- Log the model version alongside each response so you can audit.
If your product depends on output being identical across runs, an LLM is not the right tool for that part. Deterministic logic is better there.
Myth 4: «It costs little because the token is cheap»
The cost per token seems low until you multiply it by users, retries, and context. A pattern I see frequently: a chatbot that includes the full conversation history in every call. With twenty-turn conversations, cost grows quadratically with the number of turns, not linearly.
What can be done
- Summarize older history and keep only the last relevant turns.
- Cache responses for repeated questions, if the domain allows it.
- Route simple queries to small models and reserve large ones for complex cases.
- Set spending limits per user or per session.
There is no universal savings formula. Any figure shared without context on traffic and context length is anecdotal. What is consistent: without observability of input and output tokens, cost spirals out of control.
Myth 5: «If it works in the demo, it’s ready for production»
The demo shows the happy path. Production is the unhappy path: empty inputs, prompt injection attempts, users asking for things out of scope, provider outages, and timeouts. According to industry studies on service reliability, failures of external dependencies are a common cause of incidents, and an LLM is just another external dependency.
Minimum checklist before exposing it to users
- Timeouts and retries with exponential backoff, with a limit on attempts.
- Circuit breaker to avoid hammering the provider when it fails.
- Clear degradation messages when the model is unavailable.
- Input and output filters for personal data and harmful content.
- Trace logging with prompt, response, latency, and cost, with defined retention.
None of this is glamorous, but it is what separates a prototype from a service that survives a Monday morning.
Myth 6: «Evaluation is solved with good automatic metrics»
Automatic metrics like BLEU or ROUGE have weak correlation with perceived usefulness in open-ended tasks. For a data extraction system with a fixed schema, an accuracy metric works. For a conversational assistant, it does not.
What works in practice
- A small but representative evaluation set, reviewed by people from the domain.
- Pairwise evaluation or rubrics for subjective tasks.
- Periodic sampling of real responses to detect drift.
I acknowledge a limitation: building and maintaining that set takes time and does not scale as well as an automatic script. But without it, any claim about the system’s quality is an opinion.
Myth 7: «Privacy is fixed with a legal notice»
Sending user data to an external provider has implications that a notice does not resolve. You need to know what is sent, how much is retained, and in what jurisdiction it is processed. Some teams discover late that they were including emails and phone numbers in prompts because the source form collected them without filtering.
Concrete measures
- Pseudonymize identifiers before building the prompt.
- Define which fields never leave the system.
- Review the provider’s retention terms and, if necessary, use deployments that do not store data.
This is not legal advice, and you should consult with whoever is appropriate. But technically, filtering before sending is cheaper than remediating afterward.
Myth 8: «An LLM replaces the support or content team»
It can reduce load on repetitive tasks, but it introduces a supervision cost. A model that responds confidently on topics it does not master generates more work than it saves. In customer service projects, human review of low-confidence responses remains necessary to avoid costly errors.
The reality is more boring: the LLM is a tool within a workflow, with humans at the risk points. The cases where it works without supervision are those with a closed schema and low consequences of error, such as classifying tickets or summarizing internal texts.
What you should accept before starting
- You will need observability from day one, not afterward.
- Real cost is understood when there is traffic, not in the lab.
- Quality is measured with your own data, not with public benchmarks.
- There will be changes at the provider that affect your product without you touching anything.
- Some tasks should not be solved with an LLM: exact calculations, strict regulatory compliance, or decisions with irreversible consequences.
Implementing LLMs in production is less magic and more engineering discipline: versioning, evaluation, limits, and controlled degradation. Myths sell shortcuts; reality demands measuring before promising.