Deploy LLMs in Your Company: Step-by-Step Configuration

Deploying a language model in production is not about downloading a ZIP and running a command. I have seen teams spend months on a proof of concept that worked on an engineer’s laptop and crashed the first Monday with twenty concurrent users. This guide describes the real configuration process, with the decisions that matter and the points where almost everyone stumbles.

Before touching anything: define the use case and the success criteria

The technical configuration depends on the work the model will do. An internal assistant that answers questions about corporate documentation (retrieval-augmented generation over PDFs) is not the same as a support ticket classifier (a bounded task where a small, well-tuned model beats a large one with a generic prompt).

Write a concrete sentence before choosing a provider: «the salesperson types a question about pricing and receives an answer with the link to the catalog section in under three seconds.» If you can’t measure that, you can’t configure anything with sound judgment.

A common mistake: teams that start by asking for «an internal ChatGPT» without scoping the domain or the sources. The result is a chat that hallucinates about the vacation policy because no one gave it access to the right document. Scope the sources before the model.

Decision 1: managed API, self-hosted open model, or hybrid

Three real routes, with different costs and friction:

• Cloud provider API: you’re up and running in hours. Cost is variable per token and latency depends on the network. Suitable for almost everything, except when there is data that cannot leave the perimeter.
• Open model on your infrastructure: full control of the data, fixed GPU cost. It requires knowing how to operate inference: quantization, video memory, request queues. A 7-8 billion parameter model in 4-bit quantization runs on a 24 GB GPU; a 70B one needs several or aggressive quantizations that degrade quality.
• Hybrid: a small local model for classifying and routing, a large model via API only for complex queries. It reduces cost and data exposure.

According to the official documentation of the main inference frameworks (vLLM, TGI, Ollama), production performance depends more on batching and KV cache management than on the chosen model. A poorly configured server with an excellent model delivers a worse experience than a well-tuned one with a mid-range model.

The factor no one budgets for: the GPU

If you opt for self-hosting, calculate memory before buying. Rule of thumb: parameters in billions × bytes per parameter depending on quantization (≈0.5 bytes at 4 bits, ≈1 byte at 8 bits, ≈2 bytes at 16 bits) plus the space for the context. A 7B in FP16 takes about 14 GB just for the weights; with a long context you run out of VRAM sooner than you think.

Decision 2: how you give it the company’s knowledge

This is where 80% of the real work lies. Two approaches:

Retrieval-augmented generation (RAG)

You chunk documents, generate embeddings, store them in a vector index, and on each query you retrieve the relevant fragments to inject into the prompt. Configuration steps:

1. Ingestion with version control. Each document carries metadata: source, date, department, permissions. Without permissions in the metadata, an intern can read board minutes.
2. Structure-aware chunking. Fragments of 300-800 tokens with overlap. Chunking by characters breaks tables and lists; chunk by structure (headings, paragraphs).
3. Embeddings and reindexing. Choose a multilingual embedding model if there are documents in several languages. Plan to reindex when the corpus changes; an outdated index answers with data from two years ago without warning.
4. Hybrid retrieval. Combine semantic search with lexical search (BM25). Product codes and proper names fail in purely vector search.
5. Reranking and threshold. A reranker discards irrelevant fragments. Define a threshold: if similarity is low, the assistant must say «I don’t know» instead of making things up.

Fine-tuning

Useful for style, output format, and repetitive tasks with many labeled examples. It is not the tool for injecting changing knowledge: a model fine-tuned on January’s catalog will keep answering with January’s. For data that changes, RAG.

Configuring the system prompt and safeguards

The system prompt is configuration, not creative writing. Define: role, tone, limits, output format, and what to do when in doubt. Be explicit about the format if you’re going to parse the response (JSON with a fixed schema, for example) and validate the output with a parser before showing it to the user.

Minimum safeguards you must configure:

• Input and output filtering for personal data according to internal policy.
• Context length and output token limits (avoid runaway costs).
• Conversation logging with defined retention and audited access.
• Rate limiting per user and per IP.
• A «report incorrect answer» button that feeds a review queue.

A detail you discover in production: users paste entire conversations from other chats into the prompt. Without a length limit, the cost per session multiplies and the model gets distracted by irrelevant context.

Integration with company systems

The assistant rarely lives in isolation. Typical configuration:

1. Orchestration layer (your own backend or an agent framework) that receives the request, decides whether to query the index, call a tool, or answer directly.
2. Connectors to internal systems via API with read-only service credentials and minimal permissions. Never give the model write credentials on the ERP in the first version.
3. Authentication and permission inheritance. The assistant must answer only with what the user can see. This is implemented in the retrieval layer, not in the prompt: asking the model to «not reveal confidential data» is not an access control.
4. Observability. Traces per request: input and output tokens, latency, retrieved fragments, estimated cost. Without this you can’t optimize or justify the spend.

Test with real data before opening the door

Build an evaluation set with real user questions and expected answers. Measure: answer accuracy, rate of correct «I don’t know» responses, 95th percentile latency, and average cost per query. According to industry studies, a significant share of failures in corporate assistants comes from retrieval, not generation: the model answers well to the wrong fragment.

Also test the edge cases: ambiguous questions, out-of-domain requests, prompt manipulation attempts. And review behavior with poorly digitized scanned documents: dirty OCR generates answers with corrupted figures.

Gradual deployment and maintenance

Start with a pilot group of ten or twenty people from a single department. Gather feedback for two weeks, adjust the prompt and the corpus, and only then expand. Have a rollback plan: the ability to disable the assistant without breaking the team’s workflow.

Essential ongoing maintenance: reindex when documents change, review weekly costs, monitor latency, and update the model when a better version appears only after passing the evaluation. Changing models without evaluating is the fastest way to degrade a system that was working.

What this approach does not solve

An LLM configured this way does not replace a rules system in processes where accuracy is mandatory (payroll calculation, regulatory compliance with figures). Nor is it reliable for complex mathematical reasoning without external tools. And it does not eliminate the need for human review in sensitive external communications.

The step-by-step configuration that works is the one that starts scoped, measures, and expands only what demonstrates value. The rest is spending on tokens.