The silent shift: from «asking the model» to «orchestrating agents over the data»
During 2024 and much of 2025, the conversation about language models applied to data analysis revolved around chat: paste a CSV, ask for a chart, get a table. That pattern is running out of steam. What dominates recent developments is something else: models that don’t respond with text, but instead execute code, query catalogs, validate assumptions, and return auditable results.
The practical difference is enormous. A chat that «interprets» a CSV hallucinates rows and averages with a worrying ease. An agent that executes SQL or Python against a real warehouse can’t invent a figure without the database engine protesting. That is the crux of the change: the model stops being the analyst and becomes the planner that writes queries and verifies results.
Mature text-to-SQL: where it works and where it still breaks
Generating SQL queries from natural language is the most widespread application, and also the most overrated in demos. According to official documentation from the main analytical database providers, public text-to-SQL benchmarks have improved steadily, but those numbers are obtained on clean schemas, with self-explanatory column names and no semantic ambiguity.
In a real warehouse, things change. A recurring case: the table is called fact_sales_v3, there’s a fact_sales_v2 that is no longer updated but is still alive, and there’s an amount column that is actually in cents in the rows prior to a specific migration. No model guesses that. It’s solved by a data catalog with descriptions, lineage, and documented business rules.
What is working
- Exploration queries over a bounded domain (a department, a closed set of tables).
- Translation of frequently asked questions into parameterized queries previously reviewed by a human.
- Generation of SQL drafts that an analyst corrects in seconds instead of writing them from scratch.
What doesn’t work (and it’s worth admitting it)
- Questions that require knowing unwritten business definitions («active customers» can mean five different things).
- Schemas with hundreds of tables without descriptions: the error rate skyrockets.
- Calculations where a poorly chosen JOIN duplicates rows and nobody detects it until the report reaches management.
Agents with code execution: the pattern that is taking hold
The approach that has grown the most consists of giving the model an isolated code interpreter. The model writes pandas, DuckDB, or R, executes it, reads the output, and, if there’s an error, iterates. This has a decisive advantage for analysis: the result is reproducible. You can save the script, version it, and run it again the following month.
Here’s a detail you only learn by using these tools daily: most failures don’t come from statistical reasoning, but from type and null value management. Dates in mixed formats, categories with trailing spaces, amounts with decimal commas inherited from an old export. A powerful model writes an impeccable groupby and then groups incorrectly because «Madrid» and «Madrid » are two different values. The solution isn’t a better model: it’s a prior cleaning layer and explicit validations.
Integration with the stack: MCP and semantic catalogs
The most relevant architectural development of recent months is the standardization of connectors. The open protocol promoted to connect assistants with data sources (known as MCP) is enabling the same model to query a warehouse, a CRM, and a spreadsheet without each integration being a custom build.
This has a consequence that organizations are beginning to accept: the value is no longer in the model, it’s in the semantic layer. Defining metrics once (net revenue, churn, margin) and exposing them as tools the model can invoke is what separates a flashy pilot from a system that survives past the third month.
A pattern that appears in teams that have already passed the testing phase: fewer long prompts, more registered functions with a name and precise description. The model chooses among ten well-documented tools instead of improvising SQL over fifty tables.
Evaluation: the problem almost nobody solves well
Measuring whether an LLM analyzes well is harder than making it work. Public benchmarks don’t reflect your data. The practice that is consolidating among mature teams is to build a set of questions with manually verified answers over your own warehouse, and run it on every model or prompt change.
Without that evaluation set, any migration to a new model is a blind bet. And here’s a nuance worth keeping in mind: improving on a generic benchmark doesn’t guarantee improving in your domain. We’ve seen models that lower performance on aggregation tasks with date hierarchies because they change their reasoning style, not because they’re worse.
Cost and latency: the factor that decides the architecture
An interactive analysis tolerates two seconds of waiting; a scheduled report that runs over thousands of rows every night doesn’t tolerate the cost of a large model for every query. Hence the trend toward hybrid architectures:
- Small models or deterministic rules for frequent and well-typified questions.
- Large models only for open exploration and error debugging.
- Result caching for repeated queries, which in practice are the majority.
Token spending becomes a real operational cost line when the system is used by twenty people daily. Designing without taking this into account is the fast track to the project being cancelled.
Governance and privacy: the conversation that arrives late but arrives
Sending customer data to an external provider requires reviewing contracts, retention, and data residency. Private deployment options and models run on your own infrastructure have matured enough to be viable in regulated sectors, albeit with a maintenance cost that many companies underestimate.
A common mistake: assuming that «the provider doesn’t train on my data» solves the problem. It doesn’t solve it if the data leaves your perimeter. The real decision is architectural, not contractual.
What to watch in the coming months
- Result verification: mechanisms that cross-check the model’s output against a source of truth before displaying it.
- Automatic lineage: knowing which table and which transformation each figure the system presents comes from.
- Hybrid interfaces: chat doesn’t disappear, but it coexists with dashboards where the LLM only writes the explanation of the chart.
- Continuous evaluation: your own suites that run on every model update.
The honest conclusion is that LLMs already provide clear value in exploration, data documentation, and assisted query generation, and they remain fragile in any task where a wrong figure has consequences. Designing with that asymmetry in mind, putting the data and validation at the center and the model at the periphery, is today the difference between an experiment and a system in production.