Reimagining enterprise data engineeringIn the world of AI
The data supply chain was built for a human customer. AI changes the line and the customer at the same time.
Introduction
The data supply chain, and why it must be rebuilt
Every enterprise runs a data supply chain.
Source systems are the raw material, pipelines are the production line, models and dashboards are the finished goods, and a skilled person is the customer.
It is still built the way manufacturing was built before industrialization: by hand, by craftsmen, one order at a time. A request arrives, one team assembles a pipeline, another assembles a report, a third documents what was made. Lead times run in weeks. Cost tracks headcount. Every unit is bespoke, and quality depends on who happened to build it.
The chain also stops short of the value. We deliver to the loading dock of human judgment and go no further. Interpreting the numbers, knowing which can be trusted, deciding what to do, owning the outcome, all of it sits with the customer, not the chain.
AI changes both facts at once. The line can be industrialized: agents perform the work of the craftsmen, continuously, at a cost that no longer tracks headcount. And the customer changes: an agent that consumes without pause, decides, and acts inside the business process, carrying none of the human’s unwritten knowledge and none of the human’s caution.
Industrializing the existing line only produces the wrong goods faster. The chain has to be re-engineered for the new customer at the same time it is automated. Six dimensions define that rebuild.
The chain today
Figure 1
Raw material
Source systems
Production line
Pipelines
Finished goods
Models and dashboards
The customer
A skilled person
The chain ends here, at the dock of human judgment.
Overview
Six dimensions of the rebuild
The chain must be re-engineered for the new customer.
Each dimension below changes what data engineering produces, how it is controlled and what it is accountable for.
Production
Industrialize the line, then retire the products it was built for.
Representation
The customer is now an agent, and it needs a different product.
Control
The chain no longer stops at delivery; it acts.
Assurance
No human inspects the output any more.
Demand and economics
Machine consumers remove every natural limit.
Topology and retooling
The plant moves next to the business process.
What the rebuild produces
Not faster craft goods, but a governed and verifiable foundation on which the enterprise decides and acts.
Industrialize the line
Serve the agent
Govern every action
Run inside the process
Production
Industrialize the line, then retire the products it was built for.
Agents can now perform the construction work: pipelines, medallion models, lineage, business glossaries, schema-to-ontology transformation, outcome semantics, and other outputs once divided among data engineers, modelers, analysts, and BI craftsmen. This turns a bespoke, headcount-bound function into a manufactured one.
But the products coming off that line were designed for humans. Aggregate tables and fixed dashboards exist because people cannot calculate while reading, and they cannot wait. An agent calculates. It does not read.
Automating existing roles only industrializes a chain still built for a human customer. The first act of reimagination is deciding which products should survive.
The line, before and after
Figure 2
Today
Rebuilt
Work done by
Craftsmen, one order at a time
Agents, continuously
Cost follows
Headcount
Machine capacity, not headcount
Products
Aggregate tables and fixed dashboards
Only those an agent still needs
Representation
The customer is now an agent, and it needs a different product.
A person read a summary once a day and filled the gaps from experience. They knew that regions rolled up differently after a reorganization, that one plant’s yield was reported two days late, and that a forecast was not a fact. None of this was written down because it did not need to be. An agent queries continuously, has no memory of the business, and acts only on what it is given. Three things about the finished goods must therefore change.
01
What is in them
Metrics and KPIs are summaries designed for a customer who could hold only a few numbers at once. An agent needs the underlying state of the business at full granularity, with the meaning of every element defined in an ontology rather than assumed.
02
What they are made of
A human analyst could ask a colleague where a number came from. An agent cannot. Every value must carry its own context: source, transformation rule, assumptions, and whether it is observed, estimated, forecast, stale, or disputed. Data that does not explain how it was produced cannot be acted on safely. Much of this input arrives through contracts, change orders, tickets, and policies, a second stream of raw material feeding the same plant.
03
How quickly they are delivered
Overnight batches suited a customer who reviewed results in a Monday meeting. An agent making decisions inside a live business process needs data at the pace of that process. Delivery cadence becomes a business constraint, not just a technical setting.
The customer has changed
Figure 3
Human customer
Agent customer
Consumes
A summary, once a day
Queries continuously
Context
Filled gaps from experience
Has no memory of the business
Acts on
What they read and chose
Only what it is given
Control
The chain no longer stops at delivery; it acts.
With a human customer, the platform’s job ended at delivery. The planner read the report and decided whether to expedite the order, so the deciding, and the restraint, lived outside the chain. An agent consumer does not stop at reading. It places the order.
The moment that happens, the platform is writing instructions into operational systems and needs controls no analytics estate has ever carried: a catalogue of actions each agent may take, authority scoped narrowly to each one, simulation of an action and its effect before committing, thresholds above which a person approves, and a way to reverse what has been done.
There is a second consequence, less obvious and more dangerous. When the consumer also acts, it changes the raw material. Agent decisions shape the transactions that become tomorrow’s data: reduce attention to a customer segment and it generates less activity, which then appears to justify the original decision. Holding out control groups and measuring the platform’s own effect on the business become operating requirements, not analytical refinements.
Controls an acting platform needs
Figure 4
Once the platform writes instructions into operational systems, restraint has to live inside the chain.
Assurance
No human inspects the output any more.
The final quality check in today’s chain is the customer. A finance manager sees a number that looks wrong and calls it out. That check disappears when the customer is an agent that accepts what it is given, processes it at machine scale, and acts immediately.
Inspection must move into the line. It must also change because the line itself has changed. A deterministic pipeline produces the same output on every run, so reconciliation and row counts may be enough. But agents that build pipelines and interpret meaning are not deterministic. The same request can produce different logic on different days. Correctness must therefore be demonstrated continuously: a standing set of business questions with known answers tested against the ontology, new agent behavior running alongside the existing system before taking over, and output quality tracked as an operating metric.
And because the customer now acts, every action needs a record that can be reconstructed later: which values were used, at what point in time, under which version of the definitions, what outcome was expected, and what actually happened. Lineage stops being catalog documentation. It becomes the evidence trail for decisions no person made.
The evidence trail for every action
Figure 5
Values used
Point in time
Definition version
Expected outcome
Actual outcome
Demand and economics
Machine consumers remove every natural limit.
A human customer was a natural throttle. A person could request only so many reports and read fewer still, and that scarcity is the only reason a request-and-fulfil operating model ever worked. Agents express intent continuously and draw on the same data thousands of times an hour, so both sides of the model break.
On the demand side, the request disappears. Nobody asks for a dataset and waits. An agent states a business intent, and the platform returns the outcome, what is driving it, how it has changed, and where it is heading. This works only if terms are settled in advance. What counts as an active customer or a booked order is often agreed informally in meetings, with the human customer quietly absorbing the ambiguity. When agents act on those terms, an unresolved definition becomes an operational failure. Definitions therefore need owners, versions, and an approval process.
On the cost side, the economics invert. A dashboard read a few dozen times a day and a data product queried thousands of times an hour do not justify the same build decisions, so what is pre-built versus computed on demand has to be re-decided, and machine consumption budgeted as an input cost. Success is then measured by business outcomes and their impact, not by the units of data product shipped.
What the economics invert
Figure 6
Request and fulfil
Intent and outcome
Demand
A dataset is requested and awaited
An intent is stated; the outcome returns
Definitions
Agreed informally in meetings
Owned, versioned and approved
Cost
What to pre-build is fixed once
Pre-built versus on demand, re-decided
Success
Units of data product shipped
Business outcomes and impact
Topology and retooling
The plant moves next to the business process.
Transactional systems, analytics and planning were separated because compute and memory were scarce, and the human customer stitched them back together by reading in one and typing into another. Take the human out and those seams have to close in the architecture instead. Data preparation moves toward the systems where transactions happen, becoming part of the business process rather than a parallel activity running behind it. The same foundation carries the process twin, a working model of the process that lets the enterprise test a decision in simulation before committing it.
None of this is built on an empty site. Decades of pipeline code and customized ERP hold business rules that exist nowhere else in writing, and recovering them (agents reading legacy logic to reconstruct the intent behind it) is part of the build, not a migration task to schedule afterwards.
Closing the seams
Figure 7
Separated, stitched by a human
Transactions
Analytics
Planning
One foundation
Data preparation inside the business process
Conclusion
The argument, in one line
Industrialization has never been about making craft goods faster. It has changed what could be made, who could buy it, and what a company was.
The same is true here. Everything the human customer carried (meaning, judgment, action, accountability, restraint) moves inside the chain itself. Data engineering stops supplying material for people to interpret and becomes the governed, verifiable foundation on which the enterprise decides and acts.
The choice
Automate the old line
and you get the same reports, cheaper.
Rebuild the supply chain
and you get an enterprise that runs itself.
Rebuild the chain for the new customer
Share the paper with your data and platform leads, or talk to us about re-engineering your data supply chain for agents.