Skip to main content

Reimagining enterprise data engineeringIn the world of AI

The data supply chain was built for a human customer. AI changes the line and the customer at the same time.

Published

November 16, 2026

Authors

Arun Kumar, Shafeer Badharudeen

Source systemsPipelinesModelsDashboardsThe agent

Introduction

The data supply chain, and why it must be rebuilt

Every enterprise runs a data supply chain.

Source systems are the raw material, pipelines are the production line, models and dashboards are the finished goods, and a skilled person is the customer.

It is still built the way manufacturing was built before industrialization: by hand, by craftsmen, one order at a time. A request arrives, one team assembles a pipeline, another assembles a report, a third documents what was made. Lead times run in weeks. Cost tracks headcount. Every unit is bespoke, and quality depends on who happened to build it.

The chain also stops short of the value. We deliver to the loading dock of human judgment and go no further. Interpreting the numbers, knowing which can be trusted, deciding what to do, owning the outcome, all of it sits with the customer, not the chain.

AI changes both facts at once. The line can be industrialized: agents perform the work of the craftsmen, continuously, at a cost that no longer tracks headcount. And the customer changes: an agent that consumes without pause, decides, and acts inside the business process, carrying none of the human’s unwritten knowledge and none of the human’s caution.

Industrializing the existing line only produces the wrong goods faster. The chain has to be re-engineered for the new customer at the same time it is automated. Six dimensions define that rebuild.

The chain today

Figure 1

Raw material

Source systems

Production line

Pipelines

Finished goods

Models and dashboards

The customer

A skilled person

The chain ends here, at the dock of human judgment.

Overview

Six dimensions of the rebuild

The chain must be re-engineered for the new customer.

Each dimension below changes what data engineering produces, how it is controlled and what it is accountable for.

What the rebuild produces

Not faster craft goods, but a governed and verifiable foundation on which the enterprise decides and acts.

Industrialize the line

Serve the agent

Govern every action

Run inside the process

Production

Industrialize the line, then retire the products it was built for.

Agents can now perform the construction work: pipelines, medallion models, lineage, business glossaries, schema-to-ontology transformation, outcome semantics, and other outputs once divided among data engineers, modelers, analysts, and BI craftsmen. This turns a bespoke, headcount-bound function into a manufactured one.

But the products coming off that line were designed for humans. Aggregate tables and fixed dashboards exist because people cannot calculate while reading, and they cannot wait. An agent calculates. It does not read.

Automating existing roles only industrializes a chain still built for a human customer. The first act of reimagination is deciding which products should survive.

The line, before and after

Figure 2

Today

Rebuilt

Work done by

Craftsmen, one order at a time

Agents, continuously

Cost follows

Headcount

Machine capacity, not headcount

Products

Aggregate tables and fixed dashboards

Only those an agent still needs

Representation

The customer is now an agent, and it needs a different product.

A person read a summary once a day and filled the gaps from experience. They knew that regions rolled up differently after a reorganization, that one plant’s yield was reported two days late, and that a forecast was not a fact. None of this was written down because it did not need to be. An agent queries continuously, has no memory of the business, and acts only on what it is given. Three things about the finished goods must therefore change.

01

What is in them

Metrics and KPIs are summaries designed for a customer who could hold only a few numbers at once. An agent needs the underlying state of the business at full granularity, with the meaning of every element defined in an ontology rather than assumed.

02

What they are made of

A human analyst could ask a colleague where a number came from. An agent cannot. Every value must carry its own context: source, transformation rule, assumptions, and whether it is observed, estimated, forecast, stale, or disputed. Data that does not explain how it was produced cannot be acted on safely. Much of this input arrives through contracts, change orders, tickets, and policies, a second stream of raw material feeding the same plant.

03

How quickly they are delivered

Overnight batches suited a customer who reviewed results in a Monday meeting. An agent making decisions inside a live business process needs data at the pace of that process. Delivery cadence becomes a business constraint, not just a technical setting.

The customer has changed

Figure 3

Human customer

Agent customer

Consumes

A summary, once a day

Queries continuously

Context

Filled gaps from experience

Has no memory of the business

Acts on

What they read and chose

Only what it is given

Control

The chain no longer stops at delivery; it acts.

With a human customer, the platform’s job ended at delivery. The planner read the report and decided whether to expedite the order, so the deciding, and the restraint, lived outside the chain. An agent consumer does not stop at reading. It places the order.

The moment that happens, the platform is writing instructions into operational systems and needs controls no analytics estate has ever carried: a catalogue of actions each agent may take, authority scoped narrowly to each one, simulation of an action and its effect before committing, thresholds above which a person approves, and a way to reverse what has been done.

There is a second consequence, less obvious and more dangerous. When the consumer also acts, it changes the raw material. Agent decisions shape the transactions that become tomorrow’s data: reduce attention to a customer segment and it generates less activity, which then appears to justify the original decision. Holding out control groups and measuring the platform’s own effect on the business become operating requirements, not analytical refinements.

Controls an acting platform needs

Figure 4

Catalogue of permitted actions

Authority scoped to each one

Simulate before committing

Human approval above thresholds

Reverse what was done

Once the platform writes instructions into operational systems, restraint has to live inside the chain.

Assurance

No human inspects the output any more.

The final quality check in today’s chain is the customer. A finance manager sees a number that looks wrong and calls it out. That check disappears when the customer is an agent that accepts what it is given, processes it at machine scale, and acts immediately.

Inspection must move into the line. It must also change because the line itself has changed. A deterministic pipeline produces the same output on every run, so reconciliation and row counts may be enough. But agents that build pipelines and interpret meaning are not deterministic. The same request can produce different logic on different days. Correctness must therefore be demonstrated continuously: a standing set of business questions with known answers tested against the ontology, new agent behavior running alongside the existing system before taking over, and output quality tracked as an operating metric.

And because the customer now acts, every action needs a record that can be reconstructed later: which values were used, at what point in time, under which version of the definitions, what outcome was expected, and what actually happened. Lineage stops being catalog documentation. It becomes the evidence trail for decisions no person made.

The evidence trail for every action

Figure 5

Values used

Point in time

Definition version

Expected outcome

Actual outcome

Demand and economics

Machine consumers remove every natural limit.

A human customer was a natural throttle. A person could request only so many reports and read fewer still, and that scarcity is the only reason a request-and-fulfil operating model ever worked. Agents express intent continuously and draw on the same data thousands of times an hour, so both sides of the model break.

On the demand side, the request disappears. Nobody asks for a dataset and waits. An agent states a business intent, and the platform returns the outcome, what is driving it, how it has changed, and where it is heading. This works only if terms are settled in advance. What counts as an active customer or a booked order is often agreed informally in meetings, with the human customer quietly absorbing the ambiguity. When agents act on those terms, an unresolved definition becomes an operational failure. Definitions therefore need owners, versions, and an approval process.

On the cost side, the economics invert. A dashboard read a few dozen times a day and a data product queried thousands of times an hour do not justify the same build decisions, so what is pre-built versus computed on demand has to be re-decided, and machine consumption budgeted as an input cost. Success is then measured by business outcomes and their impact, not by the units of data product shipped.

What the economics invert

Figure 6

Request and fulfil

Intent and outcome

Demand

A dataset is requested and awaited

An intent is stated; the outcome returns

Definitions

Agreed informally in meetings

Owned, versioned and approved

Cost

What to pre-build is fixed once

Pre-built versus on demand, re-decided

Success

Units of data product shipped

Business outcomes and impact

Topology and retooling

The plant moves next to the business process.

Transactional systems, analytics and planning were separated because compute and memory were scarce, and the human customer stitched them back together by reading in one and typing into another. Take the human out and those seams have to close in the architecture instead. Data preparation moves toward the systems where transactions happen, becoming part of the business process rather than a parallel activity running behind it. The same foundation carries the process twin, a working model of the process that lets the enterprise test a decision in simulation before committing it.

None of this is built on an empty site. Decades of pipeline code and customized ERP hold business rules that exist nowhere else in writing, and recovering them (agents reading legacy logic to reconstruct the intent behind it) is part of the build, not a migration task to schedule afterwards.

Closing the seams

Figure 7

Separated, stitched by a human

Transactions

Analytics

Planning

One foundation

Data preparation inside the business process

Process twin

Conclusion

The argument, in one line

Industrialization has never been about making craft goods faster. It has changed what could be made, who could buy it, and what a company was.

The same is true here. Everything the human customer carried (meaning, judgment, action, accountability, restraint) moves inside the chain itself. Data engineering stops supplying material for people to interpret and becomes the governed, verifiable foundation on which the enterprise decides and acts.

The choice

Automate the old line

and you get the same reports, cheaper.

Rebuild the supply chain

and you get an enterprise that runs itself.

Rebuild the chain for the new customer

Share the paper with your data and platform leads, or talk to us about re-engineering your data supply chain for agents.