AI and ML in Data Management: What It Can and Cannot Do

AI in data management: what a model decides, and what it cannot

Machine learning finds patterns; data governance decides meaning

A machine learning model is trained on examples and learns what usually goes together. Given a new record, it produces a score: how much this looks like the things it has seen before. That is enormously useful on data management work, because most data problems are volume problems. Nobody can read four million supplier records looking for duplicates, but a model can, and it will surface the two thousand pairs worth a second look.

What it produces is a score rather than a verdict, and "these two records are ninety-four per cent similar" is a measurement. "These two records are the same supplier and this is the version we keep" is a decision about your business, and that decision belongs to the person who owns supplier data.

Every well-run AI data management deployment keeps those two things separate. The model measures and a named owner decides. When they are collapsed into one — when the score becomes the decision — that is where these projects go wrong, and the section on failure modes below is mostly about what that looks like.

Why the line between AI and data governance falls where it does

The line is not arbitrary and it is not about how advanced the technology gets, because it falls where it does for a structural reason.

Pattern tasks have a right answer that exists in the data, since whether this transaction resembles the last ten thousand is a fact about the data, and a model can get at it.

Decisions do not have an answer in the data at all. Whether "active customer" means eighteen months or twelve is a choice your business makes, and there is nothing in any dataset that settles it. A model trained on your history will faithfully reproduce whatever convention it was handed, including the one nobody agreed to.

So the line sits between measurement and meaning, and it stays there no matter how good the models get. The question is never whether the model is clever enough, but whether the thing you want is a pattern or a choice.

Four data management tasks where machine learning earns its place

Four, and they have something in common: each is a pattern problem, on high volume, where being occasionally wrong is survivable because a person checks the uncertain cases.

Each is described below in the same order — what it does, what it needs from your data, how it fails, and who confirms.

Anomaly detection for data quality monitoring

What it does. Learns what normal looks like in a field or a feed, then flags what does not fit. A revenue figure ten times the usual. A feed that arrived with a third of its usual row count. A postcode in a country you do not trade in. A date in 1900.

This is the most reliable use of machine learning in data management, because normal is genuinely a pattern and deviation from it is genuinely detectable.

What it needs from your data. History, mostly — enough of it to know what normal is, including the seasonal shapes. A model trained on three months of data will flag every December as an anomaly.

How it fails. Alert volume. Point it at an estate nobody has cleaned and it finds four thousand anomalies in week one, which is true and useless, so people stop reading the alerts by week three, and the tool gets switched off while everybody agrees it was a good idea.

Who confirms. A data steward reviews the flags, and the threshold is tuned down until the queue is a size somebody can actually work through. Start narrow — one feed, the fields that matter — rather than across the estate.

Entity resolution and duplicate detection in master data management

What it does. Decides how likely it is that two records describe the same real thing. The same supplier spelled four ways, or the same customer with two addresses and a changed surname. This is the oldest use of machine learning on data, and matching models are genuinely good at it now, including across languages and transliterations.

What it needs from your data. Labelled pairs — real examples of "these two are the same" and "these two are different" from your own records. A few thousand, produced by people who know the domain, and this is the work that no vendor demonstration includes.

How it fails. In both directions, and they are not symmetrical. Too cautious, and the duplicates stay; too aggressive, and it merges two genuinely different customers into one record, which is very hard to unpick afterwards, because the original two are now one and the history is mixed.

Who confirms. Above the threshold, a person reviews before the merge, always. That is not a temporary caution while the model beds in; a merge is irreversible in practice, and irreversible actions keep a person in the loop permanently. The survivorship rule — which field wins from which source — is written by the business first, and the model never guesses it. A data migration is where those rules get tested hardest, because a move surfaces every duplicate at once.

Data classification and metadata tagging at scale

What it does. Reads a field, a column or a document and says what kind of thing it holds. This column looks like personal data. This document contains bank details. This free-text field is being used for delivery instructions rather than notes.

For an estate nobody has catalogued, this is the fastest route to knowing what you have, and it is the task where AI in data management most clearly saves months rather than hours.

What it needs from your data. A classification scheme first, agreed by the business — three or four levels such as public, internal, confidential, regulated. The model applies a scheme; it cannot invent one, and a scheme invented by a tool is a scheme nobody agreed to.

How it fails. Quietly, on the things that matter most. A free-text field containing occasional card numbers gets classified on its majority content and marked internal, nothing looks wrong, and the exposure is discovered by an audit.

Who confirms. Anything classified as regulated or confidential is reviewed by a person before controls are applied, and the sensitive categories are sampled every month rather than signed off once. This is where classification meets data privacy and security, and where a wrong label has consequences beyond tidiness.

Schema mapping and data lineage inference

What it does. Works out how things connect. That this column in one system is the same field as that column in another. That this report is built from those three tables. That this field is populated from a feed nobody documented.

On a large estate, inferred data lineage is often the only lineage you will get, because the documentation was never written and the people who knew have left.

What it needs from your data. Access to the systems and their logs — queries, jobs, pipeline definitions. It infers from what actually runs rather than from what was documented, which is the reason it is worth having.

How it fails. Plausible and wrong, because two columns called cust_id in different systems may hold entirely different identifiers, and a model that matches on name and shape will link them confidently. The resulting lineage diagram looks authoritative and is fiction in one corner.

Who confirms. Treat inferred lineage as a draft that a person verifies for the flows that matter — regulatory reporting, board metrics, anything on a retirement list. Nobody verifies a whole estate, and nobody needs to.

The suggest-and-confirm pattern behind every AI data management tool

Read those four back and the same shape appears in all of them. The model produces a suggestion with a confidence score, and a threshold decides what happens next: above it the system acts, and below it a person looks.

That is the pattern, and three things make it work.

The threshold is a business decision. It is set by weighing the cost of a wrong action against the cost of a person's time, and it is different for every task. Merging customer records deserves a high bar and tagging a document does not, so nobody in IT should be choosing that number alone.

The queue below the threshold needs an owner. An unowned review queue is where these projects die — the same failure as an unowned exception queue in any automated process.

Confirmations are training data. Every time a person accepts or rejects a suggestion, that is a labelled example, so capture them. Teams that do have a retraining set after six months; teams that do not are back where they started when the model drifts.

What AI and machine learning cannot do in data management

Three things, and no amount of model improvement changes any of them, because none of the three is a pattern problem.

A model cannot define what a data field means

What counts as an active customer. When revenue is recognised. Whether a cancelled-then-reinstated account is new or continuing. What "region" means when sales and finance draw the map differently.

These are choices your organisation makes and writes down. A model trained on your history will learn the convention that happens to be in the data, including one an engineer picked in 2021 because a pipeline needed a value and nobody was available to ask.

That is worth sitting with. A model can make an undecided question look settled, because it produces a consistent answer at scale, and consistency is not agreement.

A model cannot write the survivorship rule for your golden record

When two records merge, something has to decide which values survive. The finance system's legal name or the CRM's trading name, the most recently entered address or the most recently verified one.

A matching model tells you the records are the same. It has no basis for choosing which version is authoritative, because that depends on which of your systems is trusted for which field — a fact about your organisation, not about the data.

Write the survivorship rule in plain language before any matching runs, because without it the tool's defaults become your policy, and nobody in the business ever agreed to them.

A model cannot own a data governance decision

Somebody has to be accountable when the merge was wrong, when the classification let something through, when the threshold was set too high. "The model decided" is not an answer an auditor accepts, and it is not an answer the business accepts either.

Accountability does not transfer to software. It stays with the data owner, and the practical consequence is that every AI data management deployment needs a named owner per domain before it starts, not after. That is the same requirement as any other governance work, and it is covered properly in our note on data management and governance.

And no AI data management tool fixes a process nobody owns

This is the one that sinks projects, and it has nothing to do with the model.

If four spellings of the same supplier exist because nobody was ever responsible for the supplier record, a matching model will find all four and surface them for a decision — and there is still nobody to make it.

The pairs sit in a queue. Three months later the queue has twelve thousand pairs and somebody suggests raising the threshold, which means doing less of the thing that was working.

The model did its job, and the gap it exposed was never a technical gap.

So the honest sequence is ownership first, then the model. A tool bought to avoid a governance conversation will hold that conversation for you, at scale, in the shape of a queue nobody empties.

What your data needs before machine learning works on it

This is the section vendor material skips, and it is where the actual project is.

Labelled training data, which your data stewards have to produce

A model learns from examples of the answer. For duplicate detection, that means pairs of records marked "same" and "different" — by somebody who knows that these two are the same supplier under an old trading name and those two are a parent and a subsidiary.

Nobody outside your business can produce those labels, not the vendor and not us. A few thousand pairs is a realistic starting point, and producing them takes a data steward several weeks.

That is the project: the model training is days, and the labelling is the months. Any plan that does not have this on it is not a plan, and any demonstration that skipped it was run on somebody else's labels.

One piece of good news: the labelling is not wasted if the tool changes. Labels belong to you and they transfer, which makes them one of the few durable assets in this whole area.

Volume, and enough of the awkward records

Models need enough examples to learn a pattern, and enough of the unusual ones to learn the edges. The common failure is a training set full of the easy cases, because the easy cases are what somebody had time to label. The model then performs beautifully in testing and badly in production, on exactly the records that needed help, so deliberately include the awkward ones: the overseas supplier with no tax identifier, the customer with no surname, the record created before the current system existed.

If your volumes are genuinely small — a few thousand records — machine learning is usually the wrong tool. Written rules and a validation at the point of entry will do more, faster, and you can explain them to anybody who asks.

An agreed definition of data quality

You cannot train a model on "good data" until somebody has written down what good means for each field that matters.

That means a standard: this field is mandatory, this one has a format, this combination is impossible, this value must exist in that reference list. Without it, there is nothing to detect deviation from, and "the data is bad" stays a feeling.

This is the dependency that catches people out. The governance work you were hoping to skip by buying a model is the input the model needs. It is not a prerequisite anybody invented to be difficult — it is arithmetic: a model measures distance from a standard, so the standard has to exist first. Where a lot of that data lives in high-volume platforms, our note on big data in business operations covers the surrounding shape.

How AI data management systems fail, and why the failure is quiet

A broken pipeline stops and somebody gets paged. A model that has started being wrong carries on producing output that looks exactly like the output it produced when it was right. That is the difference, and it is why these three failure modes need deliberate controls rather than monitoring.

Confidently wrong at scale: false positives and false negatives

Every model gets some wrong in both directions, and the two costs are rarely equal. A false positive flags something that was fine, which is usually cheap, because somebody looks and dismisses it.

A false negative misses something that was not fine. Usually expensive, and invisible, because nothing arrives to tell you about the thing that was not flagged.

The asymmetry is what sets the threshold, so on anomaly detection in data quality monitoring, misses cost more than false alarms and you run it sensitive and accept the queue. On record merging, a wrong merge costs far more than a duplicate that survives another month, so run it conservative. Getting this backwards is the single most common configuration error we see.

Model drift as your business data changes

A model trained on last year's data encodes last year's business. You launch a product line, enter a market, acquire a company, change a form — and the pattern it learned is now slightly wrong.

Drift does not announce itself: the model keeps scoring, the scores keep looking reasonable, and accuracy declines quietly over months, and nobody notices because there is nothing to notice.

The only reliable catch is a standing sample, where every month somebody reads a small set of the model's decisions — twenty is enough — and records whether they were right. That record is your drift detector, and it doubles as the retraining set.

Alert fatigue in data quality monitoring

The human failure mode, and the most common of the three.

Week one: four thousand anomalies. Week two: the team works through a few hundred. Week three: nobody opens the queue. Week six: the tool is switched off and the conclusion is that AI in data management does not work here.

What actually failed was scope. The tool was pointed at everything at once, in an estate that had never been cleaned, so it correctly reported that the estate was a mess — which everybody already knew and nobody could act on at that volume.

Start narrow: one feed, the fields that matter, a threshold tuned so the daily queue is a size one person can clear. Widen when that is working.

Three controls: confidence thresholds, sampled review, retraining

Confidence thresholds, set per task by weighing the cost of a wrong action against the cost of a person's time, owned by the business, and written down with the reasoning. Revisit them quarterly.

Sampled review, a standing monthly sample of decisions read by a person, and not a launch check but a permanent one, because drift is permanent.

Retraining, scheduled rather than triggered by somebody noticing a problem. The confirmations your reviewers produce are the training set, which is why capturing them from day one matters more than it sounds.

Those three catch all three failure modes, and none of them is technically difficult. They are operational habits, and the reason projects skip them is that nothing goes wrong on the day you skip them.

Frequently asked questions about AI and machine learning in data management

How is AI used in data management?

On four tasks, and all four are pattern problems on high volume: anomaly detection for data quality monitoring, entity resolution and duplicate detection in master data management, data classification and metadata tagging, and schema mapping and data lineage inference. In every case the model produces a suggestion with a confidence score, and a person confirms the uncertain ones. It does not decide what your data means.

Can AI improve data quality?

Yes, at finding problems, and only once you have defined what correct looks like. A model measures distance from a standard, so the standard has to exist first — which field is mandatory, what format it takes, which values are impossible. Without that written down, there is nothing for the model to detect deviation from, and "the data is bad" stays a feeling rather than a measurement.

What is the difference between AI and automation in data management?

A model decides what is likely; automation does the same thing every time. The model scores records and produces suggestions. The automation applies the ones above the threshold and routes the rest to a person. They are complementary, and a deployment with no automation layer loses its volume advantage, because every suggestion then needs somebody to action it by hand.

Does AI replace data governance?

No, and it depends on governance to work. A model cannot decide what a field means, which record survives a merge, or who may see what — those are decisions rather than patterns, and they need a named owner. A model pointed at a domain nobody owns will find every problem and resolve none of them, because there is no one to confirm what it suggests.

What data do you need before using machine learning on data quality?

Three things. Labelled examples from your own records, produced by data stewards who know the domain, of which a few thousand is a realistic start. Enough volume, including the awkward cases rather than only the easy ones. And a written definition of what correct means for the fields that matter. The labelling is the bulk of the project, and no vendor demonstration includes it because the labels have to be yours.

How do you know if an AI data quality tool is working?

By sampling it, permanently. Have a person read a small set of the model's decisions every month — twenty is enough — and record whether each was right. That record is your drift detector and your retraining set at the same time. Without it, accuracy declines quietly as the business changes, and nothing in the system tells you.

Is AI reliable for master data management?

For finding candidate duplicates, yes, and it is far better than manual review at that volume. For merging them automatically, no — a wrong merge combines two real customers into one record and is very hard to unpick. Keep a person confirming merges above the threshold permanently, and write the survivorship rule, which field wins from which system, before any matching runs.