Schema drift is a people problem

Every drift incident I have investigated had a technical trigger and an organisational cause. The tooling only helps with the first one.

A product engineer adds a field. It is optional, it defaults to null, and it ships on a Tuesday afternoon with no drama.

Six weeks later a forecasting model is quietly wrong, because the field was not optional in the sense the data team meant, and null was being read as zero.

I have investigated some version of this maybe forty times. The technical trigger is always different. The organisational cause is always the same: the person who made the change and the person who depended on it had no shared surface where the change was visible.

What does not fix it

A review process. Schema changes routed through the data team for approval works for about four months. Then the queue gets long, the exception path gets used, and the exception path becomes the normal path.

A wiki page. Documentation of the current schema is a snapshot. The failure is about a transition, and snapshots do not record transitions.

Blaming the product engineer. They shipped an optional field with a default. That is correct behaviour by every convention they have been taught.

What has worked

The deployments where this stopped happening had one thing in common, and it was not a tool. It was that schema changes produced a notification in the channel where the consuming team already worked, with the consumers named.

Not a dashboard they had to visit. Not an email alias. A message that said: this column changed, these four pipelines read it, here are their owners.

The tooling to do that is genuinely easy. Anything that tracks lineage can produce the list. The reason it works is not the list, it is that it arrives without anyone deciding to look.

The second order effect

Something changes once the notification exists.

Product engineers start writing better change descriptions, because a real named person is going to read them. The optional field that defaults to null gets a sentence explaining what null means. That sentence is worth more than the entire review process it replaced.

You do not get that by making the process stricter. You get it by making the audience visible.

Where automation belongs

Detect the drift automatically. Route it to humans deliberately. Do not try to resolve it automatically, because the resolution is a judgement about intent, and intent is not in the schema.

Author

Mara Oduya

Field CTO

Mara spends her weeks inside customer data platforms. She writes about what actually breaks in production, which is rarely the thing the architecture diagram warned about.

All articles by Mara Oduya →
Back to the journal