Here is a pattern I see in almost every data platform I am invited into.
A validation rule fires. The record fails. The pipeline logs a count, increments a metric, and drops the row. Three weeks later somebody asks why the March revenue figure is 2 percent lower than the source system, and there is no way to answer.
The count told you how many records failed. It did not keep the records.
Why teams do it
The reasoning is usually sound in isolation. Bad records are, by definition, bad. Keeping them costs storage. Keeping them in the production table risks somebody querying them by accident. Deleting them is simple and it keeps the warehouse clean.
Each of those is true. Together they produce a system that cannot explain itself.
What quarantine costs
A quarantine store is a separate location that holds every rejected record along with the rule that rejected it, the rule version, and the timestamp.
In every deployment I have measured, quarantine volume settles between 0.1 and 0.8 percent of total ingest. On a platform handling a billion rows a month that is a few million rows in cheap storage. It is not a meaningful line item.
What it buys you is the ability to answer three questions that come up constantly:
- Why is this total different from the source system?
- Was this record ever present, and when did we stop accepting it?
- If we relax this rule, what comes back?
That third one is the underrated one. Rules get written defensively and tightened over time. Without a quarantine store, loosening a rule is a leap of faith. With one, it is a query.
The part people get wrong
Quarantine is not a dead letter queue.
A dead letter queue is a retry mechanism. Things go in because something transient failed, and the expectation is that they come out. Operationally, it is sized for a backlog and alarmed on depth.
Quarantine is a record. Things go in because they were judged invalid, and most of them never come out. It is sized for retention and alarmed on rate of change, not depth. A quarantine that suddenly doubles is telling you something upstream broke. A quarantine that is simply large is telling you nothing.
Running both through one mechanism produces a queue that is always alarming and that nobody looks at.
One rule
If a record was good enough to arrive, it is good enough to keep a copy of for ninety days.
That is the whole policy. Everything else is implementation.
