The first version of our validator was a queue. Rows arrived, went into the queue, workers pulled from it, checks ran, results went to a results topic. It is the design everybody draws first, and there is a good reason for that: it is correct and it is easy to reason about.
It also topped out at roughly 300,000 rows a minute per cluster, and the latency distribution had a tail you could see from space.
Where the time actually went
Profiling was unambiguous. Less than 12 percent of wall clock was spent evaluating rules. The rest was serialisation, queue round trips, and the coordination overhead of workers agreeing on who had which batch.
We were paying distributed systems costs to run an operation that is, for the overwhelming majority of rows, a handful of comparisons against a small in memory structure.
Moving the check to the data
The change was to stop moving rows to the validator and start moving the validator to the rows.
Rules compile to a compact bytecode. That bytecode is small enough, a few kilobytes for a typical column set, to ship to every ingest node and hold in memory. A row is validated on the node that received it, before it is written anywhere, without leaving the process.
The queue still exists. It now carries only failures and periodic aggregates, which is a fraction of a percent of the volume in a healthy pipeline.
Throughput went from 300,000 to 4.2 million rows a minute on the same hardware. The p99 latency went from 340ms to 11ms.
What we gave up
This is not free, and the tradeoffs are worth stating plainly.
Rule updates are no longer instant. A changed rule has to propagate to every node. We settled on a five second bound, with a version stamp on every validation result so you can tell which rule version judged a given row.
Rules cannot query other rows. A check such as requiring a customer ID to exist in the customers table does not fit in the model. Those run in a second pass, off the hot path, on a delay. We were nervous about this and it has mattered less than we expected: roughly 4 percent of rules in customer deployments are cross row.
Debugging got harder. When validation happened in one place, you attached a debugger to one place. Now it happens in eighty. We invested in a replay tool that runs a given rule version against a captured row on your laptop, and that closed most of the gap.
The general lesson
Every time we have found a large performance win in this system, it has come from removing a hop rather than making a hop faster.
That is not a universal law. But it is a good first hypothesis, and it is cheaper to test than a rewrite.
