← All posts
Deep dive #data-modeling#debugging#firestore#schema-drift

When a filter's counts don't match, read the raw data before patching the query

A filtered view undercounted results, and querying the raw collection directly turned up two conflicting field conventions rather than a query bug — the fix was to revert to unfiltered and schedule a migration, not OR two fields together.

A grid of category tiles filtered a list of records by a field on each document. After wiring the filter, the counts were visibly wrong — one category showed 2 results when a manual count expected 6, and three others showed 0 despite records that were clearly tagged that way on screen. One report was even more specific: a record existed with a category value that, by the current data-entry form’s own dropdown, couldn’t exist.

That last detail is what made “the query has a bug” feel wrong. A malformed query drops or over-matches rows; it doesn’t produce a value the input form is incapable of writing. That’s the signature of two different write paths disagreeing about what a field means, not one write path with a broken read.

So instead of staring at the query, the next step was to query the raw collection directly, read-only, and inspect every document by hand. Fourteen records. Four of them — written through the app’s own entry form — stored the category correctly in field A, with field B used for an unrelated marker. The other ten had the exact opposite convention: field A held the unrelated marker, and the real category lived in field B. Those ten were identifiable as data from a different origin (an import/seed path, not the in-app form), which is exactly why the form’s dropdown couldn’t produce the values seen in field B — the form had never been the thing writing them.

Once that was visible, “fix the filter” stopped being a single-field problem. A filter checking only field A would miss the ten imported records entirely. A filter checking only field B would miss the four native ones. An OR across both fields does work — it was drafted and tested — but it’s a patch that encodes “this data is inconsistent” permanently into the read path, for every future query against that collection, instead of fixing the inconsistency once at the source.

The call was to not ship the OR. The filtering feature was reverted back to its unfiltered state — every tile routes to the same unfiltered list — and the actual fix was scoped as a data migration: reconcile the two field conventions across the existing records so there’s one canonical field with one meaning, then re-enable filtering against a schema that’s actually uniform. Slower to ship, but it means the next feature that touches this data doesn’t have to rediscover the same two conventions and write its own OR around them.

The general shape worth remembering: when a UI-level bug report includes a detail your code shouldn’t be able to produce, stop debugging the code that’s downstream of the data and go look at the data directly. And when you find the data itself is inconsistent, resist the urge to write a query clever enough to paper over both conventions — that cleverness has to be re-derived by every future reader and every future query. Fixing the data once is cheaper than every future query re-deriving the same workaround.