Thread by @jeremystan

1/ I often used data “quality”, “anomaly” and “outlier” interchangeably, but now have a much crisper sense of what I think each term should really mean.

Here are some definitions and advice for what to do about them.

2/ Data “quality” is when the data, as recorded or reported, is inconsistent with the reality it was intended to capture.

Data quality can only be assessed by an expert who understands the physics of the system being monitored.

3/ A data “anomaly” is a sudden change in the underlying system that generated the data, which can be observed through pattern analysis of the data over time.

4/ An “anomaly” can be caused by a data capture or processing bug, an intentional data capture change, a change in your product or service, or by an exogenous change in the real world.

It is not 100% possible to distinguish between the four, but there are tells.

5/ An “outlier” is a data point, which is in some sense “far away” from other data. Defining an outlier in a non-trivial dataset requires expert judgement about the distance metric space each column should be embedded into, and how correlated columns should be handled.

6/ All data has “outliers” - their occurrence is a natural part of the underlying data generation process (if you think broadly enough).

Some outliers are data quality issues (data entry), and a sudden change in the percentage of outliers could be indicative of an anomaly.

7/ Everyone should care about data “quality” - and given enough people using a dataset and a solid process for surfacing and remediating issues it will improve. But at a high organizational cost.

8/ Monitoring "anomalies” (if you can do so broadly while controlling for false positives) is a great way to accelerate data quality with minimal organizational burden.

9/ Outliers can be interesting, and in some domains (fraud) can be crucial to understand. But as humans, we will tend to overfit them and come to conclusions that will not generalize.