For data teams, bad data, broken data pipelines, stale dashboards, and 5 a.m. fire drills are par for the course, particularly as data workflows ingest more and more data from disparate sources. Drawing inspiration from software development, we call this phenomenon data downtime- but how can data teams proactively prevent bad data from striking in the first place? In this … [Read more...] about 5 Strategies For Stopping Bad Data In It’s Tracks
data quality
Assuring Data Quality: How to Build a Serverless Data Quality Gate on AWS
Data is a vital element in business decision-making. Modern technologies and algorithms allow for processing and storage of huge amounts of data, converting it into useful predictions and insights. But they also require high-quality data to ensure prediction accuracy and insight value. In today's world, the importance of data quality validation is hard to overestimate. For … [Read more...] about Assuring Data Quality: How to Build a Serverless Data Quality Gate on AWS
Data Lineage is Broken – Here Are 5 Solutions To Fix It
Data lineage isn't new, but automation has finally made it accessible and scalable-to a certain extent. In the old days (way back in the mid-2010s), lineage happened through a lot of manual work. This involved identifying data assets, tracking them to their ingestion sources, documenting those sources, mapping the path of data as it moved through various pipelines and stages of … [Read more...] about Data Lineage is Broken – Here Are 5 Solutions To Fix It
You Can’t Out-Architect Bad Data
Say it with me: bad data is inevitable. It doesn't care about how proactive you are at writing dbt tests, how perfectly your data is modeled, or how robust your architecture is. The possibility of a major data incident (Null value? Errant schema change? Failed model?) that reverberates across the company is always lurking around the corner. That's not to say things like data … [Read more...] about You Can’t Out-Architect Bad Data
Data quality: What and why is it important?
With the internet producing quintillions of readily available information per day, you could be forgiven to think that data is losing its value. Apparently, data is one of those weird commodities that go up in value the more they are available, or perhaps we haven't produced enough to attain the demand-supply equilibrium.Virtually all companies making good in various niches are … [Read more...] about Data quality: What and why is it important?
