Understanding And Improving Data Quality For AI And Machine Learning

Artificial Intelligence and Machine Learning are evolving at a fast pace and are set to transform businesses. By automating decisions and processes, they can make processes more efficient and streamline systems. That said, the decisions made by Artificial Intelligence and Machine Learning can only be as reliable as the data they are based on. Thus, data quality is of critical importance to the success of such projects.

Understanding data quality and its importance

Data quality can be measured along many criteria including consistency, accuracy, validity, integrity and completeness. It is important to note that data quality measured according to these criteria can be good’ in some instances and bad’ in others.

For example, the sales records of a single store may be complete and sufficient to be quantified as good’ quality data if you were looking at sales numbers for the store itself. But, if you were looking at sales records for the city, this data may be incomplete.

Thus, a better definition would be to say, data can be considered to be high quality if it is fit for its intended use in decision making, operations and planning and if it correctly represents the real-world construct it refers to.

In the era of Artificial Intelligence and Machine Learning, systems must be able to identify data quality and proactively identify potential issues. Depending on the criticality of the issue, the system may also deny publishing the data to clients or share the same while flagging alerts.

As organizations become more data-driven, checking data quality becomes even more important. Poor data quality can lead to reduced trust in the decisions based on them, poor decisions and even a waste of resources. If users cannot trust the data and decisions based on them, they will gradually abandon the system and thus impact the success criteria.

The most common types of data quality issues include:

· Records with missing information

· Records with invalid information

· Duplicate records

· Inconsistency in terms of units of measurements

· Inconsistency in terms of formatting

· Broken URLs

· Incomplete cases

· Corrupted binary data

· Missing data packages

· Incorrectly mapped properties

· Gaps in the data feed

Many factors may cause such issues. The most common amongst them include:

· Failures during certain processes that in turn lead to system-level issues

· Differences in data formats that impact the source and target data stores

· Software bugs

· Poor software implementation

Developing a strategy to improve data quality

A data-intensive AI or Machine Learning project will typically involve multiple data streams, complicated ETL processes, post-processing logic and many cognitive and analytical components. To ensure good quality data through these processes, organizations must ideally maintain at least one data store that empowers knowledge extraction, real-time decision making and advanced analytical models. There are 5 steps that can help achieve this.

Step 1: Identify and document data sources

To begin, you must be able to identify the different data sources and document details about them such as:

· Type of data recorded

· Storage types

· The time frame the data is stored for

· Type and frequency of updates

· Known limitation and data issues

· Where the data is coming from and systems involved

· Data models used

· Involved stakeholders that can provide a better understanding of the business, data and related processes.

Step 2: Data profiling

Data profiling refers to describing the data with a summary and basic descriptive statistics and analysis. The key here is to create a baseline that can be used to validate data at various stages of the process. Some of the key points to be considered when profiling data include:

· Identify the main entities such as users, customers, products, events involved, login, register, purchase, period and location.

· Select an appropriate time frame for the analysis. This could vary from a day to a month or even longer depending on the business.

· Analyze trends, peaks, seasonality, etc. involving the events and entities identified within the selected timeframe. These trends can then be interpreted according to the context of the business.

· Analyze data according to its types. For example, numerical values can be analyzed according to the minimum and maximum range, averages, standard deviation, etc.

· Review the outliers to interpret suspicions in the context of the way data should be used.

· Document the results to act as your baseline for future reference.

· Review, interpret and validate input from the data owner

Step 3: Establish a data quality reference store

Establishing a data quality reference store makes data accessible externally and helps capture and maintain validity and metadata rules. This could be a manual setup, automated or a hybrid setup. The idea is to be able to quickly validate incoming data and compare it to existing patterns. It should ideally be dynamic and flexible to deal with all the different kinds of data and issues that may arise from it. Once established, this should be accessible to standardized dashboards so as to empower data analysts to look into the data, trends, processes and issues.

Step 4: Validation for smart data

The data processing pipeline should be able to load data validation rules from the data quality reference store and use this to continually validate incoming data. It should be able to flag issues or enrich the incoming data with related metadata. This helps measure and analyze data quality. Interactive reporting can further help explore the overall state of the ETL process and identify data quality concerns. Issues could also be measured against a data quality index to prioritize certain issues over others.

Step 5: Implement smart notification

Overall, the process should be designed such that it is aware of any sudden changes to data trends and existing quality issues. The system should also know how important the issue is. Based on this information, a smart layer must be configured that notifies the relevant person so that it can be addressed and resolved.

According to a recent study, by 2025, sectors such as programmatic media will be 100% automated. If you see automation as part of your future, you need to pay attention to data quality now.