How-to Guide to Handling Missing Data in AI/ML Datasets

Artificial Intelligence and Machine Learning are the noble pursuits that depend largely on the data they are fed. With this data, systems figure out the future path and learn to handle complex scenarios. All of the applications of Machine Learning and Artificial Intelligence makes sense only when the supplied data is complete and rich.

But, in the real world, the data is not perfect, just like everything else. But, there are steps to fix the data when it is incomplete, incoherent, and unsuitable. Today, we discuss the methods to treat missing data when a comprehensive data is required for ML and AI applications.

Whether to ignore the missing values or to treat them effectively, depends on some factors to be considered such as the percentage of the missing values in the dataset, the variables these values affect, and whether the missing values belong to a dependent or an independent variable, etc.

The performance of your predictive analytics depends on the accuracy and the integrity and the completeness of the data. Therefore, it becomes necessary to treat missing data when the need arises.

Treatment by Deletion

The best avoidable method to get over the missing data is to delete the record. This can be done either listwise, where the rows that contain any missing data are deleted, or pairwise, where the missing data is simply ignored and the variables that are present are considered. Since both these approaches and the method of deletion lead to loss of information, this methodology of dealing with partial or missing data is seldom used when the deletion of some records will not substantially affect the overall system.

Treatment by Replacement

If the missing values belong to a numeric field, the values can be statistically replaced. For example, if the ages of some people in a dataset are missing. These missing values can be filled by the mean or mode or median of the present age values in the dataset. This approximation will surely add variance to the dataset, but there will be no loss of information in this case. This approach works better the size of the data is considerably small.

Treatment by Predictive Imputation

In this case, predictive techniques are used to replace the missing values with slightly better variable values than the completely randomized averages of all the values. Regression techniques can be employed to accomplish treatment of missing values in the dataset by predictive imputation. Many other algorithms can also be tried to identify the one that yields the correct predictions. If you use ML and AI as a service through a platform, say Microsoft Azure Machine Learning, then you can freely choose between the available algorithms. Amazon ML will fill up the missing values in your dataset without your involvement at all.

Using algorithms that work with missing values

There are some AI and ML algorithms that can be used when the data has some values missing. For example, KNN is a machine learning algorithm that works on the distance measure principle. The algorithm is suitable to be used when there are null values in the dataset. Using these algorithms reduces your burden to treat the missing data as the problem is handled by the algorithm itself. RandomForest is another algorithm that can be used here. Using these algorithms eliminates the need to create predictive data models for each attribute that is missing in the dataset.

Almost all datasets have values missing and other flaws, and making this data perfect for further analytics is the job of a data scientist. ML and AI are no DIY tasks, and that increases the need for a data engineer or data scientist for you.