One of the hottest questions in Information Management now is how to deal with Big Data in all its applications: how to gather, store, secure, and possibly most importantly interpret what we collect. Organizations that are able to apply effective data analysis to massive amounts of data gain significant competitive advantages in their industries.
Organizations no longer question the value of gathering and storing such data but are far more heavily focused on methods to make sense of that all the valuable information that data represents. Although security and storage remain critical issues for IT departments, organizations are finding that their commitment to Big Data cant stop there they must be able to make sense of their data, to know what data is valid, relevant, and usable, as well as how to use it.
The more data an organization has, the more difficult it is to process, store, and analyze, but conversely, the more data the organization has, the more accurate its predictions can be. As well big data comes with big responsibility. Big data requires military-grade encryption keys to keep information safe and confidential.
This is where data science comes in. Many organizations, faced with the problem of being able to measure, filter, and analyze data, are turning to data science for solutions hiring data scientists, people who are specialists in making sense out of a huge amount of data. Generally, this means making use of statistical models to create algorithms to sort, classify, and process data.
What is Data Science?
Data science has been a term in the computing field since around 1960 when it was first floated as a substitute for the term computer science. Over the next twenty years or so, it gradually came to mean that blend of statistics and methodology that specifically pertained to data analysis. However, it was not until the much more recent emergence of Big Data and its role in organizational development and direction, that data science began to be a fundamental requirement of any organization working out how to analyze such massive amounts of data.
Data science is interdisciplinary, incorporating elements of statistics, data mining, and predictive analysis, and focusing on processes and systems that extract knowledge and insights from data. It is also known as analytics transformation because the goal is to transform raw data into usable insights. It has also been called industrial analytics because the context is industrial rather than scientific to analyze data for competitive or quality improvements that can be gained by having a better understanding of ones customers, potential customers, service model, and almost any aspect of the organization that can be represented in bytes.
Because the cost of computing and analyzing organizational data is declining (as the cost of technology tends to do), we can measure and analyze huge data sets with a level of precision not previously available. Methodology and algorithmic analysis provide the tools for this precision, and we will take a look at both of these critical aspects of data science in this article.
Methods of Data Science
Data scientists need to be able to combine flexibility and agility with rigorous analysis and the scientific method. Walking this line is very difficult, but without both of these seemingly opposite qualities, it is almost impossible to extract value from huge, diverse data sets. To do so requires experimentation and exploration, as well as a commitment to long-standing principles for scientific objectivity:
- Report the facts as they are, not as you were hoping they would be;
- Conclusions cannot always be legitimately drawn from a given data set;
- A lack of evidence for a theory does not prove that the opposite is true;
- Dont compare unlike data its like trying to add grams to meters;
- Do not draw cause-and-effect conclusions that are incorrect for instance, correlation does not equal causality;
- Ensure your initial data is reliable;
- Dont oversimplify or overcomplicate; and
- Base your conclusions on the full set of data dont choose data to support a conclusion.
We can take a look at three methodologies for applied data science in an organizational context:
1. Classification
Classification is the creation of classes that represent users and use cases. Class Probability Estimation tries to predict how to classify each single individual data asset. Based on the specific question to be answered, classes are created. Other qualities of the asset are evaluated in making the prediction. A simple example is a company evaluating the likelihood of success of a given promotional plan. The classes might be will sign up and will not sign up. Based on factors such as age, location, and purchase history, the model provides a prediction of success.
In a scoring model, the classes are scores perhaps on a scale of one to ten of how likely either outcome is for a given asset.
2. Regression
The most commonly-used forecasting method is the Regression method. Regression can be confused with classification methods because the process of using known values to predict an outcome is the same. But Regression is the method of trying to predict a numerical value for a particular variable for that individual data asset.
An example Regression problem would be: What should be the cost of our new product? The price of the product is an unknown variable for which Regression is attempting to find a value. A model could query such things as expected demographics disposable income, cost to manufacture, the price of similar products on the market, etc.
3. Similarity matching
Similarity Matching looks for correlation of attributes in order to recognize similarities between individuals. If two customers or products are similar in certain ways, its reasonable to predict that they will be similar in other ways as well. This can be used to find customers for targeted marketing campaigns or for managing the companys image with targeted online ads. The similarity profile will reflect characteristics that will have a bearing on the question at hand, so it could include attributes such as age and purchase history. This approach is often used to provide recommendations to online customers especially when people are similar in selecting or purchasing products that are named Similarity Matching being applied in real life.
Data scientists use these principles in algorithms, which can be defined as sets of process rules for a computer to follow, to analyze massive amounts of data. Algorithms can be developed from statistical models, which are helpful for interpreting graphical models where there are multiple unknowns and some special dependency or relationship exists between the unknowns such as matching up your set of customers with your taxonomy of customer classifications. Some very affecting algorithms largely depend on unsupervised machine-learning so they can refine their effectiveness as they are used, despite the depth of the data and the number of unknowns.
Algorithms Used in Data Science
Businesses are increasingly relying on the analysis of their data to predict consumer response and recommend products to their customers. However, to analyze such massive amounts of data, the solution clearly has to be compute-driven. In data science, a number of algorithms built on statistical models are available for data scientists to create analytic platforms. Which algorithm is chosen is based on the goals that have been established beforehand, just as a statistician chooses the appropriate statistical model based on the problem to be solved. Although there are many algorithms, these methods, Classification, Regression, and Similarity Matching are the fundamental principles on which many of the algorithms used in data science rely.
Some algorithms were developed to address business problems. Some were developed to augment algorithms in use for other purposes, or to have them perform somewhat differently, to tune them to a business environment. These algorithms can be used, for instance, to remind customers of an event, or to target likely credit card applicants. Although one algorithm might be clearly better for a certain purpose than another, its sometimes very useful to try more than one. Doing this can provide comparisons and often turn up some unexpected results that can tell you more than you expected about your product or your customers.
Ten of the most commonly-used algorithms are:
1. K-Means Clustering Algorithm
A simple, unsupervised learning algorithm that is often used with big data sets, often as a way of pre-clustering or classifying into larger categories that other algorithms can further refine. It has some other inherent problems that make it best suited to large-scale, high-level clustering.
2. Association Rule Mining Algorithm
Sometimes referred to as Market Basket Analysis, since that was the original application of this algorithm, the association rule algorithm is a learning algorithm that looks for associations that co-occur with a high degree of frequency. It can identify associations that you might not expect in a random sampling a famous example was when Walmart found that when a hurricane was coming, along with their bottled water and flashlight batteries, people were buying a huge number of Strawberry Pop-Tarts. Whenever hurricanes were predicted, Walmart stocked up on Strawberry Pop-Tarts, and they soldlike Strawberry Pop-Tarts in a hurricane.
3. Linear Regression Algorithms
One of the most widely-used methods of statistical analysis, linear regression is applicable to many problems, particularly when the expected output is a score rather than a category. It is good for predicting trends and to forecast the effects of a new policy or other change.
4. Logistic Regression Algorithms
Logistic regression is used to find the likelihood of success of failure of a given event. It is a classification algorithm.
5. C4.5
A supervised learning algorithm used to create decision trees from the already-classified input. Decision trees can be used as diagnostic tools in medicine, as well as in the business sector.
6. Support vector machine (SVM)
This algorithm learns to define a hyperplane to separate data into two classes. A hyperplane is the line that divides a group but is based on a property or attribute rather than location.This algorithm can help figure out an underlying separation mechanism between people who will buy a product and those who wont.
7. Apriori
The Apriori algorithm is a similarity matching algorithm. It is commonly used in transactional databases with a large number of transactions, but it does run with a high degree of computational overhead.
8. EM (expectation-maximization)
A clustering algorithm used for knowledge discovery. It uses clustering to predict data models that can be used in other statistical analysis methods.
9. AdaBoost
An algorithm which constructs a classifier and then boosts it meaning it looks for the best learning algorithm among a number of machines and chooses the most effective one then propagates the improved information to the other machines, in this way it optimizes the ability to learn of participating machines.
10. Nave Bayesian
Named for Thomas Bayes, an English statistician who also gave his name to Bayes Theorem, Nave Bayes is not one algorithm, but a family of classification algorithms.It is called nave because as it learns, it assumes all attributes of an item are independent of each other. The algorithm learns to predict an attribute based on other, known features.
Conclusion
These algorithms are the tools data science uses to bring statistical analysis to large data sets by classifying data, identifying similarities, and predicting trends. They enable companies to build new services and products that better match their customers needs, personalizing customer interactions, and targeting advertising more precisely. Using data science to analyze Big Data is an effective way of tapping into the inherent value of large data stores that otherwise is impenetrably hard to decode into meaningful information.
The huge data stores that organizations collect are continuing to grow in volume and diversity. Data continues to pour into collection systems from mobile phones, social networks, online trackers, eCommerce artifacts, customer surveys, and any sources that can be tapped for feedback that has potential value for an organization. Properly harnessed data can provide insights for an organizations marketing, product and service roadmaps, and reputation management. The insights gained from Big Data enable organizations to be driven by business intelligence and insight, which themselves are driven by complex metrics and analysis. Data Science is here to stay as a necessary part of the Big Data toolset.
