Challenges Faced by ECommerce Businesses in Big Data Collection and Management

E-commerce is an important domain for big data mining due to its massive amount of click-stream and transactional data. Organizations collect this massive data, clean and transform it, and then analyze it to unveil business growing trends. In this article we will discuss big data mining related challenges in three sections: data collection, data cleansing, and data processing.

The following challenges apply to data collection in the e-commerce domain:

Sampling at collection

Large e-commerce websites generate between 5-10 million page views on an average every single day. Logging each of these user sessions in the backend can cause tremendous strain on the servers, not to mention the storage requirements necessary to handle this load. One way to counter this challenge is by sampling at the source. Sampling clickstream collection would effectively address the two issues mentioned above, although it introduces new problems. For one, sampled data will not be able to accurately capture rare events such as searching for a particular term or credit card authorization failure. Furthermore, business requirements, such as payment for advertising click-through referrals, require exact (rather than approximate) statistics.

Supporting changes in demographics

Customer demographics change; people get married, their children grow, their salaries change, etc. With these changes, the customer‘s needs, which are being modeled, also change. The challenge is to keep track of these changes and provide support for such changes in the analysis.

The following challenges apply to data cleansing in the e-commerce domain:

Detecting bots

Spiders and crawlers; collectively called bots, are automated programs that visit websites. Typical bots include web search engines (like Google), site monitoring software, and price and email harvesters. Due to the volume and type of traffic that they generate, bots can dramatically change click-stream patterns at a website. On the other hand, they can also be responsible for skewing any click-stream statistics.

For example, it is observed that the average page views per visit when bot visits are included are 1.5 to 2 times than the average page views per visit when bot visits are excluded. Large eCommerce websites typically see anywhere between 5% to 40% of their traffic from bots and other automated traffic sources. This makes it difficult to perform clickstream analysis since most of these bots do not actually identify themselves that way. In other cases, bots tend to pretend like they are real visitors. The only way to filter these bot visitors from skewing the clickstream analysis is by performing a set of heuristic and manual labeling on a continuous basis.

It is worth mentioning that page tagging methods of clickstream collection, which execute blocks of JavaScript at the client’s browser and log the statistics returned by the JavaScript at a server, avoid bots because they require the execution of JavaScript, which bots rarely execute. At the same time, if a human visitor has their JavaScript turned off in their browser or clicks on a link before the JavaScript code is completely executed could have their visits flagged erroneously as bot traffic. Such visits are normally around five percent of all human traffic and leads to an inaccurate clickstream measurement.

Performing regular deduplication of customer accounts

Transactional systems do not usually offer checks to prevent the duplication of customer records. In some businesses, one customer may have multiple accounts and it may not always be possible to consolidate all activities performed by one customer into a single record.

Challenges also occur when the same kiosk or terminal is used by multiple people to log on to a website. If the website or platform does not have enough tracking parameters to uniquely identify each of these visitors separately, then this leads to a scenario where clickstream analysis of one user could actually include information from several users.

The following challenges apply to data processing in the e-commerce domain:

Supporting hierarchical attributes

Supporting hierarchical attributes is important in practice. A few algorithms have been designed to support hierarchical attributes directly but they do not scale to large hierarchies. The process of automating the process of utilizing hierarchies effectively still remains challenging.

Related products

One of the common strategies used by eCommerce websites to drive the average order value up is by promoting related products.’ Algorithms make use of big data analytics to map the purchasing behavior of every customer to identify patterns in purchase. By extrapolating the purchasing history of other customers who purchased a particular product, the algorithm could precisely predict the kind of products a buyer may find interesting.

Handling unknown and invalid attribute values

ECommerce websites routinely tag the products in their inventory with attributes such as color, size and weight to help with searches and product filtering. These attributes are also extremely useful for data mining purposes since these attributes may be used to find generalizations and patterns in user behavior based on these product attributes.

It is also possible to find some attributes that influence user behavior in certain classes of products, but not in others. For example, size makes sense for clothes and shoes, but not for books. For books, the size attribute would have a NULL value. In this case, NULL means not applicable , rather than unknown , and needs to be treated differently. Distinction between two interpretations for NULLs is done using meta-data. For every attribute, meta-data would determine whether a NULL value should be treated as either not applicable or unknown. Not applicable is a distinct value for mining purposes, whereas unknown implies that the attribute is relevant but its value is unknown. Not many data mining algorithms can correctly accommodate this subtle difference.

Big data mining for e-commerce websites unfold new patches to look and work into for business growth. The aforementioned challenges need to be addressed in order to have an edge over the competitors. Overcoming these will help both, the customers, and the service providers in many ways.