We have come a long way in terms of the technology and infrastructure required to handle big data processes. It is not uncommon for retailers to run big data tests on their customer database to identify patterns and trends in buying behavior. Similarly, businesses routinely run big data processes on their logistics and operations to optimize routes and improve their margins. Many of these everyday big data problems have static large databases to deal with and consequently, the challenges here are mostly infrastructure-related and may be fixed by increasing capacity and investments.
This is not always sufficient though. High Frequency Trading, for instance, relies on interpreting millions of data points each second to make buy/sell calls on the fly. Instead of betting on major news stories like earnings report and product releases, HFT uses algorithms to identify and execute buy/sell calls often in fractions of seconds. The database here is extremely dynamic and this adds another layer of complexity to big data processing.
There are several other use-cases for dynamic big data processing. PR agencies monitoring brand reputation online often have to track down and reach out to negative reviews as and when they happen in order to minimize brand impact and negative virality. This process involves going through millions of tweets, Facebook posts and other community interactions each second, identifying mentions that are relevant to your business and also using inferential algorithms to identify the context and sentiment behind each post. Imagine a system that monitors thousands of such brand mentions each second and the challenges surrounding its implementation become obvious.
A dynamic big data project essentially has two challenges that are over and above the typical big data problem. Unlike traditional systems where a database with millions of datapoints is left untouched till the processes are complete, dynamic data systems require an infrastructure that is optimized to continually input and discard data. Also, in many cases, the output data may become irrelevant even if it was delayed by a few seconds. This makes it important to build technologies that can perform follow-up processes (like executing a buy/sell call or inferring potential negative brand mentions and notifying them) instantaneously.
One of the most effective ways to do this is distributed data processing. You can solve the dynamic data problem by breaking down the input into relatively smaller chunks that may then be processed in parallel. For instance, in the case of a reputation management system, the incoming aggregated stream of tweets and social media posts may be broken down into small clusters based on timestamp. Each of these clusters is one static database table that can be processed for output separately.
With a large enough infrastructure, it is possible to create thousands of such virtual clusters that may run incoming data in parallel. The output from each of these clusters may be aggregated before they are sent back to the user as results. Applications like Hadoop run their jobs in FIFO (First In, First Out) model by default. But through a distributed network or using schedulers like CapacityScheduler, you may customize your application to execute dynamic big data processes.
Dynamic data processing is in essence not too different from regular big data analytics systems, except for the additional resources it needs. But the potential use cases are vast and this could potentially be the future of big data computing.
