Big Data technologies are actively discussed in the industry for at least the last five years. It would seem that this time is enough for the transition of big data from the field of speculation to the practical plane, into the class of productive use technologies, but last year Gartner analysts excluded Big Data from breakthrough technologies and the Hype Cycle schedule. The industry even talked about “the death of big data.”
Does this mean that the technology is becoming obsolete without reaching the stage of maturity and widespread use? Not at all. Simply the concept of “big data“, which sometimes began to include almost everything associated with the storage and analysis of data, was too blurry and fuzzy.
At The Crossroads
In the meantime, technologies and platforms for big data continue to evolve rapidly. Currently, large vendors – server and storage vendors, software developers – focus on specific areas of big data, actively interact with the Open Source community, releasing joint solutions. To make it easier and more efficient to work with big data technologies, they complement the product lines with missing features and optimize existing ones.
An interesting direction is the integration of data. For example, it allows users to work with Hadoop data using the usual tools (SQL, BI, etc.). System architects, as a result, have the opportunity to choose the place of the most efficient storage of certain data – in a relational DBMS or in Hadoop.
Although databases allow you to work with both structured and unstructured data, in some cases, Hadoop is more profitable. For example, Hadoop is usually used to analyze petabytes of unstructured data, while DBMSs traditionally demonstrate their best qualities in the tasks of analyzing structured data – transactions, customer information, call data, etc.
Another direction is the acceleration and optimization of data access, which is especially important when converting or moving large amounts of data. Open Source tools are not always effective in performance. Developed by vendors, optimized solutions allow several times to accelerate the transfer of data between different locations and storage technologies.
The great importance of big data is acquired by security. If traditional relational databases have long enough strong and flexible built-in security mechanisms, then Hadoop was not originally designed for deployment in a complex corporate multi-user environment. For this purpose, “superimposed” corporate security is applied. For example, if users access Hadoop data through a DBMS, then all security settings and access delimiters are configured at the database level.
As a rule, large data are collected from different sources, have unconfigured formats. Is it worth spending time preparing these data? The newest tools of the Big Data Discovery class provide an opportunity to get an approximate idea of the contents of the data, to see some regularities, even if the data is not practically structured, and thereby assess the usefulness of the data and the need for further work with them. This direction is developing throughout the world. For example, an automotive company can, on the basis of feedback collected from customers, identify certain patterns and hidden defects, analyzing customer complaints and reviews.
Data in The Clouds
Increasingly, tools for big data are offered not only in the local version, when the product is deployed in the customer’s data center but also in the cloud (in a public cloud). For example, a customer can deploy a software and hardware complex to work with big data and analyze it “at home” or rent it in the cloud.
Develops and “cloud business analytics” (Cloud BI). Powerful computer systems in the provider’s data center can cope with very complex analytical tasks. Clouds allow you to get IT resources and not engage in the life cycle of servers and software updates.
But how to transfer large amounts of data to the cloud? The easiest way is to accumulate and analyze them directly in the cloud. There are also products for uploading data to the cloud and data integration, according to the rules, periodically collecting data from different sources and placing them in the specified location, including the cloud.
In addition, there are solutions that can read database logs and thus receive notifications of all changes in it. They can, with certain changes in the database, copy these changes to a cloud database, thereby synchronizing local and cloud data in real time. And if the same products work in the cloud and locally at the customer, there are no compatibility problems, data migration is simplified.
Cloud model eliminates the need for the customer to plan the purchase of servers, allows you to start with small volumes. Therefore, interest in cloud services for big data is growing.
From Theory to Practice
Although the use of big data has not become massive, there are more and more examples of the practical application of such technologies. Big data is already an objective reality.
Surveys show that 38% of respondents use big data, mainly in such areas as marketing and sales (53%), general management of the company (49%), finance (42%) and logistics (37%). Basically, these are large companies. SMB enterprises usually have enough traditional technologies, and they generate much lesser data.
If in 2012 companies tried to understand and realize what kind of technologies they are having, then from 2013 interest shifted to implementation examples, and since 2014 practical use of big data technologies has begun. Today they are deployed, at least at the level of pilot projects, almost all major telecommunications companies, banks, retail. Recently, customers have become interested in solutions that simplify the IT infrastructure for big data.
For example, the Hadoop multi-node cluster is quite labor intensive. Ready-to-deploy software and hardware systems are corporate-level support with a single “entry point”. The problem of support and optimization is solved when working with hundreds of terabytes of data. In addition, optimized complexes hide the complexity of technology, allow you to transparently work with different data sources, integrate data “on the fly”, unify them.
Prospects for Big Data
Big data technologies are booming, and the reduction in data storage and processing opens up new opportunities for analysis. There are promising technologies that were not even in theory before.
According to IDC forecasts, the unified architecture of data management platforms will become the basis of the corporate strategy for working with big data and analyzing them, with unification covering management, analysis, and data retrieval technologies. This should help to solve one of the main problems of large data projects – the problem of data collection and ensuring their quality, and also use more complex data source architectures.
Leading vendors are developing the concept of using big data in enterprise environments, ensuring the collaboration of Hadoop, NoSQL, and SQL technologies, as well as their safe application in any model, be it public, private cloud, or enterprise infrastructure. You can use the skills of database users – knowledge of Hadoop is not required for queries.
An interesting perspective opens up the Internet of things in combination with cloud technologies. Using compact Java-applications, it is possible to integrate quite complex algorithms into various intelligent sensors, to collect and process big data sets in the cloud. While companies are thinking where it can be used. But undoubtedly, the Internet of things and clouds will allow creating breakthrough technologies with big data.
