Most businesses today are consolidating data from multiple sources into a single customizable platform for big data analytics. Having a separate platform for data analytics lets you create dashboards to segment, aggregate, and analyse high-dimensional data and make low-latency queries to perform real-time analytics.
What platform should you use to power your data analytics machine? Data warehouses and data lakes are common two alternatives.
What’s a Data Warehouse?
A data warehouse is a central repository of data gathered from diverse sources like cloud-based applications and in-house repositories. Unlike the typical relational database, a data warehouse uses column-oriented storage known as a columnar database. Since the database stores data by columns rather than rows, it’s more suitable for data warehousing. Once you have a data warehouse set up and loaded with both current and historical data, people in the organisation can use it to create forecasting dashboards and trend reports using tools such as Looker, Chartio, Periscope Data, and Mode.
A data warehouse has the following characteristics:
- Integrated – The way the data is cleansed and extracted is uniform regardless of the original source.
- Non-volatile – Since the data in a data warehouse is periodically uploaded and not in real-time, any momentary change shouldn’t have much impact on decision making.
- Structured – A data warehouse should be structured using a columnar data store to improve analytical query speed.
- Scalable – The data warehouse is scalable to meet increasing demands for storage space.
A data warehouse also acts as a data-tier application that defines the instance level objects, schemas, and database objects used by a three-tier or client-server application.
What’s a Data Lake?
Data lake is a term coined by Pentaho CTO James Dixon in 2011 to refer to a large repository of data in its natural, unstructured form. Raw data flows into a data lake, and users can segregate, correlate, and analyse different parts of data based on their needs. Some key aspects of a data lake:
- A data lake accepts data from all sources. Enterprises choose not to reject any data inflow because, by definition, a data lake is okay with unstructured data.
- Data is collected from multiple sources in real-time and moved into the data lake in its original format.
- Data lake relies on low-cost storage options to store the raw data.
- Data in a data lake can be updated in real-time or batches making this a volatile structure
Which One Should You Use for Data analytics?
Data analytics is a process of drawing conclusions from data sets about the information they contain so that you can make informed business decisions. Both data warehouses and data lakes are capable of helping you do that – but 9 out of 10 times, you should be looking at data warehouses for data analytics. Why? Because they’re optimised for analytical queries, unlike a data lake.
But there are other use cases where you can deploy a data analytics engine where a data lake and a data warehouse can coexist. But that largely depends on your functional requirements in areas such as data structure, adaptability, and performance. Let’s have a look at that first.
Data structure
While developing a data warehouse, a major time-consuming factor is the analysis of data sources’ business purposes and profiling of different datasets to create a structured and organised data model designed for specific reporting needs. A key component of this process is deciding what data needs to be included and what should be excluded from a data warehouse.
This involves collecting data from various sources and aggregating and cleansing it. Data cleansing, also known as data scrubbing, is the process of cleaning up data and this happens before loading the data into the warehouse. The goal of data cleansing is the removal of outdated data. Once the data is cleansed, it’s ready to be analysed. However, it can take time and energy to do this because of the sequence of data cleansing processes involved. Data warehouse works best on clean data, and that comes at a cost.
A data lake, on the other hand, includes all relevant data sets irrespective of source and structure. It stores data in its original form.
This data retention is not limited to what may be currently in use but also includes historical data as well. This is made possible largely due to the nature of hardware required for a data lake. Usage of cheaper storage devices, often off-the-shelf servers, ensures that scaling a data lake is a fairly economical task. Modern data lake solutions can use cloud-based storage such as Amazon S3 to make use of the low-cost storage options.
This reduces the upfront cost of data cleansing and data transformation compared to that of a data warehouse. For instance, storing unformatted data on S3 is cheaper compared to running a full-fledged data warehouse on Redshift. This is because rules that govern the data is set on the way in the case of Redshift. However, data lakes are not performance optimised, and you’d be forced to incur additional costs at a later stage in the case of a data lake.
Adaptability
Should your data analytics platform be able to adapt to new organisational changes quickly? One of the main drawbacks of a data warehouse is the time taken to change their structure to reflect the changing needs of the organisation. Although a robust data warehouse is capable of quick change to adapt to different scenarios, the complexity of the task upfront requires developer time and resources.
A data lake, on the other hand, is contrastingly much quicker to adapt to changing requirements due to the simple fact that all the data remains in its unstructured or raw form. This unstructured data is available to users who are empowered to access and use it to form their analysis based on their needs. However, keep in mind that the developers will eventually need to jump in and devote time and resources to get meaningful information from this data.
Performance
Data warehouses are built for quick and easy analytical processing. The underlying columnar RDBMS offers accelerated performance that’s optimised for analytical query processing. This includes complex joins, high concurrency, etc. You can expect the system to return query results in a few seconds or less.
Data lakes are not performance-optimised. Those who have access to it can explore the data at their own discretion, often resulting in a non-uniform approach of data presentation to internal or external stakeholders. But since they’re working on an unstructured data set, it’s hard to join data and query meaningful information quickly. Less technical staff might find it hard even to use the data and ask questions to generate reports and acquire anything relevant from the data lake.
Modern Data Analytics in the Cloud
Amazon, Google, Microsoft, and others offer data warehouse and data lake services in the cloud that provide platforms against which businesses can run real-time analytics and BI reporting. Amazon Redshift and Microsoft Azure are built on top of a relational database model and offer elastic, large-scale data warehousing services. On the other hand, Google Cloud Datastore is a NoSQL Database as a Service (DaaS) that automatically scales. All the cloud data warehouses have BI tools integrated into their services.
For instance, Amazon Redshift has a Massive Parallel Processing Architecture and supports single node clusters to 100 nodes clusters with up to 1.6 PB of storage. Microsoft Azure’s SQL Data Warehouse also has an MPP architecture and stores data into relational tables with columnar storage. This lets you run analytics at massive scale.
So, what’s your favourite data warehousing tool in the cloud? Share them in the comments.
