Hallowed Ground: Data Federation
Data federation technology is software that provides end-users with the ability to aggregate data from disparate sources and formats with virtual database objects. The benefits of this technology include increased availability and reliability as well as improved access times for BI and data analysis.
The major data warehouse players – IBM, Oracle, SAS, Teradata – set the bar for federation, and the solutions from these companies allow seamless access to data from multiple, external sources via the vendors SQL interfaces and APIs. In other words, if you have JSON files, DB2 data, XML, Sybase data… with these federation technologies, you can query across all these sources in a single SQL statement. Of course, these solutions have a cost, both in terms of software licenses and specialized engineers. Apache hopes to alter the landscape with its Open Source Spark project.
Oracle Keeps Pace
Oracle has aggressively kept up with supporting federated queries and providing a unified experience for working with data. Their latest suite of products, including GoldenGate for Big Data and ODI make it possible to tap into data across a wide range of sources, including Hadoop. This quote is from their Data Service Integrator website:
Oracle Data Service Integrator provides companies the ability to quickly develop and manage federated data services for accessing single views of disparate information. Oracle Data Service Integrator is completely standards based, declarative, and enables re-usability of data services. Use Oracle Data Service Integrator together with Oracle Data Integrator , Oracle GoldenGate,or Oracle Enterprise Data Quality for comprehensive data integration and management.
Apache Spark Wants a Seat at the Big Kids Table
Apache Spark has recently set its sights on becoming the cluster-based execution engine that can tackle federation the Open Source way, by having community-contributed components filling in the data source connectivity gaps. From Spark version 1.3 announcement blog:
Users can also intermix SQL and data frame operators on the same data sets. New in 1.3 is the ability to read and write tables from a JDBC connection, with native support for Postgres and MySQL and other RDBMS systems. That API adds has write support for producing output tables as well, to JDBC or any other source.
Community Driven
As part of Spark 1.3, Databricks, largely the chaperone of the project, announced Spark Packages. This is similar to Pigs PiggyBank in that it is a curated set of packages that are produced by the community. Fifteen data source related packages are already posted to Spark Packages and include the following connectors:
|
|
|
Examples
In the example below, weve created two temporary tables. One is really a remote MySQL table and the other is a remote Oracle table. The remote connectivity is enabled by the USING and OPTIONS values. Lastly, we join these two tables with a large Hive table. Viola Federation.
The DataFrame API Makes it Easy to Surface Collections to the SQL Layer
The example below shows how you can have custom code that creates a Spark RDD (resilient distributed dataset) object and then expose that to the Spark SQL layer. Sparks language interoperability feature means that you can intermingle SQL and code that works with the RDDs.
Sparks Vision
Sparks vision is to become a single execution engine for any datasource. Sparks federation technology addresses the problem of working with disparate data sources. However, Spark is addressing the entire end-to-end BI workflow, from the raw data to transformations and analytics. For instance, Spark has embraced the Data Scientist community. In Spark 1.3, in addition to renaming the SchemaRDD to DataFrame (a common concept for R and Panda users), Spark has officially embraced the SparkR package, a library for R that enables the user to tap into data on the cluster using the Spark execution engine.
