In this Specialization, you’ll build practical skills for designing, deploying, and improving batch and streaming data pipelines on Google Cloud. You’ll work with Dataflow, Apache Beam, Cloud Data Fusion, Pub/Sub, BigQuery, Bigtable, Cloud Storage, and Apache Kafka to move, transform, and process data at scale. You’ll start with guided Dataflow and Data Fusion labs, then build ETL workflows, transform data with Wrangler, and create batch and real-time pipelines. You’ll also explore streaming architectures, windows, watermarks, triggers, sources and sinks, schemas, state and timers, SQL, DataFrames, and Beam notebooks. Along the way, you’ll address data quality, monitoring, orchestration, IAM, quotas, security, performance, and data locality.
By the end of this Specialization, you’ll be able to: Design and build scalable batch and streaming pipelines for common data engineering use cases. Use Dataflow, Data Fusion, Pub/Sub, Kafka, BigQuery, and Bigtable across end-to-end workflows. Apply data quality, monitoring, security, and performance practices to pipeline operations. Develop Apache Beam pipelines using streaming concepts, schemas, state, timers, SQL, and notebooks.
