What Is the Significance of Amazon Managed Workflows for Apache Airflow (MWAA)

Amazon Managed Workflows for Apache Airflow (MWAA) is a managed orchestration service for Apache Airflow. MWAA simplifies the process of setting up and operating end-to-end data pipelines in the cloud at scale. Apache Airflow is an open-source tool used to programmatically author, schedule, and monitor sequences of processes and tasks referred to as “workflows.”

With Amazon MWAA, one can use Airflow and Python to create workflows without having to manage the underlying infrastructure for scalability, availability, and security. Amazon MWAA auto-scales its workflow execution capacity to meet your requirements and this can be integrated with AWS security services to help marketers with fast and secure access to your data.

Features of Amazon Managed Workflows for Apache Airflow (MWAA)

1. Setting Up Amazon Airflow

One can quickly set up Apache Airflow by choosing Apache Airflow version when he/she creates an Amazon MWAA environment. Amazon MWAA sets up Apache Airflow for anyone who wishes to use the same Apache Airflow user interface and open-source code that one can download on the internet.

2. Auto-Scaling

Amazon MWAA can be automatically scaled up by setting up the minimum and maximum number of Workers in the user’s environment. The auto-scaling component of MWAA can be used to accomplish the demand of the end-users until the demand reaches the maximum number of Workers that the end-user defined.

3. In-Built Authentication

The end-user can enable role-based authentication and authorization for their Apache Airflow Web server by defining the access control policies in AWS Identity and Access Management (IAM). The Apache Airflow Workers assume these policies for secure access to AWS services.

4. Built-in Security

The Apache Airflow Workers and Scheduler(s) run in Amazon MWAA’s Amazon VPC. Data can also be automatically encrypted using AWS Key Management Service, so your environment is secure by default.

5. Public or private access modes

Marketers can access their Apache Airflow Web server using a private, or public access mode. The Public network access mode uses a VPC endpoint for your Apache Airflow Web server that is accessible over the Internet. With the Private network access mode, people use a VPC endpoint for their Apache Airflow Web Server that is accessible in their VPC.

In both cases, access mode uses a VPC endpoint for their Apache Airflow Web server that is accessible in your VPC. In both cases, access for your Apache Airflow users is controlled by the access control policy that users define in AWS Identity and Access Management (IAM), and AWS SSO.

6. Streamlined upgrades and patches

Amazon MWAA provides a new version of Apache Airflow periodically. The images for these versions will be updated and patched by the Amazon MWAA team.

7. Workflow monitoring

Amazon CloudWatch can be used to view Apache Airflow metrics and to identify Apache Airflow task delays or workflow errors without the need for additional third-party tools. Amazon MWAA automatically sends environment metrics and if enabled Apache Airflow logs to CloudWatch.

8. AWS integration

Amazon MWAA supports open-source integrations with Amazon Athena, AWS Batch, Amazon CloudWatch, Amazon DynamoDB, AWS DataSync, Amazon EMR, AWS Fargate, Amazon Lambda, Amazon Redshift, Amazon SQS, Amazon SNS, Amazon SageMaker, and Amazon SageMaker, and Amazon S3 as well as hundreds of in-built and community-created operators and sensors.

9. Worker fleets

Amazon MWAA offers support for using containers to scale the worker fleet on-demand and reduce scheduler outages using Amazon ECS on AWS Fargate. Operators that invoke tasks on Amazon ECS containers and Kubernetes operators that create and run pods on the Kubernetes cluster are sustained.

The MWAA Architecture

All the components contained in the outer box appear as a single Amazon MWAA environment in users’ accounts. The Apache Airflow Scheduler and Workers are AWS Fargate (Fargate) containers that connect to the private subnets in Amazon VPC for your environment. Each environment has its own Apache Airflow Metabase managed by AWS that is accessible to the Scheduler and Workers Fargate containers via a privately-secured VPC endpoint.

Amazon CloudWatch, Amazon S3, Amazon SQS, Amazon ECR, and Amazon KMS are separate from Amazon MWAA and should be accessible from the Apache Airflow Scheduler(s) and Workers in the Fargate containers.

The Apache Airflow Web server can be accessed either over the Internet by selecting the Public network Apache Airflow access mode, or within your VPC by selecting the Private Network Apache Airflow access mode. In both cases, access for your Apache Airflow users is controlled by the access control policy you define in AWS Identity and controlled by the access control policy you define in AWS Identity and Access Management (IAM).

Integration

The active and growing Apache Airflow open-source community provides operators (plugins that simplify connections to services) for Apache Airflow to integrate with AWS services. This includes services such as Amazon S3, Amazon Redshift, Amazon EMR, AWS Batch, and Amazon SageMaker, as well as services on other cloud platforms.

Region Availability

Amazon MWAA is available in the following AWS Regions.

• Europe (Stockholm) – eu-north-1
• Europe (Frankfurt) – eu-central-1
• Europe (Ireland) – eu-west-1
• Europe (London) – eu-west-2
• Europe (Paris) – eu-west-3
• Asia Pacific (Mumbai) – ap-south-1
• Asia Pacific (Singapore) – ap-southeast-1
• Asia Pacific (Sydney) – ap-southeast-2
• Asia Pacific (Tokyo) – ap-northeast-1
• Asia Pacific (Seoul) – ap-northeast-2
• US East (N. Virginia) – us-east-1
• US East (Ohio) – us-east-2
• US West (Oregon) – us-west-2
• Canada (Central) – ca-central-1
• South America (S”o Paulo) – sa-east-1

Supported versions

Amazon MWAA supports multiple versions of Apache Airflow.

What’s Next?

Marketers can get started quickly with a single AWS CloudFormation template that creates an Amazon S3 bucket for your Airflow DAGs and supporting files, and Amazon VPC with public routing, and an Amazon MWAA environment in Quickstart tutorial for Amazon Managed Workflows for Apache Airflow (MWAA).

Marketers can get started incrementally by creating an Amazon SS3 bucket for your Airflow DAGs and supporting files, choosing from one of three Amazon VPC networking options, and creating an Amazon MWAA environment in getting started with Amazon Managed Workflows for Apache Airflow (MWAA).

Top Benefits of Using Amazon Managed Workflows for Apache Airflow

MWAA is a great tool for data processing and offers the following benefits to marketers:

1. DAGs

DAGs allow users to set up workflows and to set up a sequence of operations that can be individually retried on failure and restarted where the operation failed. DAGs provide a nice abstraction for a wide series of operations.

2. Programmatic Workflow Management

MWAA provides an impeccable way for marketers to set up programmatic workflows, Tasks for instance can be generated on the fly with a dag. The users can create complex dynamic workflows and can set up an instance based on variables or connections defined with the UI of MWAA.

3. Automate Your Queries, Python Code or Jupyter Notebook

Airflow has a lot of operators set up to run code. Airflow has an operator for most databases and being set up in python it has a PythonOperator that allows quick portioning of python code for production.

Papermill is an extension to a Jupyter notebook that allows parametrization and execution of notebooks, it is supported through airflow PapermillOperator. Netflix notably has suggested a combination of airflow and papermill automate and deploy notebooks in production:

Example of a parameterized ETL script in Jupyter Notebooks

A parameterized Jupyter notebook

4. Managing the Task Dependency

Airflow is extremely good at managing different sorts of dependencies, be it a task completion, dag runs status, and file or partition presence through a specific sensor. Airflow handles the task dependency concept such as branching.

5. Extendable Model

MWAA is fully extendable through the development of custom sensors, hooks, and operators. Airflow notably benefits from a large amount of community-contributed operators.

6. Monitoring and management interface

Airflow provides a monitoring and managing interface where it is possible to have a quick overview of the status of different tasks as well as have the possibility to have a quick overview of the status of the different tasks, as well as have the possibility to trigger and clear tasks or DAGs runs.

7. Retry policy built-in

Airflow has an in-built policy for auto-reply, configurable through:

• retries: number of retries before failing the task
• retry_delays: (time delta) delay between retries
• retry_exponential_backoff: (Boolean) to set up and exponential backoff between retries
• max_retry_delay: Maximum delay (time delta) between retries

These arguments can be passed through the context to any operator, as they are supported by the BaseOperator class.

8. Easy Interface to Interact with Logs

MWAA allows easy access to the logs of each of the different tasks run through the web UI that making it easy to debug tasks in production.

9. Rest API

The Rest APIs of Airflow allows marketers to create workflows from external sources, and to be data product on top of it.

Architecture Diagram for using HTTP post to an endpoint to trigger an Airflow DAG that automates the spins up or tear down of a Dataproc cluster to run a Spark job to enhance data and writes the enhanced data to BigQuery.

The rest of API allows marketers to use the same paradigm used to build pipelines, to create asynchronous workflows, such as custom machine learning training operations.

10. Alerting System

MWAA provides a default alerting system on tasks that failed, email is the default, but alerting through slack can be set up using a callback and the slack operator.

Wrap Up

AWS released MWAA in November 2020, and we got an opportunity to explore and evaluate the managed service if it can curate most of our needs and solve some of the challenges. Leveraging MWAA as a service is easy and autoscaling for worker nodes is available in MWAA. Airflow configuration in MWAA is easy to manage and update. One can simply update the config from MWAA console and see the changes reflected in the environment. With MWAA users can also easily integrate with other AWS services like EMR when services are within the same VPC.