Automation plays a key role in increasing productivity and work efficiency across industries. Recently, with the flood of new automation and management tools on the market, it can be difficult for users to choose the best tool for their use case from the pool of new-age technologies.
What is Airflow?
Airflow allows organizations to write workflows as directed acyclic graphs (dags) in the standard Python programming language, ensuring that anyone with minimal knowledge of the language can deploy one. Each DAG contains nodes and connectors through which nodes are connected to other nodes to generate a dependency tree. Airflow helps organizations schedule their tasks by specifying the schedule and frequency of flows. Airflow also provides an interactive interface, and a bunch of different tools to monitor the workflow in real time.
Apache Airflow is very popular among organizations dealing with large amounts of data collection, processing, and analysis. Every IT specialist has a different job or workflow to perform, from collecting data from other sources to processing it, uploading it and creating reports. Every day there are many tasks that need to be performed manually by experts. Airflow triggers automated workflows that reduce the time and effort required to collect data from various sources, process it, upload it, and finally create reports.
characteristic:
- open source
- Dynamic Integration: Airflow uses the Python programming language to write workflows as dags. This allows Airflow to integrate with multiple operators, hooks and connectors to generate dynamic pipelines. It can also easily integrate with other platforms such as Amazon AWS, Microsoft Azure, Google Cloud, etc.
- Customizability: Airflow supports customization, which allows users to design their own custom operators, actuators, and hooks. You can also extend the library as needed to conform to your desired level of abstraction.
- Rich User Interface
- Scalable: Airflow is highly scalable and designed to support multiple interdependent workflows simultaneously.
What is Argo?
Argo is an open source workflow engine for orchestrating tasks on Kubernetes. Argo, introduced by Applatex, allows you to create and run advanced workflows entirely on a Kubernetes cluster. Argo workflows are built on top of Kubernetes, and each task runs as a separate Kubernetes pod. Many well-known organizations in the industry use Argo workflows for ML (machine learning), ETL (extract, transform, load), data processing, and CI/CD pipelines.
Argo is basically an extension of Kubernetes, therefore, it is installed using Kubernetes. Argo workflows allow organizations to define their tasks as dags using YAML. Argo provides native workflow archiving for auditing, Cron workflows for scheduling workflows, and a full-featured REST API. Argo offers a bunch of killer features that set it apart from its peers, let's take a look at them.
characteristic:
- Open source: cloud native cncf project
- Native integration: Argo provides native artifact support for downloading, transferring and uploading files at runtime. It supports any S3-compatible artifact repository, such as AWS, GCS, Alibaba Cloud OSS, HTTP, Git, Raw, and Minio.
- Scalability: Argo workflows have a robust retry mechanism for high reliability and scalability. It is capable of managing thousands of pods and workflows in parallel.
- Customizability: Argo is highly customizable, supporting templating and composability to create and reuse workflows.
- Powerful User Interface: Argo provides a full-featured user interface (UI) that is easy to use. Argo workflow v3.0 UI also supports Argo events, which is more robust and reliable. It has embeddable widgets and a new workflow log viewer.
Argo vs Airflow
Both Airflow and Argo allow you to define workflows as dags, but there are some differences in how the two platforms operate that are critical to choosing the one that suits your needs.
workflow language
The first key difference between Argo Workflow and Airflow is the programming language used to define the dags. As discussed in the previous sections, workflows allow organizations to define workflows as dags in the standard Python programming language. Airflow runs every task in the Python ecosystem. A basic understanding of Python is enough to write code and simplify complex pipelines and workflows. Its python-based API is one of the main reasons for its widespread popularity and adaptability.
Argo likewise allows organizations to define their workflows as dags, but unlike Airflow, these definitions are written in YAML rather than Python. Argo runs each task as a Kubernetes pod. However, workflows are often complex, and complex processes are best expressed in code rather than configuration languages like YAML.
task scheduling
Airflow is good at running tasks on schedule, and it has a fault-tolerant scheduler that recognizes when schedules are missed. Unfortunately, the scheduler cannot run in a high-availability or busy setup because it is the single point of failure for the system. However, the Airflow scheduler takes up to 5 minutes to rescan the DAG file for updates and perform a state loop to schedule new tasks. Therefore, it does not support low-latency scheduling.
Argo is also good at running scheduled tasks, but if the controller faces an interruption within the scheduled interval, it can only reschedule 1 missed task. It will reschedule a missed task until the startingdeadlinesecseconds interval is set. However, if the interruption lasts longer than startingdeadlineseconds, no tasks will be rescheduled. However, the Argo scheduler receives events from Kubernetes and is able to respond immediately to new workflows and state changes without the need for a state loop, making it ideal for low-latency scheduling.
scalability
Airflow supports horizontal scalability and has the ability to run multiple schedulers concurrently. Speaking of tasks, "Airflow" relies on a dedicated pool of workers to perform tasks. Therefore, the maximum task parallelism is equal to the number of active workers.
Argo runs each task as a separate Kubernetes pod, so it is capable of managing thousands of pods and workflows in parallel. Unlike Airflow, in Argo the parallelism of workflows is not limited by a fixed number of workers. Therefore, it is best suited for jobs with sequential and parallel step dependencies.
third party integration
Airflow uses the Python programming language to write workflows as dags. This allows Airflow to connect to almost any third-party system. Airflow also has its own community-supported database, cloud service, computing cluster and other operation libraries.
Argo is an open source container and has no prepackaged operators to interface with third-party systems. However, it supports any S3-compatible Artifact Repository, such as AWS, GCS, Alibaba Cloud OSS, HTTP, etc., to download, transfer and upload your files at runtime.
support workflow
Airflow dags are static, once defined they cannot add or modify steps at runtime. Airflow only runs dags on a schedule, so external systems cannot trigger workflows to run. This means that 2 DAG runs cannot start at the same time. Most importantly, Airflow assumes that all DAGs are independent, so it has no first-level mechanism to pass parameters to DAG runs.
In Argo, DAG definitions can be dynamically created for each run of a workflow. It can map tasks on a dynamically generated result list into parallel processing items. Argo workflow v3.0 also supports Argo Events, an agri-ecosystem project dedicated to automating event-driven workflows. Agro's parameter passing syntax allows you to pass input and output parameters at the task level, and input parameters at the workflow level.
Integration with k8s resources
Airflow has a Kubernetes operator that can be used to run pods as part of a workflow. However, it does not support creating other resources.
Argo is built on top of Kubernetes, and each task runs as a separate Kubernetes pod. Argo has a special support system for performing CRUD operations on Kubernetes objects such as pods and deployments.
Summarize
| characteristic | Argo | Airflow |
|---|---|---|
| workflow language | YAML | Python |
| low latency scheduling | Yes | No |
| high parallelism | Yes | No |
| third party integration | No | Yes |
| dynamic workflow | Yes | No |
| event-driven workflow | Yes | No |
| Parametric Workflow | Yes | No |
| K8s integration | Yes | No |
There is no silver bullet for deciding which tool is best. The choice depends largely on your use case, requirements and operating environment.
Both Argo and Airflow allow you to define tasks as dags, but airflow is more general, while Argo offers limited flexibility in interacting with third-party services. If you already use Kubernetes for most of your infrastructure, I recommend using Argo for your tasks. If your developers are more comfortable writing DAG definitions in Python rather than YAML, you might consider using Airflow.