Workflow Orchestration

Apache Airflow

Apache Airflow is a platform to programmatically author, schedule, and monitor workflows in Python. It is used for managing data pipelines and can integrate with other tools like Spark, Hive, and Kudu.

The open-ended nature of the tool is great for small ETL jobs. The extensive providers support gives hooks to all major platforms and data tools.

We cannot talk about Airflow without mentioning its community support. The slack channel is always buzzing with questions and people helping each other out. The documentation gets better every day. 9K+ questions of Stackoverflow and a tool that's forever improving.

• Workflow management

• Small ETL jobs

• One-off/schedule-based tasks

• Backups

@dag(
    start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
    catchup=False
)
def example_dag_decorator(email: str = "example@example.com"):
    get_ip = GetRequestOperator(task_id="get_ip", url="http://httpbin.org/get")

    @task(multiple_outputs=True)
    def send_email(raw_json: dict[str, Any]) -> dict[str, str]:
        # write logic here
        pass

    email_info = send_email(get_ip.output)

example_dag = example_dag_decorator()

Apache Oozie

Apache Oozie is a workflow orchestrator similar to Airflow written in Java and particularly meant to manage Hadoop jobs and support MapReduce, Pig, and Hive.

In Comparision with Airflow the Workflow Automation market, Apache Airflow has a 33.72% market share in comparison to Apache Oozie’s 5.93%

Apache Luigi

Apache Luigi developed by Spotify is a Python-based workflow management system that is used for scheduling and managing multi-node data pipelines. It is simple to use and is suited for small to medium-sized data workflows.

Though by definition Luigi sounds like Airflow, Luigi is based on pipelines of tasks that share input and output information and is target-based. Luigi does not come with built-in triggering, and you still need to rely on something like crontab to trigger workflows periodically.

Workflows in Luigi are made of Tasks and Targets. Tasks are classes with well-defined input, processing and output. Targets are data sources or destinations.

class NewTask(luigi.Task):

    def output(self):
        return luigi.LocalTarget("data/output_two.txt")

    def requires(self):
        return OldTask()

    def run(self):
        with self.input().open("r") as input_file:
            line = input_file.read()

        with self.output().open("w") as output_file:
            decorated_line = "My "+line
            output_file.write(decorated_line)

Data-Processing/Analytics

Apache Spark

6 Lakh records imagine loading that into a spreadsheet or pandas dataframe to perform some transformation. Impossible unless you're fine with your system giving up on you. We need a tool that can smartly chunk these data and process them independently

Apache Spark is a fast and general-purpose engine for large-scale data processing. It is designed to process data in parallel and can handle batch, streaming, and machine-learning workloads. It supports multiple languages like Python, Java, R.

The main advantage of Spark is that it provides a resilient distributed dataset (RDD)(More like a Pandas dataframe), which is a collection of elements partitioned across the nodes of the cluster that can be operated on in parallel

Spark Streaming is an extension of the core Spark API that allows data engineers and data scientists to process real-time data from various sources

url = \
  "jdbc:mysql://yourIP:yourPort/test?user=yourUsername;password=yourPassword"
df = sqlContext \
  .read \
  .format("jdbc") \
  .option("url", url) \
  .option("dbtable", "people") \
  .load()

# Looks the schema of this DataFrame.
df.printSchema()
countsByAge = df.groupBy("age").count()

countsByAge.write.format("json").save("s3a://...")

Apache Beam

Apache beam is a streaming ETL tool that can handle batches as a special case of stream processing

A beam like every other orchestrator tool consists of Pipelines. But what makes it different is it each node in the pipeline can be a PCollection(Data) node or PTransform(function) node. They are also great for both batch and stream-based data processing.

The best part about Apache Beam is the multi-language support of Java, Python, Go, and Scala.

Apache Flink

Apache Flink is an open-source, distributed stream processing framework for real-time, event-driven data processing. It supports batch and stream processing and is designed for scalability, fault tolerance, and low latency.

Flink is great for real-time reporting, fraud detection, stream analytics

Apache Storm

Apache Storm is a free and open-source distributed real-time computation system. is great for real-time analytics, online machine learning, continuous computation, distributed RPC, ETL, and more.

Apache Storm is fast: a benchmark clocked it at over a million tuples processed per second per node. It is scalable, fault-tolerant guarantees your data will be processed, and is easy to set up and operate.

Apache Storm integrates with the queueing and database technologies you already use. Apache Storm conducts real-time computation using topology and it gets feed into a cluster where the master node distributes the code among worker nodes that execute it. In topology data is passed in between spouts that spit out data streams as immutable sets of key-value pairs.

Apache Samza

Apache Samza developed by Linkedin is a distributed stream processing framework that is designed for real-time event processing. It supports stateful stream processing and integrates with Apache Kafka for its messaging layer.

While Apache Beam lets us define the workflow definition, Samza can be the actual executor of those transformations.

Samza is a newer ecosystem compared to Storm and is specifically designed to work well for Apache Kafka. Apache Samza streams by processing messages as they come in one at a time. The streams get divided into partitions that are ordered sequences where each has a unique ID. It supports batching and is typically used with Hadoop's YARN and Apache Kafka.

Apache Kudu

Apache Kudu is an open-source storage system for fast analytics on fast data. It supports low latency, random access, and efficient compression and is suitable for use cases such as online serving, real-time analytics, and offline batch processing.

Visualization

Apache Superset

Apache Superset is a modern, enterprise-ready business intelligence web application. It is fast, lightweight, intuitive, and loaded with options that make it easy for users of all skill sets to explore and visualize their data.

Messaging/Queuing

Apache Kafka

The moment you hear stream processing in the data world you cannot ignore Apache Kafka. The event streaming platform is used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

Apache Kafka is scalable, durable, reliable and high performant which makes it a perfect tool of choice for large volumes of data streaming.

Common usecases for Kafka includes

• Internet of Things and stream processing

• Log aggregation

• Large-scale messaging

• Customer activity tracking

• Operational alerting and reporting

Storage

Apache Hive

Apache Hive is a data warehousing and SQL-like query language for big data. It provides an interface to analyze structured data stored in the Hadoop file system (HDFS) using SQL-like syntax.

Apache Parquet

Apache Parquet is a columnar storage format for Hadoop that provides efficient data compression and encoding algorithms. It is optimized for interactive and batch analytics and supports advanced features like predicate pushdown.

Apache Iceberg

Iceberg is a high-performance format for huge analytic tables. Iceberg brings the reliability and simplicity of SQL tables to big data while making it possible for engines like Spark, Trino, Flink, Presto, Hive and Impala to safely work with the same tables, at the same time.

As of Dec 2022, these tables can also be used by BigQuery for large data analytics

Yes, Apache has an overwhelming number of tools to pick from for your data infrastructure. Choose the ones that fit you the best based on use case, community support, maintenance and the skill level of your team.

Let's discuss your data infrastructure needs. Schedule a free call today

The main difference between ETL and ELT is the order in which the data is processed.

In ETL, data is extracted from one or more sources, transformed to fit the target system's schema, and then loaded into the target system.

In ELT, data is extracted from one or more sources and loaded into the target system before it is transformed to fit the target system's schema.

When to use ETL vs. ELT?

It's worth noting that ETL and ELT are not mutually exclusive and can be used together in a hybrid approach.

A hybrid approach to ETL and ELT can be implemented in a few different ways, depending on the specific requirements of the organization and the data integration project.

Real-time vs. batch processing

If real-time data processing is a priority, ELT may be a better option as it allows for near-instant data loading and processing. ETL, on the other hand, is better suited for batch processing of data on a schedule.

Data volume and complexity

ELT is well suited for handling large volumes of data, such as data from social media or IoT devices, and can handle data in various formats and structures. ETL is better suited for handling structured data from traditional sources, such as relational databases.

Data transformation

ELT is designed to handle simple data transformations, such as data cleaning and validation, while ETL is designed to handle more complex data transformations, such as data mapping and aggregation.

Data quality and governance

ETL is typically used when data quality and governance is a top priority as it allows for the preprocessing of data before it is loaded into the target system. ELT, on the other hand, may not have the same level of data quality and governance capabilities.

Cost and resources

ELT is generally less expensive and requires fewer resources than ETL. ELT uses the processing power of the target system to perform data transformations, while ETL requires a separate system for data processing.

Hybrid approach

1. ELT for real-time data and ETL for historical data: In this approach, ELT is used to handle real-time data streams and load them into the target system in near real time. ETL is then used to process historical data in batches and load it into the target system on a schedule. This allows organizations to take advantage of the real-time capabilities of ELT while still maintaining a historical record of their data.

2. ELT for simple transformations and ETL for complex transformations: In this approach, ELT is used to handle simple data transformations, such as data cleaning and validation, while ETL is used to handle more complex data transformations, such as data mapping and aggregation. This allows organizations to take advantage of the performance benefits of ELT while still being able to handle more complex data transformation needs.

3. ELT for big data and ETL for structured data: In this approach, ELT is used to handle big data, such as data from social media or IoT devices, and load it into the target system. ETL is then used to handle structured data, such as data from a relational database, and load it into the target system. This allows organizations to take advantage of the scalability and flexibility of ELT for big data while still being able to handle structured data more traditionally.

Tools

ETL

• Informatica PowerCenter

• IBM DataStage

• Microsoft SQL Server Integration Services (SSIS)

ELT

• Apache NiFi

• Apache Kafka

• AWS Glue

The tools mentioned here are pure ELT or ETL tools. You can also use General purposes tools like Airflow/Prefect/Spark to achieve ELT/ETL operations.

While this is a good thing, there are no bounds in which the system can or cannot be used. Resulting in wasting a lot of time in scaling, testing and debugging when things aren't set properly.

You can also use this post to convince your management Why is it taking so much time? why do you have to set standards and ground rules for your data projects? Jarek approved and recognized this blog post in Airflow's slack channel.

The Dev & Ops Team

Using Airflow for your Data team means you need a good mix of Python and DevOps skills.

The Python Devs would take care of pipelining and debugging the code that runs on the pipelines whereas the Ops team ensures the infrastructure stays intact, is easy to use and debug as per need.

A badly written pipeline code can be resource hungry and the Ops teams will have no control over it. Similarly, a resource-constrained breaking infrastructure doesn't give a smooth development experience for DAG authors.

It is important for these teams to work hand in hand but also knows where their lines of responsibilities are.

Infrastructure

IaC

You will always spin up a new airflow environment. By default, any data team will need at least 3 Airflow environments - Dev, Test, and Prod. Factoring in the teams and projects you are going to need more environments. If not today you might need it 6 months down the line.

Code is the ultimate truth.

The maximum automation I've seen is bash scripts stitched together. That only gives you info about Airflow configs and not the infra configs. EC2 or EKS? Loadbalancer? DNS mapping?

There is a good chance that the DevOps engineer who set this up might move companies or to a different project. All that the teams are left with are a bunch of bash scripts and commands to run.

Instead of investing time in documenting the commands, you can spend that time automating it with Terraform or similar IaaC tools

CI/CD Pipeline

Often you will be working with multiple dag authors. It is important to set up a system that provides them with a quick turnaround time to test and deploy their features

Environments, where dag authors are doing a git pull on the server directly, aren't going to cut it. There were also cases where you have dag authors fighting over the package versions they want for their PythonOperator to run.

There are three ways to get your DAGs into the Airflow instance

• Building them into the instance

• Mounting from S3

• Using gitsync

Doesn't matter what method you choose the underlying idea is to have these update your Airflow instance when the developer is ready to do so.

Not Customizing

Airflow comes with a tonne of tools, tweaks and configs. One pattern I see commonly is following an infra guideline blindly and not customizing it to your needs.

Here are a few things to take into account

• How many environments would you need?

• How many DAGs are going to run?

• What infra tools work the best for our team?

• What are our peak schedules, will our resource allocation withstand it?

• What is it going to cost?

Even if you are building a POC keep all of these things in mind. Some teams are expected to go from POC to prod in a matter of weeks

Over Optimizing

• You don't need a complicated infrastructure unless you're running 100s of dags with 100s of tasks. Pick a pattern and scale on demand.

• Kubernetes lets you scale the best. You can run 3 pods one for each component and scale it as you need.

• But there are also cases where Airflow instances run happily on consolidated EC2 for years

• Pick your poison based on your current and projected needs.

Versions

Using old Airflow Version

Airflow made 12 releases in 2022(1 release per month). That's crazy! I know, It's hard to keep up. But if you are starting fresh.

• Start with the latest available version.

• Keep in mind you might need to upgrade to new version sooner

Perks of using the latest airflow version

• It gets you the support you require

• Incorporate the bug fixes and new features that might fit your use case

What about new bugs?

• That's a part of using opensource technologies. You get to jump in and help the community fix them.

• But if you're cautious, maybe try 2 versions down. That way, some show-stopping bugs can be found on Github issues.

Following old Docs/Blogs

Airflow has been around for years now. There are a variety of Guides, tools, and blogs out there. Some of them are so old that it doesn't do justice to the overhaul Airflow got in 2.0+.

One developer created a whole plugin following a blog, but what they should've used is a provider. The support for plugins as a provider was removed after 1.10.x versions. The result? 2 more weeks of development and testing

Even if you're using the Airflow docs page, ensure you are on the right version. The same goes for providers. Providers are also constantly updated and optimized to support the latest features of the integrating system.

Not Freezing Library Requirements

Those pip libraries with no specific version tagged are timebombs waiting to tick when you need the environment up and running 6 months down the line. Always enforce developers to tag all requirements with their respective versions.

Airflow Components

Running scheduler as a part of the webserver

The scheduler is the heart and brain of Airflow. The practice to run Scheduler as a part of the webserver was propagated by Puckel Docker images and still sticks around in most code bases.

The script suggests you run both webserver and scheduler on the same machine for LocalExecutor and SequentialExecutor This might still work well for small scale, as you add more dags, the scheduler will need more memory and computation to spin up your tasks, irrespective of the Executor you use.

Right Executor for the Job

Airflow provides a variety of executors. Choosing the right one depends on a lot of factors scale, team size, and dag use case.

I have a separate post on this checkout Pros and cons of using different Airflow executor

Database

• Using sqlite.db. It's not production friendly. The 1st thing I do with any Airflow installation is to change it to Postgres. (or MySQL)

• Running Database in a docker container or Kubernetes pod is not advised. Tutorials advise them to get things up and have consistent results for the reader. A quick alternative is to set up an RDS or managed database and use it. Services like supabase give you readily available Postgres

Not Persisting Logging

You need to track two kinds of Logs for Airflow. The logs of Airflow components and The logs from running your pipelining code. Not having access to either one of those will create problems while debugging down the line.

"Our nightly run failed and we have no idea why" is how it starts, on investigating we will find that the scheduler pod that was responsible for those spin ups evicted with all the logs associated with it.

Large-Data processing

Apache Airflow being the famous Opensource general-purpose data orchestration tool, lets you do a variety of things and gives room for lots of customization.

Yes, it is general-purpose. Yes, you can do a variety of things. But processing GBs or TBs of data is not Airflow's game. Often teams use Pandas to write small transformations or run queries, but that practice can propagate to large datasets as well.

The workaround

• Use the right tools for the right job like spark or Hadoop

• Remember Airflow is not an ETL tool, it is a Trigger -> Track -> Notify tool

Top-Level Python Code

Airflow DAGs are Python-first. You can write any kind of Python code to generate DAGs

• Reading a DB or a large JSON file to load the config

• Large operations outside the context of Airflow dags

While these may look like a fair thing to do, these pop up as huge blockers as the infrastructure scale.

Celery is used for running distributed asynchronous python tasks.

Hence, Celery Executor has been a part of Airflow for a long time, even before Kubernetes. With Celery Executors, you must set a specific number of worker instances.

Pros

1. In Airflow, you can specify the number of tasks that can run in a given worker. It is a good idea if you have a predictable number of tasks to run on a given worker.

2. Celery manages the workers. In case of a failure, Celery spins up a new one.

Cons

1. Celery needs RabbitMQ/Redis for queuing the task, an added dependency.

2. Multiple tasks run on the same worker, which means one task can clog all the resources available for another.

3. Running multiple workers all the time might lead to wasting resources when there isn't much to process.

Kubernetes Executor

KubernetesExecutor is where Airflow spins up a new pod to run an Airflow task.

Pros

Unlike Celery's executor, the advantage is you don't have a bunch of workers always running. KubernetesExecutor is on-demand, thereby reducing cost.

KuberentesExecutor lets you specify the resources required for each task giving you more control.

Cons

One downside of Kubernetes executor can be the time it takes to spin up the pod but compared to the advantages, it can be close to null.

Setting up the infrastructure can be complicated if you don't have the Kubernetes skillset in your team.

Kubernetes Celery Executor

KubernetesCeleryExecutor brings the best of both Celery and Kubernetes worlds and also the worst. It is a good idea to use them only when they are necessary.

1. You have a few resource-hungry tasks which need high resource and isolation so that it doesn't clog other tasks.

2. You have a mixture of peak times tasks with longer queues that can be run using Kubernetes and other predictable tasks with predictable resources which Kubernetes can handle.

In this post, you have seen how to utilize different Airflow executors to improve your tasks' performance while simultaneously optimizing the costs.

Got Airflow issues? I would be happy to assist you.

Schedule a free Discovery call today

tl;dr get the bash script

1. Have Python installed in your system, 3.8+

2. Create a folder

mkdir -p "/Users/$(whoami)/projects/airflow-local"
export AIRFLOW_HOME="/Users/$(whoami)/projects/airflow-local"

1. cd airflow-local

2. Set airflow version AIRFLOW_VERSION=2.4.3

3. Set airflow version PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"

4. Set constraints version CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"

5. Create virtualenv and install airflow

cd "/Users/$(whoami)/projects/airflow-local"
python -m venv venv
source venv/bin/activate
pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"

1. mkdir -p "${AIRFLOW_HOME}/dags"

2. Run airflow with airflow standalone

This will run, creating an SQLite DB. This is not really production friendly. You can use them to run basic pipelines and playgrounds. We can also make our local environment production friendly. We will see that in a bit.

The Airflow Project

1. standalone_admin_password.txt has the password for your local airflow password is admin

2. dags the folder is where you add your dags, a.k.a pipelines

3. logs the folder will have your logs

Using LocalExecutor

Airflow, by default, used SequentialExecutor this is not great for production-level systems. When exploring Airflow, it is a good idea to use LocalExecutor along with Postgres or MySQL right away.

Stop the Airflow instance. Update the airflow.cfg with the following configs. Replace sql_alchemy_conn with respective DB credentials

[core]
load_examples = False
executor=LocalExecutor

[database]
sql_alchemy_conn = postgresql://<pg-user>:<pg-password>@<host>:<port:5423>/<db-name>

If you use Postgres, You will need psycopg the library, which you can download using the following command

pip install "apache-airflow[postgres]"

Sample Dag

Copy the example dag from the Airflow repo and place it under the dags folder

Restart Airflow

airflow standalone

Airflow has a steep learning curve. I can help you adopt Airflow for your Data engineering pipeline and your team's ecosystem. Schedule a free call today.