ProgrammerHumor
thankYouColdplay
Discussion
Tl;Dr it for me
Orchestration is the automated configuration, coordination, deployment, development, and management of computer systems and software.
But what does that have to do with coldplay or that incident? Asking for a friend...
The company they worked for, created software that did the thing
Thank you random person
I thought it had to do with astronomy, disappointing
Although maybe I'm glad it wasn't about astronomy given the circumstances
So you are telling me that this concert affair scandal was orchestrated PR campaign to promote their platform?
The post is about people finding out about Airflow, thanks to Astronomer.
Lies, we all know there's no air-flow in space
There is airflow on Earth (and other planets). These planets are in space. So, actually, there is airflow in space.
No
That's some serious guerilla marketing
Apache AirFlow is an orchestration platform. Astronomer’s product is a platform built on top of AirFlow. Astronomer’s CEO was caught romantically holding their head of HR on the Jumbotron at a Coldplay concert.
Astronomer’s product is a platform built on top of AirFlow.
Does it add anything of value or is it just, you know, a scam?
Configuration, Deployment and management of the infrastructure needed to run airflow at scale is painful.
Astronomer makes it easy.
And that's when Deepspace comes in! It helps you set up the infrastructure needed to run Astronomer. Deepspace makes it easy!
And that's when Jupiter comes in! It helps you set up the infrastructure needed to run Deepspace. Jupiter makes it easy!
And that's when Galaxia comes in! It helps you set up the infrastructure needed to run Jupiter. Galaxia makes it easy!
And...
Sometimes I feel dread like I’m in some horror film.
I managed to set up Airflow for my company years ago. It was a pain in the ass, but now it runs smoothly, doesn’t require a lot of maintenance, and we don’t have to pay anybody for the service
So... a bunch of scripts?
A BUNCH OF SCRIPTS
In a cave?
So does AWS' MWAA service! ... Just kidding, that's the worst managed service offered by AWS in terms of configuration, usage, and documentation!
They're not the only ones offering the service. Google Cloud Composer is Google's managed Airflow service.
That escalated quickly from workflow management to HR drama. Nothing like mixing business with pleasure at a Coldplay show.
Awesome. That means I don't have to worry about configuring software anymore, right?
No, it means you have one more thing that requires a shitton of configuration.
Not at all, now you need to worry about another process on top of it.
Automatic development? If business execs knew about this earlier we wouldn't need developers to begin with!
Why the apparent periodic lack of interest in Airflow?
Weekends
Thanks. I forgot about those. I have been retired for too long, I guess.
in English pls /s
Just the next fancy words for "development pipeline".
CI/CD, DevOps, etc every SCRUM Master wants to pretend they reinvented shipping software
They did indeed reinvent it
Whether or not it's a valuable reinvention is debatable
Nothing to do with writing music for an orchestra then.
Why the apparent periodic lack of interest in Airflow?
Not learning container, orchestration, and microservice architectures in general during my time in college/early in my career was a big misstep for me because there's no way I'll be able to get experience with it now in my current position.
To clarify for this audience, Airflow primarily does r/DataEngineering or r/BusinessIntelligence orchestration, i.e., data pipeline orchestration.
The company the CEO ran just provides services based on an open source tool, Apache Airflow. No idea how but they aren't even profitable even after raising $360m and their main offering is AI wrapper on top of free software.
Coldplay box seats go for 18k.
And who knows how much that VP standing next to them got in hush money
Good. Astronomer represents everything I despise with tooling in tech. Layers of abstraction over layers of abstraction for very little value added.
On top of Airflow, which is just a pile of crap already IMO.
Lol at the weekly distribution of searches on airflow. The obvious weekend lol
Airflow is pretty dogwater as far as orchestration goes, not gonna lie.
It’s just easy to learn.
I’m gonna need to know whether dogwater is good or bad
Ngl, when you're thirsty at 3am with nothing else to drink nearby... It's still kinda dogwater
I guess it wouldn't be dogwater if it wasn't dogwater
The assumption is that a dog got to it first, otherwise it's just boiled ice
It is aladeen
What’s a better alternative? Genuine question, new to container orchestration
Airflow is not a container orchestration tool. That would be, for example, Kubernetes. If we are talking the infrastracture managment kind of orchestration, you would likely use Kubernetes with Argo CD, to handle lifecycle of your deployed applications.
Otherwise a more modern k8s native alternative to Airflow would be Prefect.
How is container orchestration different from infrastructure management kind of orchestration? Sorry if it's a dumb question, I'm pretty new to this.
They are kinda the same in the goal. It's the platform that differs.
Container orchestration is a kind of infrastructure orchestration, but on all infra orchestration is container orchestration.
Or a different way- It is a matter of what the orchestration tools supports. If it supports orchestrating containers, it's a containers orchestration tool.
And lastly, when deadling with kubernetes, you would usually use another orchestration tool to manager the kubernetes itself. Kubernetes itself is set of servers running kubernetes nodes, and this would usually. You want to be able to quickly spin up and update new k8s nodes, and therefore might need another orchestration tool to manage it.
You might use terraform/ansible and maybe SaltStack to manage the servers and to setup kubernetes nodes. Then you would rely on kubernetes and argocd to orchestrate the containers themselves.
Got it! Interesting how kubernetes itself needs to be managed by another orchestrator. Thanks for explaining so well.
Sooo orchestrating the orchestrator
From our experience, Dagster or Prefect, but that might just be because no-one knows how to use Airflow on the project
Seconding dagster, I'm replacing SO much custom scrappy glue with dagster jobs. I'm running it on my own auto scaling ECS and I get to just plop in existing containers with docker in docker to speed up migration.
Using polars and duckdb inside these jobs also turns it into an ETL weapon. The web GUI is lovely too.
Dagster is good.
If you are looking for an alternative to airflow, I recommend looking into dagster. It is newer compare to airflow but more user friendly.
github actions for my broke ass
Just do it yourself. You're a programmer, program.
This is 100% coming from someone who never knew what pre-airflow life was like
CRON isn't that bad. Sure Airflow is better, but under the hood it's all still just CRON. Now you just get a UI, logs and metrics.
If they were truly innovators they’d get rid of that horrible crontab syntax
Then they'd have to recreate the entire CRON engine from scratch. Why would they when it already works perfectly fine?
Are yall using airflow exclusively for single-task jobs or something? Just because you use cron to define the schedule doesn’t make it “just cron”
Airflow runs CRON under the hood, which is why I always say it's CRON with extra steps.
I already conceded it's much better, no idea why you're attacking me.
I dunno. I had to write an ELT pipeline. Initially, I thought Airflow was perfect for this, especially since GCP will host it. It had completely unreasonable resource requirements and latency for doing something as simple as kicking off some BQ jobs periodically. Plus the API is an inconsistent mess. Even crashed Python out of the box on Mac due to Python defaulting to fork multithreading. I rewrote it in Prefect. Still not good enough on latency (because Python), but the API was much improved. Then I gave up on the idea of using stupid giant frameworks just to kick off API requests periodically, so I rewrote it in Go. So yeah, I kind of think Airflow is worse than nothing, at least for programmers.
About 8 years ago I was comparing AirFlow vs NiFi. NiFi won out but that was largely because we had a deeper java skillset over Python. Have you checked out NiFi at all? I'm curious how it compares to Prefect.
Hamilton or prefect can be better. Airflow seems a bit on the heavy side too
All I want in life is decent serverless alternative
It is but its still better than many of the other popular alternatives
You guys just nerd-snipped me into looking into Airflow alternatives...
What would you suggest for orchestration then?
We heavily bought into Airflow 7 years ago, and it’s been 3 years that I regret it somewhat. We’re planning to migrate to prefect sooner than later
If there is airflow in my resume, should I switch job right now ?
I’m going to save you all a lot of time looking at the competition in this space: just go with temporal. All your favorite AI is built on top of it already.
I'm having BizTalk PTSD now. Useful tool, but creating orchestrations for Boeing was a big ass job lol
Airflow is the worst software on earth
Damn. Somebody crank up the D gain on that bad boy, he'll settle down.
okay so forgive me as I'm decently unfamiliar with programming, but what exactly does airflow do? I've heard a lot about productivity software / websites / apps that belong to this general class of "workflow managers" (personally I don't understand their utility), but apparently there's another layer? Programming workflow managers? but what advantage could something like airflow provide, if you could just keep all of your coding organized within files that you can just open and edit in an IDE? What more is there to organize? Is it for team management?
… You didn’t know about Airflow or Dagster?
What is Orchestration? Is it like a rehearsal for an Orchestra?
so orchestration is fancy word for CI/CD pipelines, like running ansible scripts from Jenkins?
Many kinds of Orchestration, from Maestros in an orchestra, to Controllers orchestrating Kubernetes resources, to bash scripts that have a specific workflow, to Airflow or Dagster for more generic stuff, to CI/CD pipelines with Jenkins/Buildkite/Github(Nektos) Runners/Gitlab Runners. To each problem its tools...
Ansible, Terraform, Nix, are declarative approaches to describe the Z state you want, pointing at resources [HEYO], and whatever state [HEYO] is, the tool engine has to figure out how to get the declarative state Z.
So, CI/CD is one of the ways to use the concept of Orchestration, not a fancy word for it, since Orchestration is much broader.
On the other side of the coin you have a concept that may make things much more complex, but has an easier time with reliability and keeping system-wide promises, which is a Choreography. Like the name suggests, each piece knows what to do at the right time. Complexity comes from the pieces being completely decoupled and only reacting to cues (e.g. events/calls). Reliability and keeping promises comes from the fact there isn't one single point of failure, like the Maestro having an heart attack and the play stops (tho, a ballet show also stops if one of the performers has an heart attack, but in computers, it usually isn't an issue)
Yeah. I understand that it's sorta "CI CD but also let's include much more functionality than just that"
No no, Airflow for CI/CD is like thinking Chocolate can be a full lunch meal. Sure it feeds you, but you have serious issues in a not distant future, and serious people will look at you with frowing eyes. Just because it's a kind of food/orchestration, don't mean it works for lunch/CICD.
Airflow was built for a completely different type of workload/requirements.
nah. i am not saying ci cd is the same thing. the principle is the same, you keep configurations and scripts and logic to automatize tasks. ci cd is just a basic or say, one aspect of this approach. obviously orchestration is supposed to be more, but that kind of simplification helps me to understand what is this about.
The triggering events don't usually overlap, CI/CD is more bound to events you have no way of escaping from in a domain's workflow.
Imagine like developing new recipes for cakes. Everytime you develop a recipe, you always have publish it to your blog, another bro makes a CI/CD pipeline to validate your recipe and include it in his restaurant menu. Hints are developing (triggering event), checking stuff (CI), adding it to menu (CD). This CI/CD pipeline will usually do only this kind of stuff, sure it can check if the restaurant can accept the menu (e.g. has all ingredients), compile the recipe to be valid to the Portuguese branch of the restaurant, the French branch, etc. But point being, it's focused on the the cake recipes.
The pipeline definition and pipe configs usually live along side the development of the cake recipes.
For Airflow, it's more like planning a wedding.
Albeit the same wedding is not recurrent, it's usually how Airflow is called. On schedules or directly called from another service (can be CI/CD even). The event types Airflow reacts to are very different from CI/CD based ones
You have a lot of moving parts, from tables, plate colors, DJ, send invites, process attendee responses, and the cake. You can create a solo repository for that wedding configuration/definition, you'll still have to send it to an Airflow server. But that definition can do an Octopus-like pull from several places to make the wedding a possibility. DJ can only be requested through their online website form. Florist services can only be requested by calling. To include the previous bros' Portuguese Branch Restaurant Cake, you have to walk into the restaurant.
In the end,
Nothing stops you from having CI/CD on an Airflow definition repository. Every time you change it, its validated and pushed to the airflow server, ready to be called. But CI/CD is focused on the small system/domain where it lives, while Airflow orchestrates stuff from several places. CI/CD lives in the domain repo (e.g. the .github/workflows folder), while an Airflow pipeline definition may be the domain repo itself. (Or a repo with several definitions).
Configuration may or may not live along side the definition, because inputs and secrets, can be given at the call time/event time.
Usually CI/CD doesn't take input, only the event type and where does it come from, and a bunch of other metadata, which you have to juggle to make an "input based" CI/CD pipeline. E.g. a pipeline only triggering on a cakes/* named branch.
thanks! that's a great analogy! I should look more into that, seems a very interesting thing to implement
I asked Gemini to make the comments more digestible:
Understanding Orchestration: From Maestros to Microservices
1 What is Orchestration?
At its core, Orchestration is the automated configuration, coordination, and management of complex computer systems and software. The goal is to streamline and automate a sequence of tasks to execute a larger, end-to-end process or workflow.
Think of it like a maestro conducting an orchestra. The maestro doesn't play any instruments but holds the complete musical score (the workflow) and signals to each musician (an individual service, script, or task) exactly when to play their part to create a cohesive symphony.
Technically, these workflows are often represented as a Directed Acyclic Graph (DAG)—a series of tasks where each task can trigger the next, but the overall flow moves in one direction without circular dependencies.
2 Types of Orchestration: Different Problems, Different Tools
While the principle is the same, the term "Orchestration" is applied in several different domains. The specific tools and patterns used are highly dependent on the problem being solved.
A. Simple Workflow Automation: The Shell Script
Analogy: A To-Do List. You write a list of sequential steps that must be done in order: 1. Go to the store, 2. Buy ingredients, 3. Cook dinner.
A bash or PowerShell script executes a series of commands one after another. It's highly imperative and typically manages tasks on a single machine.
B. CI/CD Pipelines: The Cake Recipe
Analogy: Publishing a Cake Recipe. Your CI/CD pipeline is the automated process that takes your new recipe from a draft to your restaurant's menu.
The workflow is triggered by a git push, runs tests (CI), and deploys the result (CD). The configuration (.github/workflows, Jenkinsfile) usually lives inside the same repository as the code it manages.
C. Data & Business Process
Orchestration: The Wedding Planner
Analogy: Planning a Wedding. A wedding planner coordinates many independent vendors: the caterer, the florist, the DJ, and the venue.
Tools like Apache Airflow excel here. The workflow is a master plan that interacts with each "vendor" (service/API) via their preferred method. The configuration (the DAG) often lives in its own dedicated repository.
D. Infrastructure & Resource Orchestration: The City Planner
Analogy: A City Planner. The planner holds a blueprint (declarative configuration) of the desired city. They give this blueprint to a team of controllers who make it a reality and fix any deviations.
This is less about a linear sequence and more about maintaining a desired state. Kubernetes is the quintessential example, using a self-healing control loop to match the actual state to the desired state defined in a YAML file.
3 Key Distinctions and Related Concepts
A. Orchestration vs. Choreography
Analogy: Dancers on a Stage. In a choreographed dance, there is no central conductor. Instead, each dancer knows their own part and reacts to cues (events) from other dancers.
| Aspect | Orchestration (Maestro) | Choreography (Dancers) |
|---|---|---|
| Control | Centralized | Decentralized |
| Communication | Point-to-point commands | Event-based (message bus) |
| Coupling | Tightly coupled | Loosely coupled |
| Visibility | Explicit workflow | Implicit workflow |
B. Pipeline vs. Orchestration
A Pipeline is a specific type of orchestration, one that is predominantly linear and sequential (like CI/CD). Orchestration is the broader concept that also includes complex, graph-based workflows (Airflow) and continuous control loops (Kubernetes). All pipelines are a form of orchestration, but not all orchestration is a pipeline.
C. Imperative vs. Declarative
- Imperative ("How"): You provide explicit, step-by-step commands.
Analogy: Giving someone turn-by-turn directions to your house.
- Declarative ("What"): You describe the desired final state, and the system figures out how to get there.
Analogy: Giving someone your home address and letting their GPS navigate.
4 Summary Table
| Type | Analogy | Primary Goal | Approach | Example Tools |
|---|---|---|---|---|
| Simple Workflow | To-Do List | Linear task automation | Imperative | Bash, PowerShell |
| CI/CD Pipeline | Cake Recipe | Automate software delivery | Mostly Imperative | Jenkins, GitHub Actions |
| Business Process | Wedding Planner | Coordinate cross-system workflows | DAG-based | Apache Airflow, Dagster |
| Resource Mgmt | City Planner | Maintain infrastructure state | Declarative | Kubernetes, Terraform |
5 Real-World Industry Examples
CI/CD Pipeline at Spotify: When a developer pushes code, a CI/CD pipeline automatically tests it, deploys it to a small percentage of users (a canary release), monitors for errors, and gradually rolls it out to all users.
Business Process Orchestration at an E-commerce Company: A company like Wayfair uses Apache Airflow for a nightly workflow that pulls sales data, gets user logs, joins them in a data warehouse, generates a sales report, and emails it to the finance department.
Resource Orchestration at Slack: The entire Slack application runs on Kubernetes. If traffic surges, Kubernetes automatically scales up services. If a server fails, Kubernetes automatically moves the workload to a healthy server with no manual intervention.
6 Conclusion
"Orchestration" is a broad and powerful concept, not just a fancy word for CI/CD. While a CI/CD pipeline is one specific type of orchestration, the principles extend to coordinating complex data flows and maintaining the state of entire infrastructure clusters. Understanding the distinctions between patterns like orchestration vs. choreography and imperative vs. declarative approaches is crucial for choosing the right "maestro" for the job.
See, one more reason to use airflow