ProgrammerHumor
thankYouColdplay
Discussion
Tl;Dr it for me
Orchestration is the automated configuration, coordination, deployment, development, and management of computer systems and software.
But what does that have to do with coldplay or that incident? Asking for a friend...
The company they worked for, created software that did the thing
Thank you random person
I thought it had to do with astronomy, disappointing
Although maybe I'm glad it wasn't about astronomy given the circumstances
So you are telling me that this concert affair scandal was orchestrated PR campaign to promote their platform?
No
The post is about people finding out about Airflow, thanks to Astronomer.
Apache AirFlow is an orchestration platform. Astronomer’s product is a platform built on top of AirFlow. Astronomer’s CEO was caught romantically holding their head of HR on the Jumbotron at a Coldplay concert.
Astronomer’s product is a platform built on top of AirFlow.
Does it add anything of value or is it just, you know, a scam?
Configuration, Deployment and management of the infrastructure needed to run airflow at scale is painful.
Astronomer makes it easy.
Sometimes I feel dread like I’m in some horror film.
So does AWS' MWAA service! ... Just kidding, that's the worst managed service offered by AWS in terms of configuration, usage, and documentation!
That escalated quickly from workflow management to HR drama. Nothing like mixing business with pleasure at a Coldplay show.
Awesome. That means I don't have to worry about configuring software anymore, right?
No, it means you have one more thing that requires a shitton of configuration.
in English pls /s
Automatic development? If business execs knew about this earlier we wouldn't need developers to begin with!
To clarify for this audience, Airflow primarily does r/DataEngineering or r/BusinessIntelligence orchestration, i.e., data pipeline orchestration.
Airflow is pretty dogwater as far as orchestration goes, not gonna lie.
It’s just easy to learn.
I’m gonna need to know whether dogwater is good or bad
Ngl, when you're thirsty at 3am with nothing else to drink nearby... It's still kinda dogwater
I guess it wouldn't be dogwater if it wasn't dogwater
The assumption is that a dog got to it first, otherwise it's just boiled ice
It is aladeen
What’s a better alternative? Genuine question, new to container orchestration
Airflow is not a container orchestration tool. That would be, for example, Kubernetes. If we are talking the infrastracture managment kind of orchestration, you would likely use Kubernetes with Argo CD, to handle lifecycle of your deployed applications.
Otherwise a more modern k8s native alternative to Airflow would be Prefect.
How is container orchestration different from infrastructure management kind of orchestration? Sorry if it's a dumb question, I'm pretty new to this.
They are kinda the same in the goal. It's the platform that differs.
Container orchestration is a kind of infrastructure orchestration, but on all infra orchestration is container orchestration.
Or a different way- It is a matter of what the orchestration tools supports. If it supports orchestrating containers, it's a containers orchestration tool.
And lastly, when deadling with kubernetes, you would usually use another orchestration tool to manager the kubernetes itself. Kubernetes itself is set of servers running kubernetes nodes, and this would usually. You want to be able to quickly spin up and update new k8s nodes, and therefore might need another orchestration tool to manage it.
You might use terraform/ansible and maybe SaltStack to manage the servers and to setup kubernetes nodes. Then you would rely on kubernetes and argocd to orchestrate the containers themselves.
From our experience, Dagster or Prefect, but that might just be because no-one knows how to use Airflow on the project
Seconding dagster, I'm replacing SO much custom scrappy glue with dagster jobs. I'm running it on my own auto scaling ECS and I get to just plop in existing containers with docker in docker to speed up migration.
Using polars and duckdb inside these jobs also turns it into an ETL weapon. The web GUI is lovely too.
Dagster is good.
If you are looking for an alternative to airflow, I recommend looking into dagster. It is newer compare to airflow but more user friendly.
github actions for my broken ass
Hamilton or prefect can be better. Airflow seems a bit on the heavy side too
This is 100% coming from someone who never knew what pre-airflow life was like
Lol at the weekly distribution of searches on airflow. The obvious weekend lol
If there is airflow in my resume, should I switch job right now ?
… You didn’t know about Airflow or Dagster?
Good. Astronomer represents everything I despise with tooling in tech. Layers of abstraction over layers of abstraction for very little value added.
What is Orchestration? Is it like a rehearsal for an Orchestra?
so orchestration is fancy word for CI/CD pipelines, like running ansible scripts from Jenkins?
Many kinds of Orchestration, from Maestros in an orchestra, to Controllers orchestrating Kubernetes resources, to bash scripts that have a specific workflow, to Airflow or Dagster for more generic stuff, to CI/CD pipelines with Jenkins/Buildkite/Github(Nektos) Runners/Gitlab Runners. To each problem its tools...
Ansible, Terraform, Nix, are declarative approaches to describe the Z state you want, pointing at resources [HEYO], and whatever state [HEYO] is, the tool engine has to figure out how to get the declarative state Z.
So, CI/CD is one of the ways to use the concept of Orchestration, not a fancy word for it, since Orchestration is much broader.
On the other side of the coin you have a concept that may make things much more complex, but has an easier time with reliability and keeping system-wide promises, which is a Choreography. Like the name suggests, each piece knows what to do at the right time. Complexity comes from the pieces being completely decoupled and only reacting to cues (e.g. events/calls). Reliability and keeping promises comes from the fact there isn't one single point of failure, like the Maestro having an heart attack and the play stops (tho, a ballet show also stops if one of the performers has an heart attack, but in computers, it usually isn't an issue)
Yeah. I understand that it's sorta "CI CD but also let's include much more functionality than just that"
No no, Airflow for CI/CD is like thinking Chocolate can be a full lunch meal. Sure it feeds you, but you have serious issues in a not distant future, and serious people will look at you with frowing eyes. Just because it's a kind of food/orchestration, don't mean it works for lunch/CICD.
Airflow was built for a completely different type of workload/requirements.
nah. i am not saying ci cd is the same thing. the principle is the same, you keep configurations and scripts and logic to automatize tasks. ci cd is just a basic or say, one aspect of this approach. obviously orchestration is supposed to be more, but that kind of simplification helps me to understand what is this about.
The triggering events don't usually overlap, CI/CD is more bound to events you have no way of escaping from in a domain's workflow.
Imagine like developing new recipes for cakes. Everytime you develop a recipe, you always have publish it to your blog, another bro makes a CI/CD pipeline to validate your recipe and include it in his restaurant menu. Hints are developing (triggering event), checking stuff (CI), adding it to menu (CD). This CI/CD pipeline will usually do only this kind of stuff, sure it can check if the restaurant can accept the menu (e.g. has all ingredients), compile the recipe to be valid to the Portuguese branch of the restaurant, the French branch, etc. But point being, it's focused on the the cake recipes.
The pipeline definition and pipe configs usually live along side the development of the cake recipes.
For Airflow, it's more like planning a wedding.
Albeit the same wedding is not recurrent, it's usually how Airflow is called. On schedules or directly called from another service (can be CI/CD even). The event types Airflow reacts to are very different from CI/CD based ones
You have a lot of moving parts, from tables, plate colors, DJ, send invites, process attendee responses, and the cake. You can create a solo repository for that wedding configuration/definition, you'll still have to send it to an Airflow server. But that definition can do an Octopus-like pull from several places to make the wedding a possibility. DJ can only be requested through their online website form. Florist services can only be requested by calling. To include the previous bros' Portuguese Branch Restaurant Cake, you have to walk into the restaurant.
In the end,
Nothing stops you from having CI/CD on an Airflow definition repository. Every time you change it, its validated and pushed to the airflow server, ready to be called. But CI/CD is focused on the small system/domain where it lives, while Airflow orchestrates stuff from several places. CI/CD lives in the domain repo (e.g. the .github/workflows folder), while an Airflow pipeline definition may be the domain repo itself. (Or a repo with several definitions).
Configuration may or may not live along side the definition, because inputs and secrets, can be given at the call time/event time.
Usually CI/CD doesn't take input, only the event type and where does it come from, and a bunch of other metadata, which you have to juggle to make an "input based" CI/CD pipeline. E.g. a pipeline only triggering on a cakes/* named branch.
thanks! that's a great analogy! I should look more into that, seems a very interesting thing to implement
I asked Gemini to make the comments more digestible:
Understanding Orchestration: From Maestros to Microservices
1 What is Orchestration?
At its core, Orchestration is the automated configuration, coordination, and management of complex computer systems and software. The goal is to streamline and automate a sequence of tasks to execute a larger, end-to-end process or workflow.
Think of it like a maestro conducting an orchestra. The maestro doesn't play any instruments but holds the complete musical score (the workflow) and signals to each musician (an individual service, script, or task) exactly when to play their part to create a cohesive symphony.
Technically, these workflows are often represented as a Directed Acyclic Graph (DAG)—a series of tasks where each task can trigger the next, but the overall flow moves in one direction without circular dependencies.
2 Types of Orchestration: Different Problems, Different Tools
While the principle is the same, the term "Orchestration" is applied in several different domains. The specific tools and patterns used are highly dependent on the problem being solved.
A. Simple Workflow Automation: The Shell Script
Analogy: A To-Do List. You write a list of sequential steps that must be done in order: 1. Go to the store, 2. Buy ingredients, 3. Cook dinner.
A bash or PowerShell script executes a series of commands one after another. It's highly imperative and typically manages tasks on a single machine.
B. CI/CD Pipelines: The Cake Recipe
Analogy: Publishing a Cake Recipe. Your CI/CD pipeline is the automated process that takes your new recipe from a draft to your restaurant's menu.
The workflow is triggered by a git push, runs tests (CI), and deploys the result (CD). The configuration (.github/workflows, Jenkinsfile) usually lives inside the same repository as the code it manages.
C. Data & Business Process
Orchestration: The Wedding Planner
Analogy: Planning a Wedding. A wedding planner coordinates many independent vendors: the caterer, the florist, the DJ, and the venue.
Tools like Apache Airflow excel here. The workflow is a master plan that interacts with each "vendor" (service/API) via their preferred method. The configuration (the DAG) often lives in its own dedicated repository.
D. Infrastructure & Resource Orchestration: The City Planner
Analogy: A City Planner. The planner holds a blueprint (declarative configuration) of the desired city. They give this blueprint to a team of controllers who make it a reality and fix any deviations.
This is less about a linear sequence and more about maintaining a desired state. Kubernetes is the quintessential example, using a self-healing control loop to match the actual state to the desired state defined in a YAML file.
3 Key Distinctions and Related Concepts
A. Orchestration vs. Choreography
Analogy: Dancers on a Stage. In a choreographed dance, there is no central conductor. Instead, each dancer knows their own part and reacts to cues (events) from other dancers.
| Aspect | Orchestration (Maestro) | Choreography (Dancers) |
|---|---|---|
| Control | Centralized | Decentralized |
| Communication | Point-to-point commands | Event-based (message bus) |
| Coupling | Tightly coupled | Loosely coupled |
| Visibility | Explicit workflow | Implicit workflow |
B. Pipeline vs. Orchestration
A Pipeline is a specific type of orchestration, one that is predominantly linear and sequential (like CI/CD). Orchestration is the broader concept that also includes complex, graph-based workflows (Airflow) and continuous control loops (Kubernetes). All pipelines are a form of orchestration, but not all orchestration is a pipeline.
C. Imperative vs. Declarative
- Imperative ("How"): You provide explicit, step-by-step commands.
Analogy: Giving someone turn-by-turn directions to your house.
- Declarative ("What"): You describe the desired final state, and the system figures out how to get there.
Analogy: Giving someone your home address and letting their GPS navigate.
4 Summary Table
| Type | Analogy | Primary Goal | Approach | Example Tools |
|---|---|---|---|---|
| Simple Workflow | To-Do List | Linear task automation | Imperative | Bash, PowerShell |
| CI/CD Pipeline | Cake Recipe | Automate software delivery | Mostly Imperative | Jenkins, GitHub Actions |
| Business Process | Wedding Planner | Coordinate cross-system workflows | DAG-based | Apache Airflow, Dagster |
| Resource Mgmt | City Planner | Maintain infrastructure state | Declarative | Kubernetes, Terraform |
5 Real-World Industry Examples
CI/CD Pipeline at Spotify: When a developer pushes code, a CI/CD pipeline automatically tests it, deploys it to a small percentage of users (a canary release), monitors for errors, and gradually rolls it out to all users.
Business Process Orchestration at an E-commerce Company: A company like Wayfair uses Apache Airflow for a nightly workflow that pulls sales data, gets user logs, joins them in a data warehouse, generates a sales report, and emails it to the finance department.
Resource Orchestration at Slack: The entire Slack application runs on Kubernetes. If traffic surges, Kubernetes automatically scales up services. If a server fails, Kubernetes automatically moves the workload to a healthy server with no manual intervention.
6 Conclusion
"Orchestration" is a broad and powerful concept, not just a fancy word for CI/CD. While a CI/CD pipeline is one specific type of orchestration, the principles extend to coordinating complex data flows and maintaining the state of entire infrastructure clusters. Understanding the distinctions between patterns like orchestration vs. choreography and imperative vs. declarative approaches is crucial for choosing the right "maestro" for the job.
See, one more reason to use airflow