ProgrammerHumor
thankYouColdplay
Discussion
Tl;Dr it for me
Orchestration is the automated configuration, coordination, deployment, development, and management of computer systems and software.
But what does that have to do with coldplay or that incident? Asking for a friend...
The company they worked for, created software that did the thing
Thank you random person
So you are telling me that this concert affair scandal was orchestrated PR campaign to promote their platform?
No
I thought it had to do with astronomy, disappointing
Although maybe I'm glad it wasn't about astronomy given the circumstances
Apache AirFlow is an orchestration platform. Astronomer’s product is a platform built on top of AirFlow. Astronomer’s CEO was caught romantically holding their head of HR on the Jumbotron at a Coldplay concert.
That escalated quickly from workflow management to HR drama. Nothing like mixing business with pleasure at a Coldplay show.
Astronomer’s product is a platform built on top of AirFlow.
Does it add anything of value or is it just, you know, a scam?
Awesome. That means I don't have to worry about configuring software anymore, right?
in English pls /s
Automatic development? If business execs knew about this earlier we wouldn't need developers to begin with!
To clarify for this audience, Airflow primarily does r/DataEngineering or r/BusinessIntelligence orchestration, i.e., data pipeline orchestration.
Airflow is pretty dogwater as far as orchestration goes, not gonna lie.
It’s just easy to learn.
I’m gonna need to know whether dogwater is good or bad
Ngl, when you're thirsty at 3am with nothing else to drink nearby... It's still kinda dogwater
I guess it wouldn't be dogwater if it wasn't dogwater
The assumption is that a dog got to it first, otherwise it's just boiled ice
It is aladeen
What’s a better alternative? Genuine question, new to container orchestration
Airflow is not a container orchestration tool. That would be, for example, Kubernetes. If we are talking the infrastracture managment kind of orchestration, you would likely use Kubernetes with Argo CD, to handle lifecycle of your deployed applications.
Otherwise a more modern k8s native alternative to Airflow would be Prefect.
How is container orchestration different from infrastructure management kind of orchestration? Sorry if it's a dumb question, I'm pretty new to this.
They are kinda the same in the goal. It's the platform that differs.
Container orchestration is a kind of infrastructure orchestration, but on all infra orchestration is container orchestration.
Or a different way- It is a matter of what the orchestration tools supports. If it supports orchestrating containers, it's a containers orchestration tool.
And lastly, when deadling with kubernetes, you would usually use another orchestration tool to manager the kubernetes itself. Kubernetes itself is set of servers running kubernetes nodes, and this would usually. You want to be able to quickly spin up and update new k8s nodes, and therefore might need another orchestration tool to manage it.
You might use terraform/ansible and maybe SaltStack to manage the servers and to setup kubernetes nodes. Then you would rely on kubernetes and argocd to orchestrate the containers themselves.
From our experience, Dagster or Prefect, but that might just be because no-one knows how to use Airflow on the project
Seconding dagster, I'm replacing SO much custom scrappy glue with dagster jobs. I'm running it on my own auto scaling ECS and I get to just plop in existing containers with docker in docker to speed up migration.
Using polars and duckdb inside these jobs also turns it into an ETL weapon. The web GUI is lovely too.
Dagster is good.
If you are looking for an alternative to airflow, I recommend looking into dagster. It is newer compare to airflow but more user friendly.
github actions for my broken ass
Hamilton or prefect can be better. Airflow seems a bit on the heavy side too
If there is airflow in my resume, should I switch job right now ?
… You didn’t know about Airflow or Dagster?
What is Orchestration? Is it like a rehearsal for an Orchestra?
so orchestration is fancy word for CI/CD pipelines, like running ansible scripts from Jenkins?
Many kinds of Orchestration, from Maestros in an orchestra, to Controllers orchestrating Kubernetes resources, to bash scripts that have a specific workflow, to Airflow or Dagster for more generic stuff, to CI/CD pipelines with Jenkins/Buildkite/Github(Nektos) Runners/Gitlab Runners. To each problem its tools...
Ansible, Terraform, Nix, are declarative approaches to describe the Z state you want, pointing at resources [HEYO], and whatever state [HEYO] is, the tool engine has to figure out how to get the declarative state Z.
So, CI/CD is one of the ways to use the concept of Orchestration, not a fancy word for it, since Orchestration is much broader.
On the other side of the coin you have a concept that may make things much more complex, but has an easier time with reliability and keeping system-wide promises, which is a Choreography. Like the name suggests, each piece knows what to do at the right time. Complexity comes from the pieces being completely decoupled and only reacting to cues (e.g. events/calls). Reliability and keeping promises comes from the fact there isn't one single point of failure, like the Maestro having an heart attack and the play stops (tho, a ballet show also stops if one of the performers has an heart attack, but in computers, it usually isn't an issue)
Yeah. I understand that it's sorta "CI CD but also let's include much more functionality than just that"
No no, Airflow for CI/CD is like thinking Chocolate can be a full lunch meal. Sure it feeds you, but you have serious issues in a not distant future, and serious people will look at you with frowing eyes. Just because it's a kind of food/orchestration, don't mean it works for lunch/CICD.
Airflow was built for a completely different type of workload/requirements.
nah. i am not saying ci cd is the same thing. the principle is the same, you keep configurations and scripts and logic to automatize tasks. ci cd is just a basic or say, one aspect of this approach. obviously orchestration is supposed to be more, but that kind of simplification helps me to understand what is this about.
See, one more reason to use airflow