We built an orchestrator from scratch. Here's why.

Fly.io doesn't use a typical orchestrator - ours is *decentralized*, which is kind of weird! In this video I walk us through what you might typically need to build an orchestrator from scratch, why the mainstream ones like Kubernetes and Nomad didn't work for us, and what we did instead.
BLOG POST: Carving The Scheduler Out Of Our Orchestrator
fly.io/blog/ca...
Video written and produced by Annie Sexton.

[@TannerBarcelos]

This is a really cool approach! The bidding system is really clever.

[@DennisIvy]

Awesome video Annie! 🤩You are killing it!

[@h3techsme]

Nice speed run through the problem space! 🏁

[@INDABRIT]

If I'm understanding this right, it's not really a decentralized orchestrator, just decentralized state? When a job comes in, the orchestrator polls (asks for bids from) the workers, but then the orchestrator still decides which worker gets the job. Also still not clear how that resolves bin packing. Do the workers try to maintain some percentage of available resources and not take on more jobs? If so, the same can be accomplished from a centralized orchestrator, and if not you'll end up bin packed anyway unless you intentionally overprovision the number of workers in someway. There must be more going in flyd to get the binpacking resolution that is not explained here

[@DryBones111]

By having workers bid on jobs, each worker can have a different utilisation strategy, opaque to the rest of the system.

[@Dom-zy1qy]

So you wanted to perform bin-packing, but treat the machines' resources (vcores, ram, etc.) as the "capacity" of a container? When you were talking about how it's not ideal to try and fit as many applications as possible on a single worker, did "fit" refer to the size of the fly machine images on disk, or the expected resource utilization of an application? (Unsure how you could even predict resource utilization without relying on probalistic methods anyway, so I'm assuming it was the former)

[@nileshmiskin5711]

More of a Choreographer than Orchestrator! Brilliant!

[@fernandocanas7862]

This was simply beautiful 👏

[@arak_seemonster]

I know very little about orchestrators and how they work, but I do know a bit about quadratic programming (QP) and optimal resource utilization? It seems like most of these orchistrators use some form of heuristic for scheduling, but is that actually the best approach? Would orchastrators not be betters served using QP where you add the resources and constraints and then schedule the work continuously? That way you can constrain the resources used, yet still optimize the system as a whole?

[@arieheinrich3457]

Very intresting use case. Isnt this basically the same as the inner engine of GitLab / GitHub / ADO when they run pipelines with a pull instead of a push. Its exactly how runners / agents listen to broadcasted events from the main "API" service.

[@kaankeskin2214]

thanks for watching? thanks for explaning :)

[@Rachelebanham]

nice - out of interest have you open-sourced flyd?

[@cbnsndwch]

#Miami mentioned!!! 🎉🎉🎉

[@erkintek]

So flyctl keep track of empty workers to kill or in loaded region to create more workers?

[@saiskanda]

Because "reasons" 👻

[@Jebusankel]

Shouldn't a decentralized orchestrator be called a choreographer?

[@twitchizle]

What

[@MadalinIgnisca]

😂😂😂😂 I did not know if to cry or to laugh. U know that real system engineers will not believe you?