Messaging With Limits: Concurrent, Multi-Stage Data Processing in the Real World

ElixirConf US 2024 - Brian Meeker
2024.elixircon...
Our team at GetThru found ourselves with a scaling problem. Working with our existing infrastructure for sending text messages was becoming increasingly complex. Years of bewildering external requirements had led us to the point where the existing system was both incorrect and difficult to reason about.
In this session, we’ll go over a case study of our new concurrent, multi-stage data processing pipeline. Starting from the fundamentals, we’ll see how our team efficiently handled our existing external requirements and built in flexibility for the future. Finally, we’ll look at how we did all of this in our existing monolith, avoiding the complexity of microservices.