Using serverless functions for real-time observability - Liz Fong-Jones - CPH DevFest 2024

This talk was recorded at Copenhagen Developers Festival in Copenhagen, Denmark. #cphdevfest #ndcconferences #developer #softwaredeveloper
Attend the next NDC conference near you:
ndcconferences...
cphdevfest.com/
Subscribe to our KZbin channel and learn every day:
/ ‪@NDC‬
Follow our Social Media!
/ ndcconferences
/ ndc_conferences
/ ndc_conferences
#bigdata #microservices #devops #serverless
In this talk, you will learn how a sub-second latency query engine inspired by Facebook's Scuba was extended from running in RAM only, to querying the most recent data that could still fit on local SSD, to querying months of data at a time using cloud storage and serverless functions. We'll describe the pitfalls of managing lambdas at scale, including impatience, maximum concurrency, runtime and architecture configuration/experimentation, and the price/performance of renting tens of thousands of parallel workers across the x86 and arm ecosystems.