Machine Learning in Fastly's Compute@Edge - Andrew Brown, Intel & Matthew Tamayo-Rios, Fastly

Machine Learning in Fastly's Compute@Edge - Andrew Brown, Intel & Matthew Tamayo-Rios, Fastly
Recent advances in deep learning promise to transform the way we live, learn, and work. In this session, we will explore how WebAssembly modules can efficiently run machine learning models (ML) via wasi-nn. We will explain what it takes to make this work in Fastly's Compute@Edge environment. To enable efficient execution in a stateless FaaS environment we have extended the wasi-nn spec to avoid reloading models on every request, revised host APIs to avoid interference with Wasmtime’s event loop, balanced some security-related tradeoffs, and introduced a new proxy backend based on the KServe protocol. We will demonstrate all of this functionality by demoing a Compute@Edge service using OpenVINO, ONNX, and PyTorch for classification and generative AI.