Wasm Is Becoming the Runtime for LLMs - Michael Yuan, Second State

Wasm Is Becoming the Runtime for LLMs - Michael Yuan, Second State
Today’s LLM apps, including inference apps and agents, are mostly written in Python. But this is about to change. Python is too slow, too bloated, and too complicated to install and manage. That’s why popular LLM frameworks, such as llama2.c, whisper.cpp, llama.rs, all thrive to have zero Python dependency. All those post-Python LLM applications and frameworks are written in compiled languages (C/C++/Rust) and can be compiled into Wasm. With WASI NN, you can now create complex LLM apps in Rust and run them in Wasm sandboxes. Rust and Wasm could be high-performance and developer-friendly alternatives to Python today. The combination to develop and run LLM apps is more efficient, safe, high performance with small footrprint. In this talk, Michael will demonstrate how to run llama2 series of models in Wasm, how to develop LLM agents in Rust and run them in Wasm. In-production use cases, like LLM-based code review and book-based learning assistants, will be discussed and demoed.

[@Dorisoft]

This is gold

[@jonton6981]

Agree. I think data connector and transformation plugins are a big opportunity too.

[@autohmae]

3:22 Java set out to do it and Javascript was only named Javascript because of a deal between Sun and Netscape, Javascript was just gonna be it's little brother that would be used as the glue-code in the browser to run Java... it was never supposed to be this way. Even in 1995 Javascript was more used than Java, until 1997 when Java passed it. Both surpassed C++ before 2000 and 3 years or so both passed C... C which was the portable language before all of this.

[@autohmae]

3:07 I checked, it's funny he''s "Python bashing" in this talk because it used Python and Python curl library to install wasm runtime and plugin. The runtime is 60MB and the plugins are less than 1 MB or a few MB.