Llamafile: bringing AI to the masses with fast CPU inference: Stephen Hood and Justine Tunney

[@Neltharion2k]

This is so awesome! Just tried out llava 1.5 7b llamafile and it worked out of the box running on my CPU, without eating all of my RAM! The token generation speed was good enough for me! And my CPU is ~8 years old. Holy cow!

[@bigglyguy8429]

Where gguf?

[@geomorillo]

where?

[@dskbiswas]

What did I just watch ...mindblowing! Finally someone took the initiative of going against the tide while giving CPUs some attention that they have lost to the GPU madness!

[@navodpeiris9054]

loving the llamafile already. this is how i deploy local LLMs now!

[@LeftBoot]

Local for yourself or clients?

[@longboarderanonymous5718]

These individuals are pioneers of the Personal AI. Efficient, Universal, and Economical.

[@aeu126]

This was my favorite presentation!

[@lolilollolilol7773]

Justine Tunney is a genius. Everything she does is undistinguishable from magic.

[@granite_planet]

Looking at her work, I sometimes think I should just quit programming and pick up something like gardening instead. :D

[@Viewable11]

Llamafile now supports OpenAI API and non-AVX CPUs. Finally! Having the OpenAI API is a must.

[@deadlokIV]

Justine just shifted the timeline 💥🔀

[@indylawi5021]

This is fantastic! I can't wait to try it out.

[@gunnarasmussen207]

Well, what I'm suppossed to say but: awesome...running local AI on normal consumer hardware without any worries about privacy seemed impossible just months ago. All the computational work in GPT, Gemini and others is done in the cloud on the companies servers. So you don't know, what they are doing with your data. Even if you have nothing to hide - I'm sure erveryone has certain things, he/she wants to stay private...this seems to be the right way of implementing AI in a private manner. And doing such a great afford without any commercial Interests is nothing but mindblowing. Keep up the good work, please!

[@tejaslotlikar3573]

Now this is called achievement. Meanwhile the so-called "open"AI is looting people. You guys are awesome

[@LaHoraMaker]

I really like the idea of a Threadripper configuration but... does anyone have a reference machine configuration for that? I'd like to compare the price to existing alternatives like the dual RTX4090 setup that is mentioned!

[@leejacksondev]

This is utterly brilliant. What a fantastic presentation. Amazing project.

[@Alice_Fumo]

well... I just took a look at the repo for the llama 3 70b llamafile repo and found this info about performance: "AMD Threadripper Pro 7995WX ($10k) does a good job too at 5.9 tok/sec eval with Q4_0 (49 tok/sec prompt). With F16 weights the prompt eval goes 65 tok/sec." 70b would be the lower bound for model I would enjoy using, but getting like 6 tokens per second output on a 10k$ CPU... At that point I could just as well build a GPU machine... So, even though I think this is in concept an amazing project, either it or hardware in general has a long way to go still before it is in my opinion usable for an average person such as myself.. (I'm assuming the performance data on the huggingface repo are at least somewhat accurate and not outdated)

[@craigscott4205]

Justine an absolute champion!

[@delq]

Awesome, exactly what I have been looking for, no more virtual heavy environments, no more heavy nvidea cuda drivers ! Lets fricking go !!!

[@aiforsocialbenefit]

Awesome. Great project and presenters!

[@raiumair7494]

Refreshing indeed - tokens per seconds is one measure and I like eval speed but what and how do you measure that?

[@FirstNameLastName-fv4eu]

These cloud companies trying their best to keep the valuation high!!! This guy is the new CDO manager!!

[@spookymv]

it was the first time I had the chance to listen to one of his speeches. bro i like this guy. D:

[@dbreardon]

He said,, "Who remembers using the original Netscape Navigator?" ........to that I say, who remembers using the original Mosaic browser? And then telnet before the graphical internet?

[@WoodyWilliams]

[raises hand] Doh!

[@tinkerman1790]

“Who remembers the handshaking tone in dial-up process” 😂

[@smthngsmthngsmthngdarkside]

Who remembers the original smoke signals?

[@Atonsha]

How about BTX?

[@vncstudio]

We do! and Gopher!

[@tollington9414]

Absolutely fascinating and totally genius

[@Jason_RA]

This is absolutely amazing!

[@john_blues]

Is there a way to get Windows to run llamafiles bigger than 4Gb? Without being able to do that, it is very limiting in the models you can run.

[@rayhere7925]

This is a game-changing breakthrough. Can't underplay this any other way.

[@eggmaster88]

Awesome work!

[@GandalfTheBrown117]

Justine is a GOAT

[@NeXTOoOoOoO]

Wow! Really great work!

[@RomuloMagalhaesAutoTOPO]

Amazing. Thank you.

[@OranCollins]

omg i love Justine Tunney! they are amazing!

[@CaptainSpoonsAlot]

this is just fantastic.

[@Godkidz7]

Freedom and Justices are more expensive than Money and Power. No one live and rule forever. Respects and Salute to you guys...

[@KevinKreger]

Amazing❤

[@XEQUTE]

Love it!!

[@masbuba]

Oh shit, CPU prices is going to hike

[@johnkost2514]

This is better than the Nvidia NIM solution (which is just containerization). Way better ..

[@Charles-Darwin]

Awesomesauce

[@ShieldsWebDesign]

Why is no one talking about this?

[@GandalfTheBrown117]

Tired -> wired around @9:30 😂

[@romanbauer]

👏🏻👏🏻👏🏻

[@philly_eddie]

very cool

[@cholst1]

*checking on RAM prices*

[@timchapman8539]

I need an AI that can access the files on my hard drive. Does anyone have a suggestion? I don't want to upload them to the AI. I want the AI to access them directly.

[@bigglyguy8429]

ChatGPT4all has RAG

[@constantinegeist1854]

All of this was already possible before... Already back in early 2023. What they did was just save you 15 minutes (otherwise you'd have to download an inference program and weights separately)

[@7T7Soulz]

this is future

[@omercelebi2012]

What about quality trade-off? Did they mention about that?

[@JohnnysaidWhat]

this guy is a fkn rockstar on stage I was totally blown away 🎉

[@erb34]

Don't forget the browser.

[@ravishmahajan9314]

NVIDIA has hired CIA agents to make sure this technology is not reaching in hands of public. Be safe sir !😝

[@hope42]

Am I the only one that someone AI generated Matt Perry?

[@fkxfkx]

well this feels like something out of left field.🤷‍♂️ Seems too good to be true. What are the catches?

[@projectsspecial9224]

As an AI Design Engineer and developer of original works in Unified Language Models (predecessor to LLMs) for over 20 years, this compact framework, GPU or custom hardware independence, and resource efficient methodology is the correct approach. 😊

[@fkxfkx]

⁠”a” correct approach but maybe not “the” correct approach. It’s not clear what downsides there are yet.

[@bigglyguy8429]

@@fkxfkx I'm not sure how you're supposed to run it? GGUF I can run but what the heck is the 14GB "llamafile" thing?

[@maxd3946]

@@bigglyguy8429 actually, you don't need a 14GB llamafile. It's even unable to be run on windows (4GB max executable size limit). You can keep a llamafile without embedding any model in it and call it with the -m parameter to specify the model file to load.

[@snow8725]

Fuck yeah!!!

[@pandoraeeris7860]

The Singularity is here.

[@TalsBadKidney]

let's go to the gym

[@JimAmos]

Hats off for the engineering feat. But in terms of application, we are still just talking about text summarization. And the image generation in your own demo was just as disappointing as ever. There's no killer app for LLMs yet even though we keep throwing money and science at it. What are we even doing?

[@bobtarmac1828]

Free candy, I mean, Free open source Ai for everyone. It’s a like a trick. Don’t fall for it. Cease Ai.

[@WenRolland]

Great work!