How to Run a Local LLM on Raspberry Pi: Step-by-Step Guide to Deploy AI Models Locally

[@yucelaytacakgun9928]

How can we use RAG? because I want to use local LLM for coding and run the code. Finally working robot

[@Lp-ze1tg]

Raspberry pi 4 or 5? 4gb or 8gb ram?

[@alecrimi]

in this video RPI 4 with 4GB, if you have Pi5 and 8GB of RAM, things will go definitely better. Also, you cannot run Phi LLM and LLama2 with 4GB of RAM, with 4GB you can only use TinyLLama and TinyDolphin

[@LivingLinux]

​@@alecrimi You can use a swap file. Better make sure you have some fast storage. But the larger the model, the slower it gets (even without swap).

[@alecrimi]

@@LivingLinux Swap, external hat like Coral or the Hailo 8L, there are plenty of possibilies. Though the Hailo hat works only on the RPI 5 plus given the price makes more sense to buy a Jetson

[@Levince36]

@@LivingLinux May I ask, I wonder why Pi couldn't run SSD but my HDD worked? Is it the support of Pi 4 (4GB) or the lack of power, or anything else?