How Groq’s LPUs Overtake GPUs For Fastest LLM AI!

We’ve been wanting to release this ipXperience for a while, and ipXchange is thrilled to finally share this chat with Mark Heaps to explain just what makes Groq’s AI chips so disruptive.
Learn more and apply to enter consultation with Groq on the ipXchange website: ipxchange.tech/boards/groq-lp...
It’ll change the way you think about AI chips, and you can play with this functionality today!
Keep designing!
#EW24 #EW2024 #AI #LLM #largelanguagemodel #GPU #CPU #processor #chatGPT #electronics #datacentre #datacenter #electronicsengineering #artificialintelligence #disruptivetechnology #genAI #generativeAI
• The latest disruptive technology, ready to evaluate for your next design: ipxchange.tech/
• Join the ipX community: ipxchange.tech/sign-up/
• Submit your own electronics projects: ipxchange.tech/projects/
• Follow us on LinkedIn: / ipxchangeltd
Learn more about ipXchange here: ipxchange.tech/about/

[@Maisonier]

I'd love to have a small black box at home with several Groq LPUs acting as LLMs for my local network. It would serve a typical family of five, each accessing it from their phones via WiFi while at home working, especially since internet connectivity can be an issue. I wonder if they'll ever sell such a device to the general public instead of just focusing on businesses?

[@ipXchange]

I couldn't say. They do make racks, but I wonder how many you would need to make something viable at home, and whether they'd let you buy not in bulk. That would be cool though. To be fair, you can use Groq cloud, but I guess you want to own your own infrastructure. Groq has deployed their LPU in super small use cases, so there might be a possibility you could get you hands on some private units...

[@alertbri]

How does an LPU differ from an ASIC please?

[@ipXchange]

I suppose it could be considered a type of ASIC as it is a processor designed specifically for large language model processing. The way that an LPU differs from a GPU is that it does not do any parallel processing - it's very good at doing things in sequence. For applications like LLMs or audio, going forward in time is all that's required because the next word depends on the words that came before it. It's pretty much a 1D problem. This is in contrast to GPUs because a 2D or 3D picture needs to understand the whole context of a scene, hence why it requires parallel processing of all the pixels in order to understand what's going on. While parallel processing in GPUs can be used to enable faster LLM AI, at a certain point, the recombination of data slows the whole process down. The LPU, however, is able to just keep chugging along at the same pace because any parallelism is done in separate chips. At a certain number of devices, it seems that this wins out in terms of performance as the GPUs stop providing a net gain for more units added to the system. This is an oversimplification, but you get the idea. Thank you for the comment and question.

[@Davorge]

@@ipXchange interesting, so why are billionaries dropping hundreds of millions in H100 clusters? wouldnt it be better for them to invest in LPU's moving forward?

[@kahvac]

@@Davorge You have to start somewhere..if you keep waiting for the next best thing you will be left behind.