Untold story of AI’s fastest chip

Groq's chips are the fastest in artificial intelligence. In a battle against heavily funded players, will that be enough for this underdog startup to break through?
#ai #business #artificialintelligence #entrepreneur #technology #groq #chips #google #gemini #chatgpt #nvidia #grok #elonmusk #chamath
(footage featured from Groq's marketing material's, CNN, the All-In Podcast, and the Web Summit 2024. Refer to Groq's KZbin channel to see more).

[@chankhavu]

Good video. At the end, the main flaw of the chip was pointed out: low per-chip memory, so they have to connect a lot for something that a single GPU will suffice. The inference speed is still remarkably high because all computations is orchestrated ahead of time (it's possible because Groq LPUs are entirely deterministic). Another thing to note that, as their chief architector said, "we haven't even started tapping into Moore's law yet". They are using, if I remember correctly, 14nm (while Nvidia is 5nm and they're getting down to 3nm). So, if they manage to strike a contract with some chipmaker that can print out more inticate wafers, the performance will be increased substantially.

[@chankhavu]

The fact that 100% of the chip was made in the US is huge. That means DOD can have more confidence in the potential future contracts with them.

[@Synapse.Official]

Thanks for sharing. All great insight.

[@brianmi40]

@@chankhavu Agreed. I think it's way huge that it's USA made. I see great future risk as the AI race heats up. Putin isn't the only one who's had the realization that "who gets AI (AGI/ASI) first rules the world". At some point Nvidia embargos won't define the majority of any race toward it. So, my fear is China taking Taiwan and we LOSE all state-of-the-art chip manufacturing at scale. There's already lots of discussion about scaling up local production to mitigate this huge risk exposure, but also for jobs back home. Happy to have them see great success and expansion and as always, competition most often makes things better for all of us, the beneficiaries of the end product.

[@kamu747]

@2:34 😂😂😂 All that aside, low latency conversations with AI is refreshing. That would make conversations feel so much more natural. Impressive.

[@Synapse.Official]

It’s a pretty significant jump from what we’re used to.

[@ArcSine-]

so, no 7T$ for Sam 😢

[@harrytsang1501]

"low latency" is the how, but the answer to why is actually that it would cost less time/energy and server cost would be reduced. Even a marginal reduce in server energy cost compared to GPU would be a huge selling point

[@Synapse.Official]

Great point and definitely one that Groq has advertised.

[@pixlent]

Now this is what I signed up for 👍👍

[@marvelman1331]

Omg, this channel is perfect to watch😮

[@Synapse.Official]

Thanks for watching!

[@tigranatayan2631]

Of course the speed matters! It matters a lot. When you try to develop a complex system(like an AI assistant), where LLM agents talk to each other to decide what and when to do, there are a lot of LLM calls happening, and It's too slow now. Not only that but processing the given document to extract some information from it also is too slow now. Those are all important problems to solve, and that's why the industry is going to lower the price and increase the speed.

[@BarnoBarno-jp2ks]

Loved the video, channel aesthetic is a beauty. , subbed

[@Synapse.Official]

Thanks for watching! Much more on the way

[@LambdaCreates]

Grok - Twitter AI Groq - AI Chip Company CANNOT BE A COINCIDENCE

[@pictzone]

It is, Groq even mentioned it's "unfortunate" 😂

[@macemoneta]

They are both using the term from Heinlein's 'Stranger in a Strange Land' novel, as mentioned at the beginning of the video.

[@maximus1172]

Nice channel man!! Keep it up, suscribed :)

[@Synapse.Official]

Thanks! Much appreciated.

[@lc285]

You had me until 578 Groq chips to 2 GPU chips.

[@kingki1953]

Damn, i would like to see new computer architecture to add LPU, CPU, and GPU in future computer.

[@ginebro1930]

2 things, Nvidia has TPUs since 2018, even then multiple developers stated that price per performance is quite similar to classic GPUs.

[@psyenz8946]

Exotropic just announced this new 'game changer' chip it's wild!

[@IntellectCorner]

Hi buddy, really high quality video. What editing software did you use?

[@Synapse.Official]

Thanks for watching. Good old fashioned adobe.

[@yonatan09]

Beautiful channel. I love it

[@waifulover3578]

nice video and editing mate. Enjoy my sub man....

[@Synapse.Official]

Thanks for watching! Much more coming

[@MayankPratapSingh_022]

hey it was an awesome video , just want to know which software you are using to make this video, is it after effects if yes , which plugins or extensions ?

[@Synapse.Official]

Thanks for watching! No plugins or extensions. All manual in After Effects.

[@klejdisevdari3916]

Great content, keep it up!

[@Synapse.Official]

Thanks for watching!

[@Elegant-Capybara]

The demos are very misleading. They're running Llama 2 70B with an unknown quantization vs ChatGPT model which is 8x200B. ChatGPT is 22x larger and imagine how much traffic ChatGPT gets. They are definitely fast, but the comparison with ChatGPT is apples and oranges.

[@Synapse.Official]

Interesting insight. Thanks for adding.

[@AdvantestInc]

The strategic emphasis on creating chips specialized for tasks like language processing signifies a major leap forward, offering new horizons in AI capabilities and applications.

[@akaashraj8796]

Love your content ❤

[@Synapse.Official]

Thanks for watching!

[@danielfilipek8408]

In my opinion fast computing AI chips is a big deal, for example in gaming industry. There were attempts to make NPCs in the games driven by AI where player can make real, unique conversations with them but latency was killing the immersion. Now it can drastically change which is a big step for a new era of RPG games

[@Synapse.Official]

Great point. Didn’t think about that initially, but it could be a major area where low latency makes a huge a difference.

[@LiguoKong]

The video is of very high quality.

[@Synapse.Official]

Thanks for watching!

[@mrtn951]

Comparing llama2 with chatgpt totally does not make sense. What about Benchmarks, memory bandwidth and computations per sec?

[@rameshdevasi6720]

wait 1000 request ahead, lol they need more computing

[@nuclearthrone2497]

More like “UnderGroq”

[@Synapse.Official]

Okay… okay… possibly something there

[@Reversmullac]

Bit of a different idea (as much as i like Groq) but have you seen tests in Bio chips? It's been going on for a while now i think but they've tested fungi and Neuron Cells made from Stem cells in a chip like format. This stuff is very very new and hasn't really been explored unlike the explosion in LLM's and Neural networks. Imagine if without telling the mural network to move to the next probable outcome a bit of fungi or neuron expended little energy for the same output!

[@Reversmullac]

Also on the scalability aspect - Could they not just sell Groq chip as set of chips in an Array like format or compile them into one Unit daisy chaining processors together?

[@Synapse.Official]

Innovation going to be needed to meet ambitious scaling goals for sure. Thanks for sharing.

[@mika2666]

Very cool tech but increasing performance while not relying on purely SRAM to store the model is a bigger opportunity, models will continue to grow and SRAM density isn't increasing very fast if at all. Maybe 3D stacking can make this more viable.

[@Wittgenstein.]

Great value proposition, working close with modern kernels and creating custom SDKs for which the performance can be used in achieving value creation via custom applications is crucial. Same goes for library development. Great scale economies natural for the industry, and i believe that the orchestration problem will actually add another layer of competetive complexity, as integtation part requires huge network of diverse infrastructures and clients, which will only enhance the value, as the integration costs will go down and flexibility will grow with time.

[@OCJ0001]

Hey sir. You seem pretty smart. Could you please do a video on Palantir. It’s an incredible product and company.

[@chuchel3156]

good video

[@Synapse.Official]

Thanks for watching!

[@ayush__dubey7582]

Great video quality.....just try making more business study related video related to Indian industries....channel will boom Good luck and keep it up

[@Synapse.Official]

Thanks for watching

[@usercurious]

Good video

[@Synapse.Official]

Thanks for watching

[@usercurious]

@@Synapse.Official much pleasure, keep posting

[@Synapse.Official]

🫡

[@johanavril1691]

My bet is that it's gonna fail on the long term, not flexible enough. The algorithms will change too fast and those chips will be too specialized. But hey I might be wrong.

[@mikoajlewicki3713]

❤

[@SG-gm5kq]

Bro i thought this is a channel with million subscribers. Shocked to see you only have less than 2k subs. You definitely deserves more. Keep it up 👍 ❤ from India

[@Synapse.Official]

Buy low, sell high! Thanks for watching.

[@infologichannel]

This is made by video scribe? Or not?

[@MarkBoda]

Notification squad

[@Synapse.Official]

🫡

[@Telencephelon]

Extremely superficial. Like not mentioning that GPUs right now are so hot because nobody knows what architecture will make it. For instance the multiplier unit may be rendered obsolte by 1bit Models (Quantization). Then you just add the weights. It's a nice addition to the arsenal of AI hardware for sure

[@nadeemshaikh7863]

Can this potentially result in greater efficiency along with 1 bit models?

[@famillicraft8163]

why tf are all video on youtube 5 to 7 minutes long now??? btw grok is not an AI Companie but an AI…

[@gytispranskunas4984]

There WAS NO "STANDARD" A I chips in existence. Groq is FIRST A I CHIP PERIOD. Just because a GPU can run A.I it doesn't mean its designed to do so. Its just a lucky coincidence that GPUs designed for upscaling like DLSS happens to be able to run LLM's. Its like saying a passenger car can also be used to shipp goods... But it WILL NEVER BE A SEMI TRUCK... Anyway, strange times are coming. A I is getting faster by FOLDS not linear scale. Its a bit scary. If this continues... In 10 years from now nobody knows where we will be.

[@Stroporez]

Wasn't TPU first AI chip?

[@gytispranskunas4984]

@@Stroporez maybe... But it's strange when people immediately attack GPUs for being slow at a.i... like relax. It's never been built to run a.i in the first place.

[@iritesh]

Yeah that was weird when he said GPU were made for AI lol

[@pictzone]

10 years?? I think that even predicting what the world will look in 4 years is becoming impossible

[@gytispranskunas4984]

@@pictzone my prediction that in less than 10 years Hollywood will be cooking movies just by pasting a script to A.I and letting it do everything for them. No actors, no voice lines. Text in, movie out.

[@MrTweetyhack]

made in the US by who? fast when compared wto what? junk marketing

[@Synapse.Official]

They have a partnership with Samsung to produce the chips in a US foundry in Taylor, Texas. Fast (at the very specific task of LLM processing + inference) compared to every other GPU chip competitor. Important to note it’s less dynamic than those other chips. No marketing here. Thanks for watching!

[@elvijsx]

ASML better

[@Synapse.Official]

Different beast. Their lithography systems are key for the big chip producers (TSMC, Samsung, etc.) to manufacture chips.