Groq API - 500+ Tokens/s - First Impression and Tests - WOW!

[@kate-pt2ny]

I have joined your membership. Now I just need to get the groq API application approved. Thank you again Kris for sharing.

[@cjenkinsiv]

You had to apply?

[@kate-pt2ny]

At first need to apply for groq API, but now don't need it.@@cjenkinsiv

[@GroqInc]

Thanks for the demos. We love what you're doing.

[@paul1979uk2000]

Been testing this out online and been getting around 400 to 550 tokens on it, it's crazy fast. There are only two models it lets you select, but both are big models and run lightning fast compared to any other A.I. model I've seen online and locally.

[@automatescellulaires8543]

is there a difference between the two model though ? When you test them via groq ?

[@cjenkinsiv]

It's super fast but they don't follow long-form instructions as well as ChatGPT.

[@ATLJB86]

This guy is too smart for my brain to process

[@josecastroesq]

That was amazing!

[@theflipbit01]

You wouldn't believe, I was experimenting with Siri and groq api, and I asked the same question on summarizing the "attention is all you need" paper even before coming across this video. I mean I am spooked here, what are the odds of that happening? We humans obviously do behave in patterns. lol

[@limebulls]

A video from you about Avatar AI would be awesome! Haven’t found one yet

[@KodandocomFaria]

I was wondering if you know anything about airllm? I read that this inference is capable of loading a 70B model on gpus as small as 4gb, but I don't saw no one speaking about that

[@The_Questionaut]

Here's a long wall of text about the thing you're interested in. Now I think this is very interesting myself, what I understand is that it allows llms to be used on lower end hardware or something? I used AI to do this. Understanding AirLLM and Its Significance AirLLM is a groundbreaking technique that facilitates running a 70-billion-parameter large language model (LLM) on a single 4GB GPU, overcoming traditional hardware limitations. Traditionally, such large models would require more powerful and expensive hardware, restricting accessibility. **What is AirLLM?** AirLLM stands for a method allowing large language models to operate on smaller, less powerful GPUs. It employs two main techniques: 1. **Layer-wise Inference:** Breaks down the model into individual layers, loading only the necessary layers into memory during inference, reducing the overall memory footprint. 2. **Flash Attention:** An optimization within layer-wise inference, focusing on loading and executing only one layer at a time, further minimizing memory requirements. **Why is AirLLM Significant?** The significance lies in democratizing the use of large language models, making it accessible for personal projects, educational purposes, and small businesses. Practical implications include the ability to run advanced AI models, like chatbots, on smaller hardware, fostering innovation in various fields. **Running 70B LLM on a 4GB GPU with AirLLM Technique** **What it is:** AirLLM is a groundbreaking technique enabling the operation of a 70-billion-parameter large language model (LLM) on a single 4GB GPU, overcoming traditional hardware limitations. **How it works:** The technique employs layer-wise inference, dividing the model into layers and loading only the necessary parts into memory, drastically reducing memory requirements. **Key Features:** - **Layer-wise Inference:** Splits the model into layers, significantly cutting down on GPU memory usage. - **Flash Attention:** Optimizes single-layer execution, further reducing memory needs per layer. **Benefits:** Allows for efficient and effective inference on a 4GB GPU without compromising model performance, bypassing the need for high-end GPUs or excessive RAM. **Applications:** Ideal for scenarios where hardware resources are limited, such as personal projects or low-budget research. This technique represents a significant leap in making advanced AI models more accessible, enabling users with modest hardware to leverage the power of large LLMs for various applications.

[@indikom]

How expensive is that kind of voice conversation using Groq API?

[@JaredWoodruff]

I need this kind of inference speed in Skyrim with GPT :D

[@Wilkbezstada]

lol nailed

[@khalifarmili1256]

yes i did enjoy it, thanks for the video, keep it up ❤

[@Bigjuergo]

What is price?

[@thesystemera]

Damn cool stuff.

[@THE-AI_INSIDER]

Groq api I have been using to use the mistral 8x7 and it is currently letting me use it for free , but until when it is free any idea ?

[@JNET_Reloaded]

you didnt show how to set this up i see the site now what?

[@richarddanielzoom]

SO GOOD !

[@hqcart1]

What matters most is the first token latency, question is does grow has edje on that?

[@sirrobinofloxley7156]

It's amazingly, stupendously and miraculously FAST, isn't it, haha

[@mistrz3998]

Ive applied and no answer

[@Ryan-yj4sd]

How to use JSON mode with the model?

[@cjenkinsiv]

The code is available in the playground.

[@CoderN1337]

amazin video but can you link rep?

[@Tom_Neverwinter]

Its actually depressing that hardware made for int math wasnt made already.

[@PascalMeienberg]

so 5x faster than openAI but the card cannot be used to train your own models...

[@teebu]

nvidia killer. im sure they're also working on a training dedicated card... and if not them, someone else. a lot of companies are going to try to eat nvidias 2T dollar lunch.

[@hqcart1]

nvidia killer is a company that will beat them in: 1. watt/token 2. $/token 3. first token latency 4. total generation time so far i see 4 was beaten, but it's the least important aspect.

[@MARKXHWANG]

257 LPU vs 1 H100? do the math and watch your wallet

[@thierry-le-frippon]

Not too late for Meta to cancel its order from Nvidia 😅😅😅

[@HectorDiabolucus]

We should start using AI for something that is more useful than playing games and generating p0rnographic images. Its not a toy.

[@Leto2ndAtreides]

Groq, so unfriendly about giving access...

[@aaronortiz6268]

How? I haven't tried out their API or service

[@cjenkinsiv]

What do you mean?