How to make your CPU as fast as a GPU - Advances in Sparsity w/ Nir Shavit

[@YannicKilcher]

OUTLINE: 0:00 Introduction 1:08 Sponsor: AssemblyAI 2:50 Start of Interview 4:15 How the NIR company was founded? 5:10 What is Sparsity about? 9:30 Link between the human brain and sparsity 12:10 Where should the extra resource that the human brain doesn't have go? 14:40 Analogy for Sparse Architecture 16:48 Possible future for Sparse Architecture as standard architure for Neural Networks 20:08 Pruning & Sparsification 22:57 What keeps us from building sparse models? 25:34 Why are GPUs so unsuited for sparse models? 28:47 CPU and GPU in connection with memory 30:14 What Neural Magic does? 32:54 How do you deal with overlaps in tensor columns? 33:41 The best type of sparsity to execute tons of CPU 37:24 What kind of architecture would make the best use out of a combined system of CPUs and GPUs? 41:04 Graph Neural Networks in connection to sparsity 43:04 Intrinsic connection between the Sparsification of Neural Networks, Non Layer-Wise Computation, Blockchain Technology, Smart Contracts and Distributed Computing 45:23 Neural Magic's target audience 48:16 Is there a type of model where it works particularly well and the type where it doesn't?

[@mephilees7866]

These talks are especially important. Thank you Yannic for effort to host it even when you don't have to.

[@buzztwister]

Kudos to Yannic for preparing excelent questions. This opened discussion and we could here very interesting, in-depth answers.

[@ramakrishna5480]

We need more people like nir , with completely different ideas

[@TheDoomerBlox]

They exist, they're out there, and nobody gives a shit so who cares.

[@singuto]

With cultural diversity quickly dying off it's going to be harder and harder to get people like him.

[@ThinkTank255]

We're... far... from having that. Way too many people get haircuts. Okay, that's all the jokes I have.

[@handris99]

This was amazing. Can't wait for it when they do LLMs on CPU

[@ikiphoenix9505]

Thanks for this one ! It deserves more attention !

[@HappyMathDad]

This kind of videos are why KZbin is so relevant today. You can't get this content on blog, paper or podcast. Video is the best medium and KZbin the best platform for it. Kudos!!!!

[@adityay525125]

Great interview. Learnt a lot, thanks a ton Yannic

[@jabowery]

It’s really maddening that the only reason parameter reduction of large language models is getting attention in the ML community is to mimic the time*space savings of the brain’s “sparsity”. Really, if academia had its head screwed on straight, it would introduce Solomonoff Induction during CS101. The concept of the smallest program generating a dataset, as also the most probable model of the world generating the dataset, is sufficiently intuitive for an introductory concept to computation while imbuing great relevance to the subject.

[@RaviAnnaswamy]

Good catch!

[@RaviAnnaswamy]

Solomonoff ranks with the likes of Shannon Turing if not more, and it is a shame most computer scientists have not heard of him.

[@RaviAnnaswamy]

Maybe because besides being strong theoreticians these were also engine builders or engineers

[@RaviAnnaswamy]

More people could relate to their ideas by seeing the machines they built

[@frenchmarty7446]

Those are just two perspectives on the same thing.

[@ear4400]

Really interesting stuff and I look forward to seeing the impact this knowledge has in the field.

[@neuralmagic]

We are trying to productize this knowledge! Give it a try and let us know what you think. The code is on our home page: neuralmagic.com

[@alexandermedina4950]

Thank you Yannick, and Mr Nir, this was a really educational interview, you can see the teaching experience the guest has.

[@neuralmagic]

[@anteconfig5391]

I love this. I had the idea that sequential computation and parallel computation were two sides of the same coin, so you could utilize CPUs instead of GPUs and get the same performance. I'm glad someone proved me right.

[@SimGunther]

For certain algorithms that's more branchy than data flow or so simple that you could easily compute the answer by hand, CPU could be as fast as a GPU. You also need to think about the latency of data transferring back and forth between the GPU and the CPU relative to the data crunching before deciding which batch of data needs to go to the GPU and if data transfer was necessary in the first place.

[@Djjustincase-lb9th]

Thank you for another great interview.

[@ensabinha]

Building an optimal sparse graph is an optimization problem by itself on top of everything… which imply that computing gradients is not so easy… which implies the use of gradient free optimizers… which imply a lot of parallelization during training anyway.

[@KamilCzerski]

Simply amazing. I am an advocate for sparsity in larger grain-size scope for ANNs, with Palm-like models as a starting point! So surprising title turns out to be not a clickbait but an absolutely well-defended claim! And the software is already out there. Also, I can't think of better questions to ask. Brief and exhaustive interview.

[@HappyMathDad]

Amazing interview!!!

[@scottmiller2591]

I put an R library in CRAN 8 years ago that exploits sparsity; one of the demos is that sparse representations interpolate and extrapolate (generalize ID and OOD) much better than full representations, something that current approaches are poor at.

[@rothn2]

Yannic: *Hints at extremely interesting stuff* Nir: "That's the magic in Neural Magic" Did they publish?

[@michaelgoin4760]

Our ML research, models, and optimization libraries are all open-source! Here is our model zoo where you can find the models we've already optimized: sparsezoo.neuralmagic.com/ Here are some recent papers our teams have published that contain in-depth explanations for our state-of-the-art model optimization methods: M-FAC: Efficient Matrix-Free Approximations of Second-Order Information - arxiv.org/abs/2107.03356 The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Models - arxiv.org/abs/2203.07259 Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning - arxiv.org/abs/2208.11580 Also some blogs we've written about the research: neuralmagic.com/blog/obert/ neuralmagic.com/blog/bert-large-prune-once-for-distilbert-inference-performance/

[@Finkbom1]

Excellent interview and speaker many thanks Yannic and Nir

[@MikeySalinas]

Great questions and awesome overall interview

[@black-snow]

Interesting interview. I really hope Doc Ock is gonna bring us excellent sparse models.

[@NeoShameMan]

Unless spiderman stop him first

[@aaaa9r]

Doc Ock, I knew he reminded me of someone 🤣 I'm really hyped by this project. Very interesting stuff.

[@deeplearner2634]

Interesting forecast and imaginatipn of the future Really hope DL will come up with a new paradigm in the near future

[@DavidSaintloth]

What is The goal of a shallow and broad (sparse) algorithm? It is to utilize minimum energy in order to connect or Cascade firings between neural nodes that correlate with the features of entities undergoing comparison with incoming experience in the sensory modality that the particular network of nodes is mapped to capture. It's basically an energy minimization algorithm that over time allows a maximization of the probability that a given pattern stored in the network can be matched against patterns within the data stream of incoming sensory data. Selection of the pattern is variable depending on intrinsic low-level attributes known as salience factors in biological systems these factors are autonomic (food,sex,comfort) and amygdala(emotion) driven. I formulated the salience theory of dynamic cognition and consciousness around the above hypothesis around 2008 and published it at my blog, 5 years later... Going to implement POC after I retire from the rat race.

[@rmajdodin]

Very insightful and inspiring. I think I've got ideas for my next project.

[@TeodorAngelov]

I appreciate the insightful and straight-on answers :)

[@willnutter1194]

Maybe a naïve approach, but I was seeing parallels between traffic and NNs, with the ability to turn off roads dynamically to make efficient pathways. Sparsity surely plays some role here.

[@kepplerM]

So much knowledge! Thank you ;)

[@mpjstuff]

I was just talking about this idea of using a large database of pre-computed matrix algebra -- a good terabyte should be able to cover perhaps 99% of the most common inputs and outputs in a lookup table rather than computing them, and also, macros for model shapes (sort of like shape primitives). This of course is created by having an NN "watch" an AI and find efficiencies and places where a macro might replace a complex computation -- and, if you abstract a shape or texture from color and lighting, that should help. In fact, it makes sense to keep certain parts of what you do separate until the end as they will have different optimizations. Coupled with AI to make "good enough" inferences to do 3D visualization. It should be able to take a few samples and "guess" how a large area will look given a few examples -- so, render 1 in a 1,000 light bounces or one in 100 pixels -- and, should help with animating between key frames -- and, when there is AI generated imagery, perhaps reduce the vectors in an animation in 3D space to 16x16 -- and, after computing a few frames, then estimate where the vectors would be -- this allows you to collapse the precision and computation to an 8bit space. And, in terms of AI that uses random noise, why not use estimations and reduced computations or even compression data to reduce the amount of noise you are creating a delta from? There should be a way to look at inaccuracy or estimates to find a way to reduce computation as if you were working with noise.

[@Femador]

This! Thanks a lot for your work!

[@Evan490BC]

Absolutely beautiful!

[@Ride-Tahoe]

Can this be applied to CPU gaming models like Microsoft Flight Simulator where the CPU can be a bottleneck?

[@simonkotchou9644]

Well this guy is for sure on the right track, the KAN paper just shocked us all

[@neuchesnook4181]

Check the volume intensity! i had to lower the volume when you spoke and raise it up when not. (Just a bit)

[@TiberiusSkywalker]

Can self play be used to optimize sparsity?

[@blinded6502]

It'd be nice to be able to just subdivide latent space of a neural network into separate modules of sort? Like, just imagine downloading concepts of what a flower is into your neural network from either the web or just loading it from the disk and into the RAM, thus minimizing the amount of RAM overall required, since you can unload stuff you don't need

[@MHTHINK]

Fascinating discussion. I agree with almost everything I knew enough about to have any opinion, but I think neuromrphic computing is more important than Nir seemed to indicate. Since we're looking at a 10^6 level or in efficiency for the human brain to achieve similar practical compared with current compute architecture, I figure even design with 1000x of refinement to come could be 1000x better than current compute. I wrote a paper on it recently, but still exploring AI as papers get published.

[@sean_vikoren]

I noticed for the first time that bad video makes it hard for me to understand what a person is saying. Right, so I forget that that I am a bit deaf from loud rock and roll. But I thought I would mention it since it's probably a few percent of your biz.

[@markhampton3614]

There is a lot of modularity in brains too, is Neural Magic combining sparsity with modularity? It might be interesting to support RISC-V because it can be customized more easily than ARM. The interview was very well done.

[@tedarcher9120]

Maybe combine memory and compute on one multi-layer risc-v chip

[@ericpmoss]

I'm not sure if it was meant that way, but the "10% of the brain" idea is a mis-reading of the original work. The researchers only understood maybe 10% of the regions of the brain that were lighting up with certain stimuli. It was not that only 10% of the brain was being used.

[@coemgeincraobhach236]

Really interesting discussion

[@dr.mikeybee]

I've been amazed by how well large models run on an M1 Mac's CPU's cores because of integrated memory.

[@maloxi1472]

That's only a fraction of what neuromorphic chips will be able to do... and we haven't even begun to scratch the surface of unconventional computing architectures. I remember having to pick up my jaw from the floor when l read Merkle's paper on a 1cm^3 reversible chip design capable of cranking out a ZettaFLOPS of compute per Watt (more powerful than the current Bitcoin network for less energy than your smartphone). Until I see a concrete implementation of such an energy efficient design, new chips are gonna look pretty banal to me 😄

[@SimonJackson13]

Sparse makes a delta from the dense. Does batching a few deltas help add some sparse training? If you remove some of the carry propergation on the multiply the clock speed can go up. Who cares if the weighting multiply becomes a bit statistically trendy?

[@Phasma6969]

Would be cool if we can combine the compute of CPU and GPU to calculate output...

[@TheGenerationGapPodcast]

Sparsity is a promising neural network architecture for blockchain on-chain computation.

[@ulamss5]

This sounds very closely related to 'weight agnostic neural networks'

[@asrjy]

Is it similar to what TensorFlow lite does? IIRC it also uses sparsity

[@mjohnk11]

Yes, somewhat! SparseML enables sparsification and leveraging better algorithms for higher sparsities, performance, and recovery. DeepSparse enables utilizing that sparsity for inference speedup on X86 CPUs

[@codonology]

Very Cool and hopeful system for future-AI-for-everyone

[@cem_kaya]

Computing NN's depth wise (in Tensor columns) seems very interesting

[@carlotonydaristotile7420]

Super interesting talk.

[@АлексейТучак-м4ч]

you guys might be interested in this: "SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systems" they beat Tesla v100 with 44 core processor in unusual settings

[@supremebeme]

Super trippy conversation

[@thegistofcalculus]

It is not entirely clear if he just aggressively prunes or actually increases the parameter count but only executes a fraction of the network. It should not be too hard to train a large dense network to switch off parts of itself during execution depending on input but the real computational savings would be realized with flexible CPU execution.

[@Rizhiy13]

Could sparsity help with decentralized training, e.g. Hivemind? If only a part of the network can be evaluated depth-wise, can different machines evaluate different paths and train in parallel that way?

[@revimfadli4666]

Ooh that's interesting

[@anteckningar]

GPUs can do sparse models. Just look at graph networks. The issue is rather the optimization method, we don't get gradients if we don't backprop every possible connection

[@KW-jj9uy]

Exciting to see large transformer models on cpu. Large parts of the model do not get activated for each run

[@tedarcher9120]

What do you think about Numenta?

[@Georgesbarsukov]

I've always been interested in the idea of training a large model, then training a small model to learn to mimic the large model. Obviously that creates a bit of a covariance shift since it'd be trying to predict what the larger model outputs instead of what the training data. The benefit though would be a more easily understandable dataset and a faster model.

[@Georgesbarsukov]

An extension to this idea that I was thinking about applying at my FAANG company was to create a cheat model, one that can see more data than is available, then train another version that can't see the cheating data, but you're able to use the cheat model to label or generate more data for the non-cheat model.

[@shibash1]

@@Georgesbarsukov It looks like finetuning on pre-trained model for me)

[@revimfadli4666]

​@@shibash1 or bootstrapping a la actor-critic

[@GoriIIaTactics]

That's literally just teacher student model. Maybe give your ideas a quick search on Google before thinking you had a revolutionary idea

[@Georgesbarsukov]

@@GoriIIaTactics I didn't say it was a revolutionary idea. I mentioned it because I recall a paper released from Stanford where students were able to create a 'Google translate' clone model. I don't understand why you need to be antagonistic about it.

[@IvanGarcia-cx5jm]

He mentions that the brain is very sparse and uses little computation on input. But I have also heard that the brain spends a lot of energy on visual tasks. Like in computers visual tasks are computationally expensive. I suspect that vision tasks require some level of density, because after all you are processing a set of images (2-D dense arrays) with a lot of data. Even when you are paying attention to a small part of the image, still the data has some level of density.

[@blinded6502]

Hasn't that been already mentioned in the video?

[@tedarcher9120]

Metaneuroscience is I think the future. Using neural networks to understand neural networks

[@NeoShameMan]

It's always cool to see two AGI talking to each other

[@neuralmagic]

🤖

[@player75826]

I see no artificial, only general intelligence in combination with lots of creativity, ideating, and combining of concepts from multiple fields that were deduced by perception in the real world, which were then transferred into the AI field, and those ideas that were transferred bring insight into a field to make something that will benefit not just a single corporation, but the world.

[@Aldraz]

I was excited, but disappointed to find out my Zen 1 CPU is unsupported lol. And also it's more suited for Linux right now, I am not sure how well it works on Windows. Probably doesn't, only in docker or something like that.

[@ClydeCoulter]

If the inputs to a part of a layer makes no changes to nodes value(s) in a part of that layer, then there is an opportunity for optimization.

[@thet0ast3r]

lol the audio transkription thing was r3kd by openai whisper. 🤣

[@ericpmoss]

So, given the memory throughput advantages of the Apple M1 Ultra architecture, what kind of match is this software for this CPU? I mean, 128GB RAM with 200-400GB/sec must be a dream match among OTS components.

[@dr.mikeybee]

Can you sparsify the big Bloom model and put it on Huggingface?

[@lizardy2867]

Time for evolutionary algorithms!

[@neuralmagic]

😍

[@zerothprinciples]

Many great ideas here

[@ChibatZ]

was wondering how exactly 176 Billion Parameters fit on your desktop; is this calculation about right for the argument?: 176 Billion Params * 2 Byte/Param (Because of BF16 Format or similar) * 0.05 (because of 5% Density) = 17.6Billion Bytes = 16.4 GigaByte and 16.4 GigaByte fit or your desktop easily is this calculation fair?

[@billykotsos4642]

Their deep sparse engine can't be used in production right?

[@billykotsos4642]

The comments about them targeting the edge are really interesting.... imagine you get a Jetson nano.... and you get better performance out of your AI by using the cpu cores rather than Nvidias GPU.... insane stuff...

[@andrewowens5653]

Yannick, it's nice to see that you shaved today. 8-)

[@yes-vy6bn]

wen sparse stable diffusion 👀

[@rothn2]

Well, PyTorch DataLoaders and TensorFlow could probably do some of this pipelining through inversion of control by putting it in a framework...

[@JTMoustache]

Boom !

[@AviPars]

Skip to 2:05

[@billykotsos4642]

Inference can work on CPUs it seems... But training in def bespoke hardware territory

[@neuralmagic]

It for sure is. We are at an early stage of exploring training on CPUs. Subscribe to our newsletters on our website to stay in the know when we release something!

[@HappyMathDad]

The story of the interview. Is that GPU creates base models. And CPU executes. That CPU + GPU question for now seems to be how much model evolution does the application has.

[@DonCat-sc3qo]

Amazing guy. 😮

[@ramakrishna5480]

Scientists like nir give hope to future researchers who don't have unlimited computing resources

[@HappyMathDad]

Wow. The sql db story all over again. Column storage!!! This is so crazy. Clearly sparsity is the biggest difference to the brain.

[@novantha1]

Hm.... Sparsity, on reflection, seems like an obvious solution. If you imagine a fairly simple neural network, a Boltzman brain, it essentially the opposite of a "sparse" architecture in that everything is completely connected...But if you were to analyze any problem that you're dealing with when using the Boltzman brain, odds are good that 20% of those connections are doing 80% of the work. That kind of makes me wonder if you really have to have "full" connections. Like, if there's a connection that's important to have, but isn't important to use 100% of the time, could you fire it, say, 20% of the training runs?

[@mpjstuff]

I think this is a great example of how the human brain works with very incomplete data -- I myself do a lot of SPARSE thinking. ;-)

[@gleleylo]

Thanks for the video! Take a tip on "donate 4 fun"

[@mattizzle81]

Unfortunately didn’t hear much on whether this currently works in applications like a GAN or Pix2pix which have a hard time running in real time on a CPU. Object detection and image classification already run in real time

[@4.0.4]

Imagine if you could buy VRAM in sticks? That's a real limit on GPU compute.

[@sophontec2822]

To my understanding, the current DL model with layers is already sparse.

[@jahcane3711]

It would have been really nice if you turned up his volume or reduced yours for the video so that i could hear him more clearly without being blasted by your voice :)

[@digitalkind]

Sparsity vs in-memory computing?

[@neuralmagic]

The combination of both is the "magic" behind Neural Magic 😇

[@socaity-ai]

I miss your paper videos. The interviews are way too sparse in information.

[@dislike__button]

hehe

[@neuralmagic]

Pun intended? Check out our website for all the papers that back this research up, including the code to replicate the results on CPUs!

[@socaity-ai]

@@neuralmagic Yep the pun was intended. I like your research and intention. I know the pain of deploying ai models and speeding them up. I used them all: coral tpus, openvino, ncnn, tensorrt, Nvidia jetson but in my opinion none of them is really mature enough. In addition with all the libs sparse models are bringing no inference speed ups just model compression. So it's great to see people working on it! My point is, that Yannic would be able to explain all the content of the 50minute video much clearer, more entertaining and also in more detail in less time. The information density of his paper review videos is quite high while the information density of most of the interviews is quite low.

[@djmips]

But can't you just use sparsity on the GPU and you're back to square one.

[@neuralmagic]

We've seen NVIDIA show that the A100 GPU gets a 1.25x speedup with sparsity on a BERT-Large model. We were able to replicate these BERT-Large results but haven't been able to see this performance for any other models. CPU execution shows the benefits of sparsity across many models at more than 5x improvements. See our website for ways to test this out and replicate it with your own data!

[@michealkinney6205]

I've been thinking about this a bit lately and I think we are talking about conceptual knowledge (sparse) over that of repetitive/regurgitated knowledge (dense, memorization, often taught in schools in order to take "quizzes" in order to "prove knowledge") which is easily cheated with a transcript of the learning material now that everything is "taught" online. Conceptual and therefor sparse knowledge, with select dense key knowledge (like tokenized phrases, dates/times, etc.) I think would be the way forward for AGI (much the way that applicable knowledge in human brains works).

[@vast634]

There are many operations that are faster on a CPU than on a GPU, if they are not compatible to the way GPUs process data.

[@DigitalNomadOnFIRE]

You know you're supposed to use a green screen to overlay an image on. Right? We can see your green screen.

[@AustinGlamourPhoto]

You guys really need to figure that the brain isn't digital. It's works with analog waves with complex data flowing over carrier waves.

[@kostian8354]

At 40 he is describing apple M1 family chips

[@wadahadlan]

As above so below - sparcity

[@synthoelectro]

optimization should have been the key from the beginning.

[@synthoelectro]

@@things_leftunsaid should one agree upon it though? Case in point, MS loves to release broken software. They've been doing it for years, and yet lately it's even more broken. Seems counterproductive to me.

[@sunorcio3901]

​@@things_leftunsaid i suspect that means nothing

[@zackbrandigan9209]

How realistic is training on CPUs? If you want to get rid of GPUs you need to takle sparse training on CPUs.

[@barrettvelker198]

Imagine if this were on Huggingface

[@mjohnk11]

We have integrations with HuggingFace! Check them out here: github.com/neuralmagic/sparseml/tree/main/integrations/huggingface-transformers github.com/neuralmagic/deepsparse/tree/main/src/deepsparse/transformers

[@sathyanarayanankulasekaran1674]

Moore's law, we will have more n more computation in the coming years..so this problem might not be as tedious as we see today...but a grt intention to make AI more democratic

[@stackoverflow8260]

The other one is, "At memory computing".