Neural Sheaf Diffusion: A Topological Perspective on Heterophily and Oversmoothing in GNNs

❤️ Become The AI Epiphany Patreon ❤️
/ theaiepiphany
👨‍👩‍👧‍👦 Join our Discord community 👨‍👩‍👧‍👦
/ discord
In this video I cover "Neural Sheaf Diffusion:
A Topological Perspective on Heterophily and Oversmoothing in GNNs" paper. The paper takes ideas from the sheaf theory - a branch of algebraic topology - and combines them with GNNs, enriching them with a rich geometric structure (sheaves) achieving provably more expressive diffusion-based graph neural networks!
It took a lot of time to prepare this video and read everything that was necessary as a background reading - check out the resource section below for how to get started!
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
✅ Paper: arxiv.org/pdf/2202.04579.pdf
Truly gentle introduction to sheaves:
✅ A Very Elementary Introduction to Sheaves:
arxiv.org/pdf/2202.01379.pdf
✅ Opinion dynamics on discourse sheaves: arxiv.org/pdf/2005.12798.pdf
✅ Sheaf Neural Networks: arxiv.org/pdf/2012.06333.pdf
Blogs:
✅ Blog accompanying the paper: towardsdatascience.com/neural...
✅ Differential geometry and algebraic topology papers overview: towardsdatascience.com/graph-...
✅ Beginner intro to topology: www.cantorsparadise.com/what-...
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
⌚️ Timetable:
00:00:00 Gentle intro to sheaf theory and algebraic topology
00:12:00 Making it less abstract: examples from sheaf theory
00:23:50 Formal terminology
00:27:40 Sheaf Laplacian dissected
00:40:00 The separation power of sheaf diffusion
00:53:00 Dirichlet energy and converging to harmonic space
01:01:05 Neural Sheaf Diffusion GNN
01:06:00 Results and outro (feedback appreciated)
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️
If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon!
The AI Epiphany - / theaiepiphany
One-time donation - www.paypal.com/paypalme/theai...
Huge thank you to these AI Epiphany patreons:
Eli Mahler
Kevin Stone
Petar Veličković
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
💼 LinkedIn - / aleksagordic
🐦 Twitter - / gordic_aleksa
👨‍👩‍👧‍👦 Discord - / discord
📺 KZbin - / theaiepiphany
📚 Medium - / gordicaleksa
💻 GitHub - github.com/gordicaleksa
📢 AI Newsletter - aiepiphany.substack.com/
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
#sheaftheory #algebraictopology #oversmoothing #heterophily

[@TheAIEpiphany]

Algebraic topology ftw - learned a lot of cool ideas preparing this one! Hope you learn something new as well

[@keeperofthelight9681]

I love the track your channel is taking. More towards geometric deep learning than the mainstream. I am very noob when it comes to algebraic topology manifold etc so your hyperbolic gcn and now this are great stuffs. I would love to learn more so i can implement deep learning architectures in non Euclidean spaces

[@keeperofthelight9681]

Yeah?! Now let’s see if you can combine hyperbolic convolution operation for neural sheaf diffusion is clear crisp colab notebook code xd

[@keeperofthelight9681]

Waking up 4 in the morning and discovering an Epiphany video is a rare experience

[@TheAIEpiphany]

Hahaha here for you my man!

[@vedantnilabh2260]

Great video, really appreciate how well you introduce algebraic topology for those without a background and really try to explain the connection with traditional graph neural nets.

[@francomarchesoni9004]

Thanks for not underestimating your public while correctly assuming that many of us don't know about topology! You generate the urge to find more and more difficult ideas

[@TheAIEpiphany]

Nobody knows topology not even topologists - so it was an easy bet to make (laughs in Grigori Perelman)

[@garydejong8693]

Superb presentation. It feels like you are a helpful colleague getting team members up to speed. kudos1

[@knowledgereact]

Thank you. There was a lot of good explanation of what's going on in the paper. Watched all of it.

[@TarunGupta360]

Watched till the end, although would have to watch it multiple times & read the paper to fully grasp it. It's very visible how much effort you have put into this video. Thanks for making this!

[@rachaelschwartz2289]

Thank you for spotlighting this work! As a researcher in this field, I really appreciate you taking such care and time to explain this paper's utility in an accessible and practical way. Applied topologists often struggle to explain why our work is valuable, so it's awesome to see interest and such positive feedback from other AI researchers. :)

[@TheAIEpiphany]

Thank you!

[@oncedidactic]

12:15 YES great point, these kinds of knowledge-passing are so important for overall advancement, thanks for calling this out!

[@TheAIEpiphany]

100%!

[@fireclub493]

This was amazing! I have followed some of the work (Michael Robinson, Justin curry, and Robert thrust, to name a few great researchers) on computational applications of sheaves, and I had a feeling someone had to be pairing these with NNs to get some really interesting results! I’d love to see more of this type of content in the future, and I really appreciate the amount of work you put into providing good intuition for sheaves for those that haven’t seen them before. If you have any interest in more stuff in the Topological Data Analysis realm, I’d be psyched for some paper reviews in that area. Cheers!

[@linminhtoo]

it took me several weeks to watch the whole video due to my schedule, but I finally finished it today. as i got towards the back with the linear separability part it all started to make a tonne of sense and I was very impressed by the elegance of sheaf theory applied to graphs. amazing job on the explanation and effort!

[@TheAIEpiphany]

😱 respect for the perseverance!

[@Alpha_Lyn]

Thank you for great video! Even though I have b.s. (double) degree in math, modern algebra is too abstract to deal with intuition.

[@TheAIEpiphany]

I think the failure is oftentimes on their side. Hard to decrypt hieroglyphics without the Rosetta stone.

[@diegosorte]

Thank you very much for this video! I’m starting to get more into this field (I’m more like an applied computational scientist) and this kind of explanations help me a lot!!! I watched from start to end :) Now I’m feeling more motivated for reading my topology textbooks 🙈

[@oliveiracaio57]

I watched til the end and I have a suggestion: maybe you should explain a little bit more the choice of the papers you're explaining, why you think they're useful and important etc. I give this suggestion for two reasons: (1) to motivate even more people with a non-math background to study the papers or the math behind the papers because they're definitely not easy and (2) so people with a math background that likes applications, such as myself, can appreciate/understand (even help improving) some real-life problems solved by the fancy math they already know. So, for example, I understood what's in the abstract, but I lack the expertise to see why heterophily and oversmoothing matters in the first place and how useful they're in real-life. The same thing happened in the last video. Why those scale-free networks are useful, where they appear etc? With that said, great video and content, I really appreciate your work and this channel.

[@TheAIEpiphany]

Super appreciate the feedback thanks!

[@Fordance100]

Thanks, great job. I was able to follow along. This was great introduction to the topic for me.

[@TheAIEpiphany]

Super glad to hear that!

[@zbaker0071]

Damn, this paper was DENSE! Excellent video. I felt like it was very approachable (especially after the links!) as someone who has never seen sheafs before. I can't say I understood everything, but I think you did a great job explaining everything.

[@alexanderchernyavskiy9538]

It took me some time but I got the idea of the paper! Thank you for all the links and beautiful explanation. It is so good when people try to use math power rather than just computational power to unveil new beautiful results. Looking forward to watching some more like this! polynomials to solve casual Laplacian equations (Dirichlet setting). hehe

[@TheAIEpiphany]

Heh, I don't think either is better tbh - our brains are also amazingly complex computing machines with far more params than our nets (albeit much more efficient). So I say push both directions.

[@alexanderchernyavskiy9538]

@@TheAIEpiphany makes sense!

[@UziLutzi]

Watched it over the course of two days, but until the end! 😁 It certainly got me thinking and hence I will probably revisit it in case I try to build something based on the ideas presented. Importantly I am not sure if I would even have considered using the idea since I simply did not understand it well enough. So thanks for that and keep up the great work! 🙏

[@connorshorten6311]

Very impressed with the topic! Looks intimidating to me haha, congratulations on summarizing this one -- excited to watch!

[@TheAIEpiphany]

Thanks Connor! Hahah yeah take some time to let it sink in. 😅 I don't promise you'll understand it after my video but I tried my best! 😅

[@olayinkajosiahajayi8330]

Hello. This was great! I watched it to the end and learned quite a lot following your explanation. Thank you so much for taking time out to do this.

[@TheAIEpiphany]

Thanks man!

[@AndyHOTlife]

Thank you very much for going through these papers. Tough geometry concepts explained in a very nice and intuitive way. Congratz!

[@TheAIEpiphany]

Thank you!!

[@johnhart1790]

Nice video. 10:35 homeomorphism

[@superman39756]

Thank you so much for making this video! Lots of cool ideas and useful references packed into one video but you keep the energy levels up throughout and do well on the explanations.

[@TheAIEpiphany]

Thanks I appreciate it!

[@hassaannaeem4374]

Great vid. And great depth for the abstract machinery.

[@janolszewski6161]

The video was great. I watched it from start to finish in one go. Really enjoyed it

[@TheAIEpiphany]

Thanks a lot Jan! Appreciate the feedback!

[@munozariasjm]

Very good content! it is worth every single second until the end

[@TheAIEpiphany]

Thanks a lot Jose!

[@muskduh]

This is amazing. Thanks

[@hoijanlai3362]

Great video! I watched the whole thing : ) but it took me days to pick up all the prerequisites… XD

[@fabszs]

Great job, thank you! Short comment: Please don't add or substract temperatures. 20 degrees + 20 degrees != 40 degrees. It's even weirder if you substract and get negative numbers at the end. Or substract temperatures in equilbrium, which would lead to sth like 30-30=0. You'd rather think and calculate in terms of average kinetic energy per molecule, with the emphasis on average. E.g. think of two (ideal) gases with the same temperature. Their average kin. energy per molecule stays the same when mixing, thus temperature stays.

[@TheAIEpiphany]

Thanks! The main point was to make a mental model that people can relate to - at the risk of being inaccurate - but good point!

[@kazz811]

This seems unnecessarily pedantic and not entirely reasonable. Once you write the diffusion equation, all the molecular information is encapsulated by the diffusivity (the parameter that multiplies the laplacian. At this point Temperature is a coarse-grained macroscopic *field*. Now when you discretize this equation using (say) second order finite differences, you are literally adding and subtracting the temperatures. This is perfectly consistent .

[@robertjulesyoung9994]

I'm new to AI and have a question: we have so many models when it comes to NNs. the question I have, would there be a way to allow n nodes self-organize in order to generate the best performing model (or combination of models) for a specific task? have you heard any kind of study on that matter?

[@TheAIEpiphany]

I saw you found your answer on Twitter - feel free to share the link here!

[@INLF]

@@TheAIEpiphany could you share the link?

[@UziLutzi]

What was the answer found on Twitter?

[@mandyokpoji8787]

@4:01 - that made me laugh.

[@INLF]

Watched it till the end.

[@TheAIEpiphany]

🙏🙏🙏

[@hansriess1435]

Really nice explanations! Watched to the end… I wonder what the sheaf NN would look like if we iterated the laplacian in a single layer and learned filter coefficients (cf. Gamma and Ribeiro). Btw the homological thing is totally a joke!