OpenAI CLIP Explained | Multi-modal ML

[@ricardojung3849]

Thanks for reporting, explaining and lastly opening up recent ML! I found clip to be very interesting since I always frowned at the lost potential of two different embeddings being arbitrary and methodically separate. This is huge!

[@jamesbriggs]

yes there will be plenty more on CLIP and other similar models very soon - some of stuff I've built (and will demo) is awesome and nothing more than zero-shot CLIP, excited to share!

[@mszak50]

This was really excellent - some of the pieces are starting to make sense

[@konichiwatanabi]

Thank you so much for this great walkthrough! Looking forward to more

[@DallanQuass]

Great video! Looking forward to your next video diving more into using CLIP for zero-shot classification!

[@jamesbriggs]

Me too, it's fascinating. Thanks for watching!

[@ismailashraq9697]

This is amazing James. Thanks for the detailed explanation. I am excited for the future CLIP videos 🙂.

[@jamesbriggs]

Thanks Ashraq! As you know, I'm excited for them too

Thanks James, very good video about CLIP. Funny thing is that you display twice the cos_sim, so the second time it is not the dot_sim which is displayed. And you fighted to find any difference between the two similarity matrices. LOL 🤣

[@jamesbriggs]

ah did I do that, oops 😅

[@justinmiller7150]

Great video. I think you may be plotting the same graph twice though (cos sim). In practice it is almost the same though it would seem.

[@adrianarroyo9839]

Nice video and explanation! I think on min 28:45 you plotted cos_sim instead of dot_sim!

[@AdeleHaghighatHoseiniA]

Thank you for the good explanation, if we have 2 different embeddings like texts and 3D images, we can use CLIP to predict images?

[@abdirahmann]

is there a hosted API for clip where you can provide your image data and get the vectors instead of having to host it yourself, kinda like how you give an input to `ada-002`?

[@behnamplays]

Excellent content! As a suggestion, can you please keep the images/diagrams a bit longer? They move pretty fast in the video, which means I'll have to rewind the video every now and then.

[@jamesbriggs]

Sure that’s great feedback, thanks!

[@valentinfontanger4962]

Excellent video

[@debashisghosh3133]

Really liked the content...thanks for sharing

[@jamesbriggs]

Thanks for watching!

[@anantzen171]

10:23 I believe CLIP is an abbreviation of Contrastive Language Image Pretraining

[@Gabriel-ey5ky]

Great video really ! I have just one thing to say, you should let the images longer in the screen I had to pause the video multiple times to be able to understand them

[@jamesbriggs]

Thanks Gabriel, I head the same from another viewer - will do this going forwards :)

[@PurpleRivar]

Thanks. It is very informative. Can you pls explain and teach us how to do fine tunning on the custome dataset. Pls

[@mvrdara]

Excellent explanation! We can build a KZbin video search engine powered by clip, perhaps you can iterate on the Nlp KZbin search video you did?

[@jamesbriggs]

That's a great idea, but it might be difficult for KZbin videos where it is just someone talking, as the image embedding would just be something like "a person talking" Possibly it could be interesting to embed both the text + images with CLIP, and maybe even an averaged text+image embedding for parts of videos where both the speech + image are important. I will think about this more, it's a great idea so thankyou!

[@sharanbabu2001]

Nice explanation!

[@shaheerzaman620]

fantastic stuff!

[@dancinghoka]

Thanks a lot!

[@pyalgoGPT]

Plz post on Deep Reinforcement Learning tutorials & projects with python !

[@jamesbriggs]

Eventually I’m sure I will, RL is very cool

[@debayudhmitra9432]

can you give the github code please

[@mackenzieclarkson8322]

Transitions are too flashy and triggering to my eyes. Good explainer however.

[@davide0965]

Too much talk and very few illustrations