OpenAI CLIP: ConnectingText and Images (Paper Explained)

[@hmate1119]

This channel is insanely good. Deserves even more recognition. Great work! Subscribed

[@MachineLearningStreetTalk]

This is a really important paper, I suggest people pay particular attention to Yannic's "robustness to data shift" section if you are short on time. I hope we can get the authors on to discuss this!

[@ghostlv4030]

The idea is so simple and so hard to believe it is this effective! Okay, I see, NLP is so useful in vision now.

[@jonatan01i]

Thank you so much for this, especially for not keeping the promise on cutting the video short!

[@ПавелШтыков-х9т]

Man, you have a talent to explain hard things! And your english is awesome!!

[@shengyaozhuang3748]

Interestingly, similar training methods have been explored in the field of information retrieval for searching relevant documents to the given query. So, probably a good application of CLIP could be searching a wanted photo on the internet by using a text query.

[@dreadfulbodyguard7288]

Google images?

[@charrylee3671]

so good video. I understand CLIP after your explanation.

[@G12GilbertProduction]

In a 8 of 20 examples presented in this paper review is really measured by different compilers of models, but not only this same in 20, 45, 60 bites for a 1mm³ pixel outer the third output layer.

[@alteshaus3149]

Thank you so much for this video! It really helped me understand Clip! Best regards from Vienna!

[@MeatFingerSteam]

Absolutely loved the Alec meme, thanks!

[@key2thacity87]

Hey @YannicKilcher /all, it seems like OpenAI is only referring to performance on the class of bananas at 39:05 (figure 13) not that zero-shot CLIP outperforms resnet in general on ImageNet. Earlier in the paper (8:15) they achieve 40% accuracy on ImageNet. Is 39:05, (figure 13) showing 72% accuracy on bananas or overall?

[@p.z.8355]

Do they have a specific strategy to sample the batches ? Maybe sampling totally unrelated captions initially of e.g dogs and planes, then in a later state in training sampling more subtly differing captions of e.g different breeds of dogs.

[@YannicKilcher]

I think it's just pure random.

[@p.z.8355]

@@YannicKilcher merci :)

[@bukovelby]

Just a Brilliant overview!

[@vsiegel]

Trained on "the internet" - so technically speaking, it is a porn classifier, right? Except if it used a separate algorithm for "adult image filtering". Fascinating! (And funny!)

[@naifalkhunaizi7847]

Truly great explanation!

[@ophir1080]

Great video, thanks for sharing! Just one wonder if mine, why are we 100% sure that all these old known datasets are not just subsets of the images CLIP was trained on?

[@jeshweedleon3960]

Imagine this but with more sensory data - audio, video, text, hell any string of bytes even. Wild...

[@ashrafg4668]

Thank you for the explanation!

[@TechNewsReviews]

Very good explanation....👌

[@aminasadi1040]

Thanks a lot for this awesome video! The explanations are very digestible even for a beginner.

[@oflasch]

Great explanation! 👍

[@SaidAzizov-c3l]

Excellent! Thank you a lot!

[@theocachet6496]

Do they check that Ti =! Tj for i =! j with (i, j) indexes of a minibatch? If it is not the case, than sometimes it may have conflict in the contrastive loss (max Ti,Ti and min Ti,Ti in the same computation). Do we agree?

[@44Kokoloko]

Am I understanding this right: The CLIP training results in having both a text and image encoder that are able to numerically represent the proximity between words and image representations, with vectors. These encoders can then be used on different datasets to good effect. In other words, it relies on the findings related to text embeddings (word2vec) to train corresponding "image embeddings" in a way that allows matching an image embedding to a text embedding. Text embeddings having proved to be able to encode relations between concepts in 3d space (king - man + woman = queen), you can then move between text and image representation of these concepts. Does that sound right? Also, what is the pretraining done on?

[@jenishah9825]

I can't thank you enough for making such useful videos.

[@Xaelum]

Just imagine a version of CLIP trained on random KZbin video frames + Title or Subtitles.

[@ThetaPhiPsi]

just for ppl watching this lately: They revised the results for STL-10 in another version of the paper. On p. 40 they write "We updated the STL10 scores from the previous version of this paper after fixing a CUDA-related bug."

[@antonio.7557]

thanks yannic, great video! but the biggest question i havs is how they got this dataset with images+descriptions 🤔

[@akhilezai]

Hey Yannic! I wanna know what software you use to "extend" your PDF with empty space that you use to write notes. Please tell us

[@tsunamidestructor]

OneNote, afaik

[@fayeq1745]

I was also wondering about that and figured out it might be OneNote.

[@akhilezai]

So I found another way to do it. Using latex's includepdf

[@tsunamidestructor]

@@akhilezai you could also use LiquidText if you have an iPad

[@akhilezai]

@@tsunamidestructor thanks! I was sure it was possible on some apps on iPad, but I own Samsung tab s7+

[@uniqued4ve]

I'm missing a bit your critique points here! But thanks, good intro to CLIP

[@ShivamSingh-xf8nb]

Amazing explaination!

[@user-vr3bl6cn9e]

after understanding the paper, how do we approach to understand the code?

[@prabhupadpradhan489]

The dataset which was used for pretraining the model (in the paper it is mentioned as WebImageText) is it made available for public use ?

[@bhavikdhandhalya]

I thought you will explain how those image and words are processes so that they have some connection. No issue.

[@omkarpanhalkar1857]

what is linear probing between 2 visual models?

[@Qumeric]

It's weird that ImageNet-A performance is higher than ordinary ImageNet performance.

[@norik1616]

Could it be, because the images are more artistic ≈ closer to labeled images ppl put on the internet?

[@srinathtangudu4899]

Your videos are so good. Thanks:)

[@frankd1156]

Very good Yanic......

[@Kram1032]

Can't wait for this to be done to, like, entire movies. "Just" take the actual movie scripts as text input and the entire resulting movies (the frames) as image input, and add the modality of sound on top. Could also add a bunch of other production data if available (such as, say, concept art, or voices and music unmixed or even making-of documentaries and interviews or entire books which those movies are based on etc.) Between (such versions of) CLIP and Dall-E you probably could make entire movies from scratch with just writing out scripts, and then refine them by giving some concept art or something. I mean that level is a long ways off I expect - mostly due to how much data needs to be fit into a model that has to be long-time coherent etc. - just the memory requirements as of right now would be quite insane. But *in principle* I think this could be possible. Resource-needs aside, I suspect adding a sound modality wouldn't even be that difficult in CLIP, right? You'd basically do the same symmetric contrastive classification but add a third concept to it dealing with sound.

[@GiangNguyen-of4qf]

best ever video explained Yannic :)

[@florianhonicke5448]

New video from yannic!!! Saved my day :D

[@raphaelsaeed]

Well explained

[@morkovija]

Chuckled at that narrator cut! x)

[@maryamaghili1148]

Thank you for your great work! So is there any way we could find the actual label (text) they have used for training? I need to use this model for some classification tasks that I have, but I am wondering how to organize labels? I have only images with no annotation.

[@yaka169]

How it works is similar to siamese network, or how? I quite confused

[@dl569]

thank you a lot!

[@GUINTHERKOVALSKI]

24:55 "i think prompt engineering will become quite a bit more relevant"

[@Abdulazizab2]

Great explanation! But I wonder how they measure the accuracy of zero-shot prediction, is it by containing the original word of the label only? or some sort of combination as the output of zero-shot CLIP would be a sentence I assume.

[@gocomputing8529]

It is a bit too late, but I'll answer for the future people. From the video the classification is performed by creating a prompt. For example, if you know they are photos, you would say 'a photo of {label}'. As the video shows, the prompt you choose is really important for some applications (datasets)

[@xingjian417]

thanks for sharing

[@eliteari]

great video

[@ranam]

can i make an orc text recognizer with it

[@black-snow]

"random GoPro fallen into a bunch of bananas" xD

[@simonstrandgaard5503]

Mindblown again.

[@jonatan01i]

29:38 Voice borrowed from Josh from Let's Game It Out

[@morkovija]

funny how we all watch same channels

[@h3rtc]

that Alec meme is fire haha!

[@willrazen]

"We'll forgive it"

[@herp_derpingson]

18:20 This symmetric classification looks like a good idea. I wonder if we can use this for all classification tasks in general. 28:40 If you look at the datasets it is weak at. They involve some form of arithmetic. This paper is a big deal. Kudos to the authors.

[@YannicKilcher]

good thought, but if you apply this to standard classification, you always have the same N labels, which would just reduce to the classic crossentropy loss

[@imranq9241]

Is it zero shot if you consider image captioning as a single task ?

[@DajesOfficial]

it is zero shot in terms of not using dataset-specific data. Otherwise it is obviously heavily trained

[@chandrahmmouleb9611]

super Hit

[@emilyme9478]

👍👍

[@antonio.7557]

shouldn't this easily beat imagenet state of the art if you actually finetune it on the full imagent dataset?

[@p.z.8355]

Why do you even need a prompt ? Can't you just use the original label set ?

[@DajesOfficial]

They show in the paper and it is demonstrated in the video that prompt engineering adds 5 percent points to accuracy.

[@nakshatrasingh9202]

Switch transformer, Google. Video please 😭😭🙏🙏🙏

[@Lee-vs5ez]

Better with vision to do nlp

[@harinkumar1073]

44:00 "human model" lmao

[@jointcc2]

"logit" XDDDD

[@chenjieY-z3q]

one of the worst explanation. It's supprising how you can put a lot of clear, well-defined, and neat concepts/terms togather and end up with a presentations that is a mess.