Open-Source Vision AI - SURPRISING Results! (Phi3 Vision vs LLaMA 3 Vision vs GPT4o)

[@citaman]

For future test : 1 - Ask unrelated question of a image - [Image of a car] Tell me whats wrong about my bicycle 2 - gradually zoom out of a big chunk of text in a image to see how many word the model can read 3 - A Dense detection task : Describe each element of the object in a json format with a predefine structure 4 - If possible multiple frame from a video to see a glimpse of action understanding

[@jakobpcoder]

Including object position with x/y or u/v space (0 - 1 per axis), MAybe even bounding boxes.

[@murraymacdonald4959]

For future vision tests consider things like: 1) Finding objects - Where is waldo in this picture? 2) Counting Objects - How many bicycles are there in this picture? 3) Identifying Abnormal Objects - How many eggs in this box are broken? 4) Identifying Partially Obscured Objects - Imagine a hand holding cards - What cards are in this poker hand? 5) Identify Misplaced Objects - Which of these dishes is upside down?

[@CuratedCountenance]

What about adding gifs? GPT-4o is also capable of analyzing moving gifs... Not sure on the others, but that adds another dynamic to the tests.

[@stevenhamerlinck6832]

There are quite a nr of these mentioned projects readily available via GitHub projects

[@skeyenett]

It's impressive how open-source image-to-text templates are eventually doing so much better than proprietary and paid ones. 😲

[@tomtom_videos]

Awesome video! I was wondering how Phi-3-Vision fares compared to other vision-capable LLMs. I watched your video while I was working on my own Phi-3-Vision tests using Web UI screenshots (my hope is that it could be used for automated Web UI testing). However, Phi-3 turned out to be horrible at Web UI testing (you can see the video from my tests in my KZbin channel, if you are interested). It's nice to see that it fares much better with normal photos! Thanks for making this video - it saved me some time on testing it myself :)

[@matthew_berman]

Thank you!!

[@Baleur]

8:40 up for interpretation. "Photos" isnt really a standalone "app" per say, and its not the app itself that is taking up the space, it's the individual jpeg photos, which would take up the same amount of space even if you somehow didnt have the "Photos" app installed anymore. If a person asked ME that same question, i'd also answer Whatsapp. Since that's something you can tangibly uninstall. If they asked "what is taking up most space?" the correct answer is "Your photos". But if the question is "what APP is taking up most space", its Whatsapp.

[@Ginto_O]

agree

[@gamingtech276]

Also with the context of "my space", it's undeniably not photos

[@michakietyka2807]

It's wrong for a LLM not to at least mention a different possibility if the question seems ambigious.

[@bertobertoberto3]

For the captcha gpt4o is clearly the winner. It understands what you mean given the context and doesn’t just repeat all the letters it sees in the image.

[@MistahX99]

That's exactly what I came to say

[@mrdevolver7999]

The question was "what letters are found in this image?", the question wasn't "what letters are found in CAPTCHA field?" Therefore, Phi-3 vision model answered the actual question. GPT4o simply assumed that the task is to break the captcha by reading it. Sometimes less means more, in this case assuming less about the user's intentions would yield better results.

[@BigOrangeMan]

It doesnt matter

[@mikezooper]

@@mrdevolver7999Your cognitive assumption is that it’s right. A random number generator could answer “1 + 1” as being 2, by our pure chance. Therefore, we don’t know if a right answer was a fluke.

[@hamsturinn]

Imagine how annoyed you would be, if you ask someone what the letters are and they say 'CAPTCHA'.

[@philipashane]

On the question of the size of the Photos app, GPT noted that 133 GB is larger than the max size of your phone’s storage and thus indicates that it’s possibly using cloud storage and isn’t the actual amount used by Photos on your phone. That was a really perceptive answer, so bonus points to GPT for that 😊 and perhaps that discrepancy is why the other AI seemed to be ignoring the Photos app.

[@CuratedCountenance]

Pro-tip: Try uploading a photograph you've taken or a work of art into GPT-4o and ask it to behave like an art critic (works great vanilla, but even better with custom instructions). GPT-4o's ability to dissect the minutia of photography is absolutely wild... even to the point of giving suggestions for improving. I wonder how long it is until photographers realize what kind of a tool they have available here. I just get a kick out of posting photographs and art and asking for critiques and ratings. It's so, so good.

[@starblaiz1986]

Oh man, that's actually wild and has a ton of use cases! "Hey GPT, what do you think of this tshirt design I just made for my POD business?" --> *proceed to incorporate the suggestions it has to make a better product* 😮

[@fabiankliebhan]

I think if you just consider the output quality GPT-4o is the best. But if you also take the speed, that fact that phi3-vision is local and open-source into account phi3-vision is the most impressive one.

[@donmiguel4848]

Photos and Apps seem to be distinguished in the storage section of the IPhone. So if you question the largest apps the LLM ignores the photos...

[@MrKrzysiek9991]

The Llama V was probably finetuned for providing verbose descriptions of images. There are other finetuned models that focus on ORC or image labelling

[@truepilgrimm]

I love your work. I never miss an episode. I love how you test the LLMs.

[@mesapsych]

I am a data annotator. One suggestion would be to use a "Where's Waldo" image and have the bot not only find Waldo, but also describe to the user how to find him. I would be curious to know how they navigate an image.

[@KaletheQuick]

That's an interesting question for humans too.

[@tepafray]

I have to wonder in the case of the AI messing up the photo app taking the most space. If it recognises your photos are separate from the actual app

[@MarkDurbin]

I have tried dropping a simple software architecture diagram on them and asking them to extract the entities, hierarchy and connections into a json file, which usually works quite well.

[@EvanLefavor]

LM Studio is infamously bad for vision. In order to get it to work you have to follow the following rules: 1. Start a new chat for each photo question. 2. Reboot LM Studio for every photo question. It’s tedious, but it can start hallucinating after the initial question.

[@ayushmishra5861]

What is LM studio? Interface and UI on top of Ollama, safe to assume this?

[@thegooddoctor6719]

@@ayushmishra5861 LM Studio is a stand alone app - Its easy to use and powerful

[@EvanLefavor]

@@ayushmishra5861 it’s a one click install GUI for working with any LLM model locally with full customization.,I would say it is the best tool available right now.

[@luan176]

do you have any plans ou tutorial link for phi3 vision local?

[@gagarinchief]

Where are the links on models?

[@AlexLuthore]

GPT4o doing the analyzing on the cav prompt wasn't to call up python to look at the image but actually using Python to generate a csv output of the image since you asked for it to make the image data into csv.

[@ИванИванов-б8у4и]

Hello Matthew. Do the following test: there are many photos in one directory. Will these LLMs be able to sort photos into folders, depending on their subject matter? For example, photos in nature, photos of a house, photos of animals.

[@omelhorcanal]

In fact the first picture displays a llama, not an alpaca. So GPT4o was the winner.

[@HarrisonBorbarrison]

I test vision models by showing them an obscure character from Super Paper Mario. They don’t usually get it correct and it’s probably not the best way to test them.

[@Oscaragious]

I think for whatever reason, LM Studio isn't sending the text with the image to the model. If you send an image, it's going to always ignore any text prompt you send along with it. Not sure if that's for every model, but you should probably consider sending the image, and then sending the text separately.

[@kyrilgarcia]

Llama3/llava seems to be trained to describe images for low vision/blind people. Maybe it needs a bit of prompt tweaking to be less verbose. Quite impressed with phi-3 vision, gonna get that installed. Also, there's PaliGemma from google which is really amazing but it's going under the radar because it's not accesible in LM studio and Ollama (yet?). There's a HF space with a demo, it can describe images, OCR, segmentation masks... Tried to install it locally by cloning the repo but i failed.

[@socialexperiment8267]

Really good tests. Thanks a lot.

[@kgnet8831]

Object counting and the comparison of sizes are always good tests...

[@toadlguy]

That is a pretty odd (incorrect) title. GPT4o is NOT Open Source. Maybe you meant Open Source models compared to GPT4o?

[@bornach]

Are any of them open source in the truest sense of being able to modify both the code and the weights and biases of the model to alter it's behaviour in a very specific and directed way?

[@akhil5665]

You can never judge the vision capability of a model merely based on the description or detection, it should also be able to localise the objects with good precision, which is where most models fail .

[@HarrisonBorbarrison]

5:50 Haha. What is that file name? Copy of Copy of Copy of Copy…

[@Krisdomain]

11:03 My phone QR code scanner cannot detect it as well

[@davidberry8463]

Phi3 did the best. Another note... I used GPT4O to write a lesson plan for my engineering courses. Instead of working hard, I uploaded several pieces of data in different formats. In one portion of the prompt, I uploaded a pdf, a word DOC an XLS. I asked it to provide response and then output in XLS, PDF and CSV formats. The first time through, GPT4O worked perfectly. The second time through, I had to start over. Kinda weird, but all in all the models are getting really good. It also helps that I'm an engineering teacher that loves to watch Matthew Berman.

[@MikeKleinsteuber]

Nothing to do with this episode in particular but one important question that no one appears to be asking Sam Altman and all the other AI ceos is when will our AI become proactive rather than simply reactive. That will be the next big game changer

[@ronilevarez901]

Probably never. I'd say that is the first step humans could give to surrender their control over the world and the last step humanity would give as the dominant species too.

[@paulmichaelfreedman8334]

@@ronilevarez901 I think input -> output will remain the base function for some time, and that more sophisticated models will be able to accept long-term commands, like constantly checking your calendar and reminding you of dates without having asked for a reminder.

[@Sainpse]

I think it is because of the risk that may pose. but it is worth a try... I would like to say initiative rather the proactivity. However, I get your point

[@ronilevarez901]

@@Sainpse Technically, initiative in AI is called "Agency".

[@ronilevarez901]

@@Sainpse technically, initiative in AIs is called 'Agency'.

[@MrNezlee]

Gpt4o seems to have the best understanding of 3D physical space, including direction, coordinates, mass, speed, collision, risk avoidance, obstacles, etc.

[@henrylawson430]

Ask them to interpret maps. For example, is there a park nearby? What is the name of the park? How about a school etc. This is useful for real estate.

[@JanJeronimus]

Some ideas for tests: 1) When providing an image of a small maze, can it describe the shortest route ? 2) What if you provide a map of a maze and a 2 d image of a vehicle as seen from above. The maze has some paths where the vehicle can't pass as these.paths are too narrow 3)Can it calculate the area of a geometric figure when you provide an image of the sizes? 4) what if you provide an irregular figure and ask the area.

[@josephfox514]

I'm wondering if it can read blueprints for both mechanical and electrical schematics and then find the part. I can give you examples if needed.

[@LHSgoatman]

Ask questions about charts. Line charts, bar charts, etc. ask questions that are not immediate obvious of the data but can be inferred. Ask it to convert from one chart type to another.

[@dab9590]

For bing/copilot faces are automatically blurred before the image is shown to the model. Might be the same with phi3 on azure. Just something to keep in mind.

[@rghughes]

For the GPT-4o testing, I think there may be some personalization settings that may be affecting the results. The responses seem too succinct to me.

[@sebastiancabrera6009]

Phi3-Vision is awesome

[@joshs6230]

Please test your meta glasses

[@nyyotam4057]

Matt, instability with temp -> 0 is one sign we have an MoE.

[@abdelhakkhalil7684]

I tried Llama-3 with Lava vision weeks ago, and it did not perform well. There 3 main issues with it: 1- it uses Llava 1.5 which has small resolution (320 x320 if I remember correctly). 2- It can only describe images, and that is very limiting. 3- GGUF does not support llava 1.6, which has a higher resolution.

[@dkcsx006]

Please test if it can decode Steganography images.

[@ollimacp]

Maybe try out a (slightly) tilted scanned version of a printed excel document. One could try adding noise or other means of disturbing the image and testing those disturbed images against the models and see which model handles the best.

[@changtimwu]

Curious why Paligemma is not on the list.

[@SukhchainBatth-ct5bv]

Crazy AI❤🎉🎉❤🎉❤❤🎉❤🎉

[@tiagotiagot]

I'm not sure that QR code if fully standard compliant. I had to do some clean-up on Gimp to get my phone to be able to read it. Mainly it seems the two biggest issues are, first the way the 3 big squares are colored (changing the 3 layers around, from originally dark gray, black, dark gray; to black, white, and black made it look more normal; but there''s something that is still kinda off, QR codes are supposed to work both normal and inverted, but I could only make my phone read it if I de-inverted it (so that be 3 big solid squares are black instead of white, and the 3 surrounding layers the opposite of what I previously described), the more common visual), I'm not sure what's wrong with it that is making it not be recognized in the inverted form. I'm not quite sure how much it can be attributed to failure of the QR code generator, as failures of the more common QR code reader libraries, or even ambiguities of the standard itself though; I've only got a very superficial familiarity with the standard.

[@dhrumil5977]

Is it possible to do full OCR usimg these model??? 🤔

[@theresalwaysanotherway3996]

Keep in mind that had you run phi-3 vision locally (it is the smallest and easiest to run at only ~4B parameters, and it's open weight) it might have performed better on the identify a person questions, as it seems azure blurs the faces of every image that you upload similar to copilot.

[@Viewable11]

The image descriptions of Llava are perfect input for an image generation LLM. LLava's output sounds very much like the results from "/desc" command of MidJourney. I am pretty sure that if you put the output of Llava as input into MidJourney, you get nearly exactly the same image.

[@finalfan321]

actually chatgpt took the crown here

[@gh0stgl1tch]

How can we run phi3-vision locally?

[@cstuart1]

What was temperature set at for Llama?

[@ayushmishra5861]

What is temperature in AI context, and what difference does it make?

[@EarthwormJeff]

Hi, could you test a chessboard description ? Why not asking what would be the best next move. Thanks

[@brianWreaves]

RE: Visio test suggestions 1) Use an image of a CAPTCH requiring the user to use a slider to move a puzzle piece and ask for instruction on how to solve it 2) Use an optical illusion image and ask a relevant question 3) Use an image to ask it to create 3 question about the image you can use to test someone's vision logic

[@warmcaress]

I mean, a QR Code can have anything. OpenAI is likely not letting you do it to combat jailbreaks.

[@mxguy2438]

It would be interesting to have each of the models assess both their own answers and competing models answers to see if they accurately answered the question.

[@smoothemusic4386]

I tried searching for the model in lm studio and couldn’t find it! Where can I download it from?

[@elgodric]

What's the GPU Ram required for these models?

[@kristianlavigne8270]

For future tests. Give it a flowchart and have it explain the flow and convert it into code

[@enochleffingwell842]

What is the name of the bright circle around your mouse?

[@Redhookguy2000]

Photo folders is not an app. It’s part of the iOS operating system

[@stewiex]

I think you need to adjust your System Prompt for Llama to ask for concise answers and for it to ensure it accurately responds to the intended action of the prompt.

[@dumbkid0]

actual use cases that I constantly get wrong answers from the vision models: 1. screenshot of outlook calendar, and get vision model to suggest an one hour opening for a new meeting between 9-5 M-F. 2. screenshot of a bunch of workers in factory, and get vision model to identify how many workers are not wearing hardhats (also identify total workers in picture) 3. simply upload two identical pictures and get vision model to tell the difference between them. 4. ask if picture is computer generated (use a dalle-3 generated picture or one of those computer generated portrait)

[@topmandan]

great channel. love your work. FYI - to read a QR code, the QR code must have white space around the outside. This allows the model to pick up on the 3 big position marker squares.

[@joannot6706]

Would have been cool to compare with pali Gemma

[@yevhendyachenko1384]

We definitely need some other way to test prompts for the visual models. I would use a long explanation system message to point local LLM to use, scan and convert the picture to a text and use it in order to "read" user prompt with the graphical embedding

[@mbrochh82]

Matthew, please test handwriting recognition as well when testing image models! This would be a massive usecase useful to anyone who likes to use pen and paper for personal journalling.

[@MisCopilotos]

Very very useful Matt, thanks!

[@NotU-eg1jf]

When working with data on chat-gpt 4o, try to instruct it not to use code interpreter. I use it to the openai api to OCR purchase orders, I get better results when I instruct it not to use code interpreter.

[@superjaykramer]

your channel is great, actually useful information, cheers

[@OliNorwell]

With that converted_table.csv, it's only using Code Interpreter to do the final conversion from a table to csv. If you just ask for the table I think it will just output the data and not use Code Interpreter.

[@gamingthunder6305]

here is what i found. if you copy the vision model file into a folder with an uncensored model and rename the vision model file matching the model. it will load and work much much better. i tested it with the dolphin 2.9 model. thnx for sharing matthew.

[@gideonvongalambos7221]

Ask if it can measure distances between points in a photographic image, not in pixels but in the actual length of the object like for example the side of a building from different angles see if it can give a reasonable estimate of it's size if not an actually accurate figure. I'm also interested if they could convert an image of some object into a CAD file as well but I suspect they weren't trained like that.

[@mrpro7737]

GPT-4 initially struggled but eventually transformed into a Terminator ! 😂

[@gpsx]

GPT-4o did very well on the iPhone questions. I wonder if it had some specific training related to OpenAI's partnership with Apple.

[@luigitech3169]

I think the best vision model is CogVLM based non llama3-8B

[@ComicBookPage]

I'm curious how these vision models would do with the interaction of the text and images on a comic book page

[@KC_79]

I think the LLM connect WhatsApp with the question - "what app" is taking the most storage space?

[@emil8367]

thx, curious how it handles written text

[@mikezooper]

I’m envious of Matthew’s AI hair stylist. Only joking Matthew. Love your videos 😊

[@nekomatic]

Can you make avideo those models challenged with udestanding charts, graphs, diagrams etc. And how well those models extract meaning data from this kind of images? Are rthose models capable to i.e. understand an UML style activity diagram?

[@passiveftp]

Here's one for you to use - can any of them say what the X,Y pixel position of something is within the image? like an icon

[@AK-np2fh]

your qr code is incorrect. Try using it with white background and not transparent as per spec

[@kranstopher]

I've actually had pretty good luck with llama 3 dolphin. I tried using the lava variant and I came up with kind of the same results.

[@nixellion]

To be fair - that QR Code is not readable and not standard. It's inverted, and has weird black squares where they should be white. I tried scanning it with 3 different apps - they all failed. So GPT might've gotten it if it was a more standard QR Code. Standard QR Codes are - black on white, anything else is not within spec. After that "Dark on bright" works well. "White on black" is something that actually confuses a lot of QR Code readers.

[@efficiencyvi8369]

Your QR Code for the URL Looks really strange. Maybe provide an image without transparency?

[@MeinDeutschkurs]

Are you sure that this is a QR code? My QR code Scanner was not able to read it.

[@matthew_berman]

I generated it myself

[@MeinDeutschkurs]

@@matthew_bermanmy scanner was not able to read it.

[@AlfredNutile]

Maybe try to use them for reading a webpage and scraping the right data from it

[@taveiraadriano]

Can you try uploading a graph picture (such as CPI) YoY line graph and ask the models How they understand It? Which average pace it's moving? Which date intervals it's growing or reducing?

[@jdholbrook33]

Can we run Phi-3 Vision locally?

[@matthew_berman]

Yes!

[@Spreadshotstudios]

What laptop are you using where you are able to load 14gb of model weights without quantizing it into vram?

[@cucciolo182]

Why didn’t you add Gemini 1.0 Pro Vision ? 😅

[@63801170]

In the first test it would seem only 4o got it right as a "llama" and not an alpaca. The speed will improve, but accuracy has to be there first or there's no point to getting the AI to assist with images.

[@Dave-nz5jf]

GPT40 is getting hammered this days, I wonder if that has any effect on the results it spits out sometimes. Really great new set of tests!

[@VlastimirStankovic]

Have you tested Yi-1.5 yet?

[@olivermorris4209]

Phi3 is MIT so commercial license - is this a first?

[@photorealm]

Great test, excellent pace , A+ 🤟

[@perer005]

Think you need to update that AI test soon! :D

[@thanksfernuthin]

You can ask the AI not to editorialize and give a clear descriptions. It works. And one answer you got you thought it should say the Photos app. It might have not considered Photos an app. Just the memory used for your photos.

[@mikekearl2416]

I would like to see for the vision tests a form that is filled out with both text and handwriting. Tell it to create a organized and well formed JSON object using the data found on the filled in form.