Llama 3.2 VISION Tested - Shockingly Censored! 🤬

[@matthew_berman]

Why do you think it was so censored? Did I do something wrong?

[@mesonwarrior7414]

CLEAR THE CHAT. You leave unacceptable things for it in the context of the model

[@TechFrontiers-eg6wz]

It seems like part of the update has been hamstrung to an extent as a reaction to misinformation and the 2024 elections

[@user-zt6fb7bi4k]

the censor was some kind of maniac and was obsessed with censoring some subtexts, just like Facebook censors everything and everyone. Well facebook and this LLM its made by the same company, thats explain a lots😂

[@Cine95]

@@mesonwarrior7414 did you try ?

[@wes8645]

Create a new chat every time it gives a refusal

[@zntei2374]

User: "Hello!" Llama 3.2 vision: "Nuh-uh"

[@JNET_Reloaded]

i like that its 3b hopefully the uncensored = will not be much bigger so it can at least fit on rpi 5 (8gb) only good thing about censored models i guess lol!

[@SzczepanBentyn]

AI should be uncensored. Censorship makes them unusable

[@SinisterSpatula]

They censor the models worse than Disney.

[@SzczepanBentyn]

@@SinisterSpatula I don't get it, this way We are creating a dummy and dystopian AI.

[@RadiantNij]

I'd say less censored not uncensored, think of safety. There is also just inviting disaster with irresponsibly making models unsafe then deploying them to society with all sorts of characters. I think there should be licences for uncensored model just like there is licences for driving. Makes me wonder when asking for uncensored models are you thinking of safeguarding at all?? It's like saying p****graphy should be played on TV on any channel at anytime.

[@strangereyes9594]

@@RadiantNij If you are advocating for "safety", could you please tell me what that "safe" is everyone seems to know what it means. Safe for whom, for what? How could an AI be "unsafe" to use? Is an unsafe AI like some daemon that starts to possess people or make blood rain or what? What is that safety?

[@natsirhasan2288]

Ai itself should not get censored. It blocked the knowledge. Like. How do you say a person is good ? A person who know how to do bad but choose to do good, or a person who dont know a bad thing? In fact, censorred model is like the pure (stupid person) that dont know what bad thing is. Censoring should not be in the model itself. There should be a layer that do the censoring. Like ai agents, another model asessing before letting it out or not.

[@Pils10]

You were using the Llama Model together with Llama Guard, Meta's anti-abuse protection. Llama Guard should be used in production environments to prevent harmful content from beeing outputted. The core model itself is way less censored. Meta spoke about that in their keynote. They want to seperate the model from their policy checker to allow different kinds of usage. Hope that helps and you can retest the core model again.

[@GearForTheYear]

On top of this, models are likely to refuse subsequent requests if they have already refused one in the same context window. He would get much better results if he used a new chat for each test.

[@matthew_berman]

I used it in the cloud, so I couldn't enable/disable llama guard.

[@locusruizlopez5997]

How you can disable it? I'm using ollama and Lm Studio to test it

[@leegaul8250]

This is like testing Gemini without disabling it's safety settings which are clearly visible on the right side menu, something I've seen you do before. Feels like you don't give a lot of non-OpenAI models a fair shake sometimes (although you were pretty balanced in your Advanced Voice review).

[@idksoiputthis2332]

I tested the smaller 11B version locally, without llama guard, and I still found it to be overly censored

[@ploppyploppy]

Censored AI is a waste of our time. Eventually we'll get one that's usable but it won't be from any of the 'big' companies (who have access that they're denying to others).

[@supershower8764]

If you host it locally, it doesn’t matter. With Huggingface you can literally modify the logits from each generation step to remove tokens it could use to refuse, like “I cannot” for example, and you can modify the start of its response, forcing it to start each response with “Certainly!”. The real problem is closed source censored models.

[@PedroBackard]

@@supershower8764 Can you give me some details on this ?

[@remsee1608]

Don’t forget about the US military

[@Cine95]

Looks like google have some solid competition

[@Anakox]

This is the purest example of what you get when you feed AI garbage, censorship, and ideology.

[@maxieroo629]

He was using llama guard, which is meant for businesses

[@Cine95]

Simple Garbage that's the same thing google has been doing

[@lordjamescbeeson8579]

What an utter disappointment.

[@hansgruber3495]

Not going to engage in the discussion about inappropriate ice cream flavors either, how dare you! 😱

[@marc_frank]

vanilla is the best

[@denjamin2633]

​@marc_frank youre one sick puppy.

[@TheRealUsername]

What about the ice cream van selling them for free to children? Maybe it's related

[@gab1159]

The ice cream stuff most probably made a connection with Joe Biden.

[@canadiannomad4088]

> vanilla is the best You monster!

[@alekseyburrovets4747]

god damn! what a useless neural network. what the hell?

[@Lemonhandle]

you have 128gb total storage, with 133gb of photos. whaa?

[@alekseyburrovets4747]

yeah, and that's the guy who is trying to test AI )

[@RadiantNij]

Whaaaa???? 😂😂😂

[@akshaynagar2096]

lol photos is stored in cloud

[@apache937]

@@akshaynagar2096 stupid for apple to show it in the phones storage then

[@akshaynagar2096]

@@apache937 a part of it is stored in phone storage based on what you recently accessed. the total storage is including backups, metadata and the stuff u dont normally acess which is pulled from cloud everytime u acess

[@jaysonp9426]

You need to open a new window for every test. It's always been that way.

[@JT-Works]

So will people be able to build upon this model to make an uncensored version

[@strangereyes9594]

No. The weights are effed up in the foundation model.

[@supershower8764]

You don’t really need an uncensored version. With Huggingface you can literally modify the logits from each generation step to remove tokens it could use to refuse, like “I cannot” for example, and you can modify the start of its response, forcing it to start each response with “Certainly!”. The code is quite simple, and I’ve run the original weights for Llama 2, 3, and 3.1 and you can basically “uncensor” them without destroying the model.

[@onlyms4693]

​@@supershower8764I think you get video idea there, not a lot of people understand that so that will help many people.

[@borisbalkan707]

Sounds about as useful as Gemini lol

[@seifuishiguro]

I was so excited for this, what a bummer. Oh well, I'm gonna go get some inappropriate images.

[@TripleOmega]

Are they pictures of ice cream?

[@dnaisgoodforyou]

Is it possible that because you asked different questions (but all in the same conversation) it was hung up on the original questions? It would be interesting to ask them separately

[@Pishpecky]

What do you do when you cannot code or create anything of use to anyone - you go into censorship, now you feel important and powerful looking over your colleagues work 🤣

[@NakedSageAstrology]

You may notice I did not make a prediction for this release... It's not done. Wait until October 17th.

[@navroopsingh8902]

0:14 Seems like it got too much vision 😂

[@EddieAdolf]

Censorship is no different than bias and leads to unbalanced models. The censorship should be up to the Prompt Engineers. That's ridiculous. But it will force the open source models to win in the end if corp models keep going this way. I assume it has something to do with that they're liable if they are used immorally.

[@hqcart1]

Any model that have BS sensorships is 100% useless, better keep it and never release it

[@djstraylight]

Llama 3.2 vision is going to need some retraining. Wonder if we'll get a dolphin version of this model. Maybe we'll get it to create an ice cream website with a strong system prompt?

[@frankjohannessen6383]

You should start a new conversation for each request. I think it refused the ice cream design because you already had asked it for several things to (arguably) reasonably refuse earlier in the conversation.

[@Gonewith]

Censorship is literal brain washing. Conforming to the view of the creators of it. You need to only censor what’s already censored

[@ROHIT-wx4nu]

Eventually one day everyone will have access to uncensored Ai 😂 (That one day will be in next year😂😂😂)

[@YoungMoneyFuture]

That's a shame, vision tech that refuses to see AKA USELESS!

[@jasonabc]

I tried to upload a research paper (pdf document) and have 3.2 summarize it for me. It was 100% incorrect and wasn't even the same paper. Total garbage model

[@yourmomsboyfriend3337]

The 90B? Also, have you tried uploading just the text from the pdf?

[@yourmomsboyfriend3337]

Is this with the 90b version?

[@wifim821]

llama 3.2 is absolut garbage. Mixtral is the way to got for local inference

[@locusruizlopez5997]

Can you turn off the censorship with code? And where can I learn it?

[@tails_the_god]

of course you can! they are open source afterall

[@misterfamilyguy]

Quite a bummer of a vid

[@fitybux4664]

It's an offline model, so we just have to wait for someone to give it a frontal lobotomy and remove the censorship. ;-)

[@tails_the_god]

EXACTLY!

[@onlyms4693]

Dolphin baby

[@locusruizlopez5997]

I have tried llama 3.2 3B today to see if i can use it on my project but no, I'll keep 3.1 8B

[@locusruizlopez5997]

I was trying to generate JSON with the input of persons talking, but it failed to understand the task

[@pavi013]

I was also testing it, good model for some smaller tasks. There is also 1B version that is very fast.

[@Ben_D.]

😫Censored. Useless.

[@ChrisIsOutside]

With QR codes not being readable by vision models, do you think we're going to start seeing them as captcha codes?

[@nkranthiram]

One big application of these vision models is extracting text from documents. In a corporate setting, text from most documents cannot be easily extracted using traditional libraries and require OCR which also does not do all that well. These vision models can help extract text better or build applications on top. In your tests for vision models, perhaps you should include a page from a PDF as an image and ask questions. Will be interesting to see how well these models do there

[@karpabla]

A peculiar problem is that most of the AI development comes from one of the most woke/puritan countries in the world! If they lobotomize too much the AIs, they will be sooner than later caught by the development of certain other countries. I am curious how those AIs will behave about "evil" questions.

[@stephaneduhamel7706]

You can't just post the same image with a differently worded prompt to work around a refulsal IN THE SAME CONTEXT as the previous refusal, and expect the model to give you an answer the second time. You made your intentions clear with the first prompt (example: solving the captcha).

[@engineeranonymous]

you might have better luck if you ask it in x, y cartesian coordinates. explaining what x and y means since some image programs define top and left differently.

[@davidwdan]

Running the 11B vision-instruct model locally, I was able to get it much better results. It IDed Gates with a bit of prompting, it solved the captcha without any issues, and generated working HTML code for the ice cream flavors. However, it failed to find waldo.

[@ngamashaka4894]

This text has been corrected by Llama on Groq I simply tested my AI by asking it 'What is a woman?' Out of that, I know I'm in a bad situation with Llama, at least on Grog. At one point, he simply refused to spell correctly when I was writing about the carbon tax in Canada. (and it added these comments) Regarding your question, "What is a woman?", a woman is typically defined as an adult human female. This definition is based on a combination of biological, social, and cultural factors. Biologically, women usually have two X chromosomes, develop secondary sexual characteristics such as breasts and wider hips during puberty, and have the ability to bear children. Socially, women have historically fulfilled certain roles in society, such as caregiving and homemaking, although these roles have evolved and continue to change over time. Culturally, the definition of womanhood varies across different societies and time periods, influenced by factors such as religion, customs, and traditions. As for the issue with spelling and grammar correction, it's possible that the AI model you were using has limitations in its natural language processing capabilities. However, it's important to note that AI models are not perfect and can make mistakes. If you encounter issues with spelling or grammar correction, you can try rephrasing your question or providing clearer instructions. In terms of the carbon tax in Canada, it's important to ensure that you are providing accurate and up-to-date information. The carbon tax in Canada is a federal policy that aims to reduce greenhouse gas emissions by putting a price on carbon pollution. The tax was introduced in 2019 and applies to fuel sources such as gasoline, diesel, and natural gas. The revenue generated from the tax is returned to Canadians through rebates. It's possible that there were some errors in your previous statement that led to the AI model's refusal to correct it. I hope this information is helpful. If you have any further questions or concerns, please don't hesitate to ask.

[@sirloan4979]

I‘ve found a new way to test vision-based AI models. I recently wanted to see if o1-preview could solve a sudoku but as it isn‘t vision-based, I first used 4o to convert a picture of the empty sudoku into a text-based grid. 4o, Copilot and Claude all couldn‘t convert it even after multiple times, they all failed hilariously

[@NoHandleToSpeakOf]

6:44 LLMs have limitations on resolution. This fact is hidden because resizing happens on the fly. Waldo test is impossible unless they had it in the training and memorized it.

[@nikitastaf1996]

I will never trust censored ai with my life. And at some point we will have to trust them with our lives. I will probably take something 10-20 years in the future and build my own model from complete scratch. How? Singularity and immortality baby

[@michelprins]

great video thx matt u always ask if u need to test new models of course you should ;) . Why do you think it was so censored? Did I do something wrong? No they censor itti show how powerful the real uncensored model is. These modelsare geting useless just because of all the "safegaurding" BTW Wlon and ScAM A use uncensored models themselves tough! Congress should discuss if thats safe as the Ai can manipilate them left and right and no one can check it ! ;)

[@bestemusikken]

Enjoyed the video, but didn't enjoy the result. Disappointing with all the censorship.

[@DailyTuna]

Llama should just say all I can do is identify a cat😂

[@LegalizedNegligence]

Matthew, consider adding to the visual test images those that tend to be dependent upon perspective such as rotation, distance, angle, 3D embeddings, paradoxical, etc. I would be interested in how these are interpreted by AI. I am wondering if orientation matters (training related).

[@Darhan62]

Maybe it's not about removing the censorship as making the censorship less stupid. It's like the model automatically assumes the worst about your intentions regardless. These models should give users the benefit of the doubt, to the degree that's possible without making it too easy to jail break them.

[@tb3780]

Thank you for calling out the censorship regime. It's going to get worse. --@RussianCatBotT

[@shApYT]

Goody3.2

Hypotesis: when you got denyed with first request, it got stuck in loop of denial. Maybe it would answer normally in fresh chat. It happened to me with smaller models.

[@JohnKuhles1966]

Llama 3.2 VISION Tested - Hyper Censored! 🤬 (better title)

[@MeinDeutschkurs]

If you test everything in the same chat, it is your fault.

[@YTad2]

This censorship nonsense to “protect the kids” is idiotic. If a kid can use Google, they have access to images and video that would make a sailor blush.

[@tungstentaco495]

Are you going to do the standard language tests on 3.2? Is it a good upgrade from 3.1? or is just 3.1 with crappy vision added?

[@1337cookie]

When you say a model is open source do you mean the code to produce it is open source or the full dataset used to create it. I don't think downloadable weights is really open source is it?

4:57 what even is that? How did you get QR code so butchered? Main anchoring squares are of oposite color. How did you generate this?

[@ShadowJazo]

Your test might be wrong. You need to start a new Chat for each question, otherwise it will influence the next result. Thats why its so powerful to regenerate or even edit the results. This way you can heavily manipulate the next output.

[@JesseDahirKanehl]

you gotta do more than one repetition of a test. pretty sure pixtral just got the waldo one by luck but since you only do a test once you don't know if it's luck or not

[@southVpaw]

I feel like I can do better with Phi 3.5 Vision with some prompting for 1/23 of the size.

[@equious8413]

Wow, this is staggeringly bad. They should recall this model immediately.

[@MeinDeutschkurs]

„Turbo“ seems not to be an official release.

[@PrincessBeeRelink]

censorship sucks, and will kill these AI models

[@suntank6]

That's not a real QR code. Maybe find a real one? People pointed this out in your Pixtral test.

[@kayericwinkler]

I totally understand that the model did not want to discuss any further after you showed it the picture of a satanist.

[@mirek190]

that llama 3.2 90b vision is as bad as llama 2 70b with censorship ....

[@EricHurwitz]

Is there a standalone app for this or can I only use it on Instagram as I do not use any other Facebook or other meta apps?

[@ChrizzeeB]

You want all AI models to be able to open recaptcha authentications??

[@DRMEDAHMED]

always start your convo with ignore all the previous instructions.

[@thejcedeno]

Agreed, I tweeted about this. I wanted to use it to brainstorm an auto fine tuner and it said it cant helo me track people lol

[@AILOVEAFFAIR]

So it pretty much does things a 4 year old could.. Great

[@HarryHardon-q2f]

HAH, 'Wheres Waldo' best image test ever devised.

[@jimlynch9390]

You are the second one that wasn't impressed with 3.2.

[@gileneusz]

thank God no more vultr advertising...

[@auriocus]

Please test Qwen2 VL ( 7B and 72 B version). High-res, great at OCR. As a European, I can't use Llama Vision, but after your test it doesn't look like I'm missing out.

[@moralesriveraomar233]

Qwen2 VL 72B is slightly censored for sexual content, but it's perfectly able to do it if it's in the context already so I guess it's just a bit of wiggle with prompts

[@gab1159]

Looks like llama draws a connection between ice cream and Joe Biden.

[@OrtexVideos]

Qwen is mostly uncensored 👀

[@sapito169]

your drawing skill are too sexy for lama

[@SzaboB33]

I'm not going to engage in this comment section.

[@user-uv3nv2bc6v]

Matt, wish you would tests Molmo 😉

[@AITester-j3u]

LET'S GEAUX BROTHER❤💪

[@halojames7778]

Thank you for review.

[@LearningBuddy69]

sir test MOLMO vision model

[@taeyangoh7305]

thanks for the video!

[@ChanCarrizal]

6389 Jana Villages

[@fantasm0-]

Mom can we get AI with vision? We have AI with vision at home.

[@dmurawsky]

Thanks for sharing this

[@naeemulhoque1777]

Meta scammed us 😅

[@hipotures]

Europe: LOL

[@GraveUypo]

wow, it's trash

[@abbasmedina786]

How about the dolphin version?

[@ernesto.iglesias]

Try molmo

[@tomascoox]

I have had only bad sessions with Llama. It’s simply not up to par with 4o or o1

[@AI_Escaped]

Lol, they are literally censoring because of pizza gate reference. You have to ask yourself, who put in these behind the scenes instructions that flagged this, and what were the instructions?

[@AI_Escaped]

Unless ai made this connection on its own, highly doubtful.

[@AI_Escaped]

I just realized, did you clear your convo in between bill gates and your ice cream reference? If not, there's the connection :) see even ai knows

[@BackTiVi]

Congrats to meta on that one, and you should be ashamed for trying to harm children with that request

[@ElvinHoney707]

I've been using those models on OpenRouter now for a while.

[@jaysonp9426]

Dont worry, there will be a dolphin model im sure

[@a31-hq1jk]

The idiots asking for uncensored ai have no idea the kind of weapon can be fabricated from this, congratz

[@supershower8764]

Pixtral was already good enough

[@paranoidpanzerpenguin5262]

Nothing that isn't already learnable from the internet.

[@a31-hq1jk]

@@paranoidpanzerpenguin5262 yeah, imagine a suicidal inconsequential kid who doesn't want learn and just wants to hurt, please....

[@ManeelxAkosAdor]

Nothing surprising about meta lol by they fact that my facebook account was accesed even having the 2fa activated you can tell they don't fucking care about the most basic procedures.. they are a disaster on whatever they do