DEF CON 32 - Your AI Assistant has a Big Mouth: A New Side Channel Attack - Yisroel Mirsky

[@thygrrr]

11:55 "As manufacturers of a machine that guesses entire paragraphs, we doubt that it will be possible for an attacker to guess an entire paragraph."

[@KernelGhost]

It is both unexpected and fascinating that token lengths alone can be utilised to determine the assistant response text with such accuracy.

[@jimmy000]

I love the attack vector, great discovery and extremely well demo'd and presentation. One of the cleanest and concise talks, kudos to the presenters as well, very well spoken and no fluff.

[@TheJogug]

Interesting idea and execution. The accuracy of Predicting First Sentence: 55% and Predicting Entire Text: 38% seems high. The sample space of the dataset prompts probably has a huge impact on these numbers.

[@ChiefMasterGuru]

Christ the background noise is unbearable

[@DrZbo]

Add it to the list of incredible talks that are fucked by poor sound

[@gavinknight8560]

Probably the closing session in an adjacent room… defcon is a boisterous affair.

[@Arisekiwi]

Could be that their breaking into a system for one of the challenges that happen

[@LolWutMikehSM]

Consider subtitles

[@ChiefMasterGuru]

@@LolWutMikehSM or maybe the event can consider doing the bare minimum audio set up lmao

[@shrimpkins]

My goodness, what you fid'na tell me next, Liz Lemon? That some employees at AI companies might have full unencrypted access to my convos with their products? Nah, nothing to worry about there--that guy from Amazon said a long time ago that Alexa doesn't listen to anything in the room until you say "Hey, Alexa!" and I believe him. If you can't trust a company with a moral mission statement, good God man, who can you trust?!?

[@xj0ex39]

Sam Altman has already scanned you and your entire families optics.

[@SarahKchannel]

if you decode the encrypted tokens, from known text, you will get a very high confidence level on thr result. That data you can use as labeled training data. Which you can use to reverse the encryption keys used. From there there is no more guessing.

[@blkauxpro]

Who mixed this FOH setup? It's awful. This is riddled with easily-filterable ambient noise - and from the 2nd stage?! Next Defcon call a pro to run your board and rack. I'm available.

[@marco208]

It's like they're watching a sports game next to this space. Keep your crowd under control if there is no isolation. Almost as if it's on purpose.

[@pelic9608]

I really want to see that other talk now. Sounds like they revived Jeopardy. 😄

[@royweiss1]

@@pelic9608it was the closing ceremony 😅

[@AlecArmbruster]

If DEFCON hired an actual professional to run their AV, then it wouldn’t be DEFCON.

[@nxxxxzn]

dude, be thankful there's audio at all instead of a fart on the left chan at -90dbfs

[@j_t_eklund]

The biggest problem is they don't employ people with the right mentality for hacking their stuff... So they never detect shit about anything possible to exploit.

[@pneuma33]

outstanding work and very scary stuff.

[@elvinaguero4651]

What a great work and Collaborations.

[@marco208]

Nice work. I like human understandable attacks. Takes some out of the box thinking to get to this.

[@willhatch7721]

Reminds me of how they cracked enigma

[@xj0ex39]

Was my magnum opus.

[@petevenuti7355]

I'd have to check out what is going on next door! I wonder if the speaker was thinking that too?

[@DreadFox_official]

ohhh that's very interesting!!!!

[@PassionforSpace]

It is clear that the token-length is a vulnerabillty that needs to be addressed.I am wondering,why not encrypt the length? I mean,why not change the length into something that would make it impossible to guess the word:If a word has a token-length of 6-make it something else, if eaves dropping is happening.

[@77rdcasa]

Maybe, but itis probably more complexity. I like you're looking for solution, but the direction of all controllers has decided say to society, can't choose back in time. The mobility of social status are only B and C. The class A Justin making adjustments between themselves with the new commodity. Good luck 4civilization! I'm sorry if It's not understandable.

[@legoguy217]

~50% accuracy isn't that bad, but not exactly a stat I'd be scared of. Given a set of known/trained tokens the chances of the guess being wrong is literally a coin flip. God forbid it starts responding in Spanish, then you'll need to train a model for each language.

[@thomass9457]

How accurate is the attack if the user is speaking to a custom persona, not the default?

[@yahmaar]

Thank you for sharing

[@RoughGanome]

Great talk

[@mechadense]

14:06 second part

[@j_t_eklund]

This is why recall is bad..

[@recklessroges]

Nice CV application.

[@JoshtheFifith]

endlech a heimishe guy at defcon

[@brianhirt5027]

So they used how much LLM crunchpower training LLM's to decode LLM's? How much power & water got blown up so they could build this training model? Where is the point of vulnerbility? Shared networks?

[@FireStormOOO_]

They covered this, about 2 days/$200 of compute on Azure. So almost nothing by AI standards.

[@lunafoxfire]

brian hirt more like brain hurt

[@cit0110]

@@lunafoxfirefr😂😂

[@brianhirt5027]

@@FireStormOOO_ That seems a little sus. You know how large a dataset is required just to give the models a starting point.

[@brianhirt5027]

@@lunafoxfire sounds like you're ready for your big move to middleschool next school year, kiddo. Go find somewhere else to play now. The *adults* are talking.