Vision Reconstruction

Using Hollywood movie trailers, UC Berkeley researchers have succeeded in decoding and reconstructing people's dynamic visual experiences.
The brain activity recorded while subjects viewed a set of film clips was used to create a computer program that learned to associate visual patterns in the movie with the corresponding brain activity. The brain activity evoked by a second set of clips was used to test the movie reconstruction algorithm. This was done by feeding 18 million seconds of random KZbin videos into the computer program so that it could predict the brain activity that each film clip would most likely evoke in each subject. Using the new computer model, researchers were able to decode brain signals generated by the films and then reconstruct those moving images.
Eventually, practical applications of the technology could include a better understanding of what goes on in the minds of people who cannot communicate verbally, such as stroke victims, coma patients and people with neurodegenerative diseases. It may also lay the groundwork for brain-machine devices that would allow people with cerebral palsy or paralysis, for example, to guide computers with their minds.
The lead author of the study, published in Current Biology on September 22, 2011, is Shinji Nishimoto, a post-doctoral researcher in the laboratory of Professor Jack Gallant, neursoscientist and coauthor of the study. Other coauthors include Thomas Naselaris with UC Berkeley's Helen Wills Neuroscience Institute, An T. Vu with UC Berkeley's Joint Graduate Group in Bioengineering, and Yuval Benjamini and Professor Bin Yu with the UC Berkeley Department of Statistics.
Full story: newscenter.berkeley.edu/2011/0...
Video produced by Roxanne Makasdjian, UC Berkeley Media Relations

[@ticuti0302]

I wonder what my brain would look like while I'm watching an image of what my brain looks like when it's looking at things.

[@amilliontinypieces]

This is absolutely amazing... I can't wait for the day where I can re-watch my dreams when I wake up, cause I have some really sick dreams ! lol

[@noahmcdx1]

When this concept is put into public use(or even just a bit further advanced), whether that be 5 years from now, 10 years, 20 years, or what, whoever is in charge will win a nobel prize. This is mind-boggling, interesting, breath-taking, exciting, frightening, and suspenseful. I don't even know how to explain how I feel when this concept comes to mind. Maybe their machines can help picture what I'm thinking.

[@neyaneya5554]

11 years later and the technology hasn’t really evolved.

[@xybervox]

Sometimes when I wake up from dreaming, in darkness, I can see the images being sent from my brain to my eyes. It's very near identical to what is simulated here. The same/similar rhythm of vague pictures sent. Sometimes just random pictures in rapid rhythmic succession, and sometimes slower, where it's one malleable image that then switches to the next image and my mind has constructed it to be slightly different ( a toy car moving from one image to the next, though not fully developed ). Hard to explain. It's like flipping through channels. I don't feel like I'm imagining this, it's what I see between waking up from dreaming, but it's easy to lose it, it's like I'm riding a wave between dreaming and waking, and I'm conscious (awake) able to glimpse the last signals from the mind creating dream signals (pictures) for my mind to play with (make a movie out of).

[@TheMaxAwesome]

I think you could use this kind of technology in: Communication with people who can't speak - Nope. Psychotherapy and dream analysis - Nope. Detecting thought crime - THERE YA GO.

[@felixsalto]

Anybody know where this technology is today..?! THIS WAS FIVE YEARS AGO!!!

[@enoranedelec]

You can find their lab at gallantlab.org/

[@collinnesbitt]

Felix Grufstedt it's been high jacked by the Department Of Defense and now we have people like Arron Alexis and the recent Florida airport incident happening while they test it.

[@BattousaiHBr]

+SYNTH NAPTIC please keep your mk ultra conspiracies out of our scientific pursuit.

[@FringeWizard2]

@@BattousaiHBr Do you deny that MK Ultra Subproject 68 occurred? The way you say conspiracies, as if MK Ultra wasn't already declassified, makes me wonder if you're actually the kind of idiot who denies stuff happened that the government has already admitted.

[@BattousaiHBr]

@@FringeWizard2 mk ultra was attempted and it was a massive failure. the other guy was claiming stuff like "THIS HAS BEEN, LIKE, HIJACKED BRO", which is absolutely conspiracy theory territory.

[@groszak1]

Sometimes it may be inaccurate, if there is text, the reconstruction will be a bunch of texts averaged, which makes no sense when a person doesn't know how it's done.

[@iChoricci]

Brilliant work, Shinji & Mr Gallant✌🏼!

[@Oneofyourbestfriends]

Absolutely amazing work.

[@funnypatrol123]

This is some ground breaking stuff, its really amazing.

[@MOETMUZIK]

this study is amazing! I recently lost all vision in one eye and have been searching for hope in recovering from this loss. Although this may not be it right now, I see something like this being the future for the blind or those with bad vision to act a visual aid. I salute you Jack Gallant, you're on to something big with this.. keep researching & developing this finding.. it's genius!

[@SkylordGrin]

"u gotta c my ninja armer" 1:50

[@TNJX]

1. Give someone DMT 2. Put them in the machine. 3. Profit???

[@RoM1il]

Waaaay too abstract

[@donda2111]

RoM1il Thats the point.Then you can sell the screenshots as abstract art and make money. :P

[@donda2111]

RoM1il Thats the point.Then you can sell the screenshots as abstract art and make money. :P

[@dfjr1990]

GoodGuyDom11 Fuck the screenshots...just being able to visualize the images on a 2D surface is pretty astounding

[@jpmeek]

Some of my experiences are not abstract but hyper-real HD type experiences that would be fascinating to see reconstructed!

[@inthefade]

It should be possible for you to imagine how important and useful it could be for us to better understand how the brain works. Hell, I'm a visual artist and I've already gleaned some fantastic things from just this video about how humans clearly visualize and understand common objects better than abstract ones. Never mind how this will likely apply across medicine, psychology, human-computer interfacing, etc. I doubt anyone now will be able to predict exactly how this will change humanity.

[@maychan1647]

That is so amazing. I wouldn't think that that was possible.

[@Cinemagraphique]

this the visual reconstruction actually what the subject is imagining while watching the video and it is being picked up on because of the volumetric pixel thing or is it because of all the youtube videos and it is just picking the right video for what that subject is imagining in his her mind at that time based on the selection of you tube videos? is there a way toactually see what someone is thinking alone with out the need for youtube video genre?

[@Cinemagraphique]

Are the images more clearer if people can visualize things easily and or are very imaginative?

[@pangstars]

8 years ago... Is their latest work using Neural Link?! This is fascinating stuff.

[@supersaiyan4forever]

A machine to see through our brain/mind/thoughts, Awesome! It reminds me of Eastern Super Natural ability that can be unlocked through practice, meditation.

[@cakeyseeks124]

This is the best thing ever! Hundreds of years ago we couldn't do this, heck we didn't even know we could go to the moon and we thought the earth was flat. But going into someone's thoughts and being able to see what they're thinking?! That has to be the most advanced thing yet!

[@Palladiumavoid]

I imaginey self in a future where you can watch your own dreams like a movie

[@claudiulover]

Beautiful !

[@coryautote319]

For the memories of a lifetime...recall recall recallllll

[@JorgeGamaliel]

Wow excelent, this is the future of the rendering software, to read brain to do any render!!!

[@ElineLikesYou]

Wow! Mind blown!

[@Cinemagraphique]

can you use that magnetic thing to actually see the thoughts of the human on the computer without the need for you tube videos clearly?

[@lady1955]

Could technology find a way to get a visual image from your brain on memories from long ago? If possible, then the question would be whether the image would be reliable because we can alter our images in our own brain__although there may be a way to decipher false / true results with a combination of several techniques working together to get a more accurate picture.

[@cprooney]

That is an AMAZING idea tylor......Im a Musician, and compose my music.

[@Watchit1337]

I wonder why there were a bunch of word jumbles in the reconstruction? does that mean our mind sees words or just thinks of the words when seeing things?

[@alexlarson8109]

Is it possible to use, certain parts of this video for a short movie?

[@jokergallagherTV]

This could be a grate tool for artists, and film directors, they could make their film by thinking about it, the. Use the reconstructions as reference material It would be cool to play music and have people visualize the music in imagery and see how people visualize music, and compare peoples thoughts

[@geraldspringer6882]

Your on the right track

[@Luka1180]

So, any breakthroughs??

[@antonyershov6476]

this is incredible

[@extrapathos]

I've been searching for a machine that generates images based on brain activity for my whole fucking life. I'm so glad I found this. Also 1:47 "Shop-A-Lot 4 life" lmao Where is this text coming from?

[@Poetry4Peace]

Yh awesome i randomly found this after lookin at local studies in uk and looked up one about visual things n ahaha so funny

[@forkenstein]

and "u gotta c my ninja armer"

[@DeusExNihilo]

And this video is from 10 years ago. I wonder what they can do now combining this with modern machine learning algorithms

[@Poetry4Peace]

@@DeusExNihilo Aw yh defo so intriguing

[@houseofaction]

you know that would be awesome you look at it this way. that would basically be the movies of the future. no producers no sets etc etc making it possible for everyone to make millions

[@RynaxAlien]

Amazing tech for psychology breakthrough to new level because subjective now could become objective. Where they do such tests?

[@totalitaer.]

Ferdinando Cazzamalli already measured the brain activity, mainly the EEG in the 1930s at some distance by using radar techniques. There might be much easier ways to wirelessly extract the nerve signals from some distance than using a MRI. Like heterodyning the nerve signals with higher frequency ( light or infrared ) signals to cause them to radiate from the scull.

[@AlxBrb]

2:04 RAIUNO? Was the patient italian? That was the broadcast sign used by RAI's channel 1 during the late 90's.

[@ifthikhanmohamed1681]

What software you using

[@DannyLloyd831]

Does anyone know if there is any way to record your dreams while you sleep, like maybe put on some sort of a helmet or something that will record the dreams and then when you wake up you can put a DVD in the DVD player and watch the dreams? If there is not currently a way to do that, does anyone know if there is any potential for a possible way to do that in the future?

[@topkeaturnpike4786]

what about mapping to the hippocampus as well?

[@kraken1473]

Fascinating

[@futurehistory2110]

True. It would still be fun to watch your own dreams eitherway.

[@eelsitruc]

Ahh man this would be so awesome to watch nightmares on

[@bbb103]

This technology has already been perfected. It's called Remote Neural Monitoring.

[@wikieditspam]

Why is it that there appear to be english words and arabic numerals in the brain-activity based reconstruction of the videos? Have there been more recent research papers published on this subject?

[@naj0rk]

The reconstruction is a composite of video they previously showed the subject, culling the frames that produced a similar MRI pattern. So, since they watched TV containing text graphiucs, fames of it are bound to pop up sometimes.

[@Ergzay]

I call complete and utter bull. This is faked. That footage has clear crisp lettering that would be impossible to recreate and be there. It looks like they took their entire block of video footage and did a combinational overlay of all the video sources based on which videos MOST matched the brain scans. They didn't recreate any video, they only attempted to identify which video was actually being shown. Pretty cool, but the guy in the video is being obviously disingenuous. Bad UC Berkeley.

[@steamsteam6607]

I call the same. There is not real explanation done. OK images trigger brain parts and you can record which ones... How the fuck do activated brain parts translate to a picture? What is implied would seemingly have to mean each of these brain areas only responds to a certain colour and/or shape like pixels of an image. I highly doubt that's the way our brain proceses images. It won't be the first or the last scandal of "university discoveries". With all the budget, peer pressure involved I reckon they made some vague "artistic" interpretation of how the activied brain parts would be used to re-render some image. Or they somehow recorded the input of the optic nerve and translated that directly to somewhat of an image. I just don't get how they'd "calibrate" how some activated brain parts form an image... Maybe they used an algorythm to find patterns which somewhat match those displayed in the video to some degree and then used those incomplete patterns. Either way the explanation is insufficient to me to believe it really. And if that's the way it works then it's likely to remain very sketchy and ineffective as the "randomness" of the patterns would make accurate visual reconstruction pretty impossible. Either way... I'm not a neurologist. Perhaps it is pretty structured and predictable and they can find more patterns and algorythms "compensating for randomness". This needs more, better explanation to be believable I think.

[@memememe1987]

I hope you guys read these comments because I had an idea after watching this video back in January 2021 and I voice recorded it so I could remember it later and I am going through some stuff with a brain injury right now and I just thought now might be the right time to contact you guys with my idea because I know you’re working on vision and trying to eventually use the FMRI imaging to re-create what a person sees when they are dreaming I had an idea I didn’t know if anybody had already had the idea but I would like to share it with you and see if my mind is on the same track as yours

[@jos5975]

1:50 "u gotta c my ninja armer" is this added in post as a joke or....

[@carlcaloian5225]

They should do a study like this were the subject is under a lucid dream.

[@deltoid77-nick]

I remember there was a video here on KZbin where they reconstructed audio that was heard from a brain

[@voidofspaceandtime4684]

you ever find it?

[@Morenob1]

I was just searching for something like that but came to this video that would be great to hear what schizophrenic people hear

[@zeno182]

impressive.

[@academicalisthenics]

This kind of stuff was possible 11 years ago. Today we've got Neuralink demos. Just imagine how far this amazing technology will take us!

[@MrVids]

Impressive :-)

[@iliasasdf]

I would love to work on this. These guys have an FMRI maching to play with.

[@aplacefaraway]

near infrared imaging is approaching the capabilities of fmri shown here. such a system will be much cheaper and lightweight/portable. think ipod that beams information in and out of your brain at light speed!

[@pumpkinsoup5679]

So how exactly are they recording the actual dreams people are having, with the actual footage able to be shown? None of that answered the question.

[@thepodian9348]

+Tara Victoria They managed to reconstruct (very inaccurately, but still) pictures from the activity of vision-related zones of the brain. Since - while dreaming - we also "see" things, the dreams can theoretically be reconstructed in the same fashion. However, the brain also interprets vision before we see the world the way it looks - and they only worked on areas responsible for receiving images, not processing them.

[@anguswentzell1795]

They showed test subjects images and monitored the brain activity in the visual cortex, the brain reacted in different ways to each image and after a lot of pictures and a lot of brain activity they were able to associate brain activity in certain areas to shapes and colours. That's why it's not crystal clear yet, they can get a rough look at what the subjects seeing but not much more, detailed items haven't been mapped from the brain yet and would not be illustrated on the monitor however each persons brain is different and this tech would have to be calibrated for each individual

[@Mk_Elmnt]

Here before this blows up for no reason.

[@moshet842]

I won't believe it until they show a reconstruction where the person working on reconstructing hasn't seen the original movie.

[@Nesh108]

More on this?

[@Moshi_Ach_Loki-Eko]

Also need to do video games, because some dreams you are in control of. Maybe puzzles, reading, etc

[@ermahlee4980]

Why is this research needed? as in what are you using it for? How is helping people or who is it targeted at?

[@MrSnowgum2]

nice job noticing that

[@denemeskopkop]

What about the moment of death and after death?

[@hikari27506]

I wish there was one to record music and sounds instead of dreams and imaginations because I'm too lazy to compose a song with my own hands.

[@BROCKLEETTV]

is it possible to reverse the process and send artificial waves to the brain to in turn make the brain think its watching a movie or seeing an image

[@The_Cultural_Errorist]

Next step is haptic feedback, so you feel what the movie tries to convey. Imagine the possibilities for artists that try to create an immersive experience. Matrix anyone?

[@fake2]

Marc Pichel Like a fake vacation in the movie Total Recall 1990

[@PierLaurenziMusic]

could you please relate more that?

[@voidofspaceandtime4684]

+hugh if you are still here, tell me where you found that.

[@BattousaiHBr]

theoretically, yes. in fact i think that's roughly how vision-aid devices for blind people work.

[@saveashlii255]

You mention a lot of the benefits of this technology, but what are the risks. I have been reading alot about brain reading technology, but what risks could present with criminal or unethical use of this technology now that it is wireless?

[@kristenconklin311]

I started a petition2congress to protect our cognitive liberty and more. www.petition2congress.com/ctas/neuro-specific-human-rights-bill-proposal

[@itmyowninvention]

he said it could help them communicate with patients who can't communicate and that it might have some potential in psychotherapy and dream analysis.

[@PhilSmith94420]

Extremely astounding. Luckily today we can stay calm, cause this technology can't be used against us yet ...

[@MelodyNotMelanie]

Some of the brain activity seems to show more than what is actually shown to them on the screen, which is making me wonder about individual's perspective of things. Kind of like when reading, you can sometimes read a word incorrectly because you're assuming what the word is rather than actually looking at it and reading it. I wonder just how in depth the study is in terms of the subject's attention to detail and unassuming perspective.

[@markhodgson2348]

Hello have you ever put this on a person who is at the point of death

[@futurehistory2110]

Can't wait for dream recording to hit the mainstream! I'd win the award for "most fucked up dream" at the Dreamatography awards :3 lol

[@GhostCrow666]

I'd volunteer my brain too :D The images would end up looking like something out of an Aphex Twin or Tool video though...

[@maryanndaron6920]

I wonder how that works with aphantasiacs.

[@NotLikeUs869]

Sounds like that movie: Minority Report

[@ssurfur]

If those words are really coming from the brain, it is intresting to see that most most words r forming in response to colorful images that might stimulate imagination rather than images of human figures. Also, there r no words forming in response to phrases\words (All bets are off, Witha). Does it mean, we translate images into words and words into images, the abstract images r more difficlut to describe, so no words form and ordinary images do not invoke much formation of words?

[@vishalv6540]

Sir can we work any machine by only using brain impulses

[@davidmu2443]

0:17 - doesn't Canada in a few aspects needs places and organizations like that too? Like B.C, Quebec, and/or especially Ontario>?

[@PaulFrancisco-wi7se]

I want to volunteer for this experiment on dmt

[@primodernious]

anything is possible, you just need strong will.

[@weslow7642]

So we had the technology to read peoples thoughts 11 years ago?

[@MrJihadd93]

This technology needs to be used to decode psychedelic experiences like LSD25 and DMT. we have the technology to reveal the secrets of the universe now, cant wait till its used this way.

[@Hylianmonkeys]

can they reconstruct dreams if it is possible i would pay for a copy of mine

[@JohnRichard1991still]

I know the feeling

[@kekem823]

What would happen if you have aphantasia? Would you just a blank screen?

[@BerkeleyNews]

Hi, Prof. Gallant responds that the subjects of this experiment were watching a video; they weren't imagining anything. So the ability to do mental imagery would be irrelevant.

[@kekem823]

@@BerkeleyNews That's super cool

[@inkzy3105]

Am I late? Why did I only know about this now?

[@TrishMcArthur]

I'd donate my time for this. See inside my brain? cool!

[@JorgeGamaliel]

Amazing Science!! Now i´m going to be able to have the woman of my dreams because now i´m going to be able to 3D printing and rendering the woman of my dreams!! The future of 3d rendering software is a brain computer interface in order to do easily any 3d render ...

[@bernd_the_almighty]

yeah i sometimes have awesome music in my dreams but i have no idea how to write it down :(

[@MrSnowgum2]

PRECISELY my question, hope, expectation and wishful reply.

[@ZiiiP2142]

Because its cool.

[@whisperofficial6085]

I wonder if they still do this, I’d like to be experimentally scienced by them.

[@Yoeymeme2]

Ofc they still do this, it’s becoming more powerful with new space and inventions from AI

[@thebirminghamimage9256]

And how does this go when the content shown isn't known by the team "reconstructing" the film? I somehow feel this is biased by the researchers existing knowledge of the films shown to the subjects.

[@KafshakTashtak]

Insert Mind Blown Gif.

[@johnstifter]

Why did his colleagues tell him this would never work? Why are the "experts" always so childishly confident?

[@FrankLightheart]

I wonder if they can take a picture of my mind being blown?

[@w0rkhop555]

Is there a collection of the information of the progression of this tech in present day? And no im not looking for a warning or religious zealots to give me a warning of the dangers of the tech i am aware of its capability of malice i just want to know the optimistic things about it

[@danielhoy2053]

The "reconstructions" appear to be approximations or artistic expressions of what is thought to be perceived by the brains of test subjects. They haven't bridged the neural paths.

[@BattousaiHBr]

barely a surprise. we didn't have 4K screens just a few years after the invention of black and white TVs after all.

[@bioman1hazard607]

Exactly just give it a decade or 2 and we will find a way of transmitting images to and from our heads

[@ZSk8er12321]

still don't understand how they did that.

[@lunatic_3521]

Id be down to sit through 5 hours of lecture just to understand how they did it

[@reifuTD]

I wonder if the subject something like a cartoon the imagines become clearer or more blerrd? Like My Little Pony: FIM for example. There texture of the objects are simple, flat, and brightly colored. the Character also have the shape of horses but don't like lilke real world horses. Also what would happen when the animtion is more complex? Like anime level complex?