Can a robot pass a university entrance exam? | Noriko Arai

Meet Todai Robot, an AI project that performed in the top 20 percent of students on the entrance exam for the University of Tokyo -- without actually understanding a thing. While it's not matriculating anytime soon, Todai Robot's success raises alarming questions for the future of human education. How can we help kids learn the things that humans can do better than AI?
The TED Talks channel features the best talks and performances from the TED Conference, where the world's leading thinkers and doers give the talk of their lives in 18 minutes (or less). Look for talks on Technology, Entertainment and Design -- plus science, business, global issues, the arts and more.
Follow TED on Twitter: / tedtalks
Like TED on Facebook: / ted
Subscribe to our channel: / ted

[@gabriellaruth83]

This is what a bunch of people having been saying for years, the Bad student forgets all of the content BEFORE the test, and the Good student forgets it all AFTER the test. Memorizing isnt learning.

[@m_sedziwoj]

I age where books was expensive, and time to access to them was in hours or even days, memorizing stuff was needed. But now we leave in different time, were we find information very fast, so we should stop try to memorized things, we should start more understand things.

[@clairegittens3707]

Wow! That’s a brilliant summary. Last month, I was doing some review work in an English as a foreign language class. My partner teacher whispered to me, “Hey, that’s not on the test.” But here’s the thing. Language skills are cumulative. You need to build on top of your old skills. So, theoretically, every test should involve everything you’ve ever learned. But that’s not the case. Just remember this chapter and forget everything else.

[@wisdomsquare28]

Finally, someone who is awake. Thank you.

[@janaebert3059]

I wait for the day when the presenter reveals that he/she is a robot.

[@MRLONG758]

Oh what a creepy day that would be.

[@macrobody]

Jana Ebert I am sure I would have understand a robot better.

[@awethesound]

pshhhh

[@niveshproag8660]

"I would have understand" Oh the irony.

[@macrobody]

Excuse me, English is not my native language so I don't know what you mean.

[@JL2579]

Impressive talk and great summary at the end !

[@user-wx5lu8jq4w]

日本で2019年のベストセラー「AI vs 教科書が読めない子供たち」の著者・新井紀子氏の講演、もっと早く見たかったです。 復習を兼ねて、非常に参考になりました。大変良い内容ありがとうございました。

[@MaZe741]

Congratulations on passing the math exam. Now, Captchas!

[@lovefoolishh]

best comment here

[@Alex55555]

One of the best talks on AI

[@MegtaBubble]

This is what we want from Ted!

[@Morkvonork]

I love the math answer :D

[@xYoz]

The book which was written by Ms.Arai is very interesting!

[@lizzam]

I'm surprised a talk about AI ends up about our quality of education, our learning, teaching and assessment method.

[@vincenr8822]

Amazing talk.

[@catherine_404]

Entrance exam is not a test of being able to do any type of a job. It's a test of being able to pass a test. It also shows, you don't need to really understand a test to pass it.

[@donjuan69420]

I call dibs on are you smarter than a robot tv show

[@waveforms]

Need robots to give relationship advice. It'd probably be better than 80% we get.

[@nyx211]

If an AI program were able to get a high score on an IQ test would you say that the program is intelligent? If you say, "no", then isn't that an admission that IQ tests don't measure intelligence?

[@fetchstixRHD]

Interesting point you raise there.

[@qncsc]

you don't understand the purpose of IQ tests.

[@samikilic9149]

Marcus Aurelius I agree 👍🏻

[@lachlansleight]

The IQ test was designed to measure human intelligence. It's an interesting thought to think about what an IQ test designed to directly compare humans and AI systems would be like...

[@redtails]

Replace AI with monkey and answer the question yourself ;)

[@davidswe98]

There is always an asian robot better than u

[@SirNightmareFuel]

kzbin.info/www/bejne/jqnOfpKZo895nNk

[@emersonlee6422]

we dont need robot! fcking satan!

[@thanasisathanasiou1260]

It responded with "2. Were almost there" probably due to the procedural understanding of events starting and finishing. Or probably because "wait" is a remark directing your attention to time. Possibly a combination of the two. One thing is for sure tho, AI's will never forget to tie their shoes... so yeah; makes sense.

[@liemnguyenhuynh8245]

thank you for giving me the meaning of AIs

[@notead]

Allow the community to create subtitles for this video please. What she is saying is very interesting but it is hard to understand her sometimes.

[@sushitime9496]

I read the 1rst sentence in the english test, I automaticaly assumed it was answer 2. Am I a robot or is the robot as lazy as me ?

[@sushitime9496]

When you read it doesn't (or maybe ironically) but if you just look over the first line quickly, it makes some sense

[@09I4_]

With the development of artificial intelligence, a major change in education will be needed. In order to solve this problem, I think it is necessary to develop students' literacy and introduce an understanding-based education method. In order to develop literacy, I think it would be better to actively engage in various activities such as reading texts and writing diaries at school, and to introduce a backward design model into the class for understanding-oriented education.

[@memoryhero]

I kept hoping Robert would eventually get into the university.

[@sethgrasse9082]

memoryhero Robert? Isn't it Todai Robot?

[@CDmc98]

I can't believe the overtime sign isn't more discussed 12:16

[@nicolasmicaux8674]

Very interesting

[@primustorus5116]

The Tokyo University entrance exam is Todai for.

[@metleon]

I could barely understand her accent so I turned subtitles on. Further proof that the robots are winning.

[@SnowmansApartment]

so robots are dyslexic.. 😵.. i might bee a robot after all..

[@5even5eals]

As well as a majority of students, it seems.

[@overwrite_oversweet]

I wonder if we can get some of those AIs into teaching.

[@MrMielke]

So the machine prefers to have Puns in its code? I can't blame it. :P I am curious to know how it failed the shoelace question. I assume it picked up on the "just another few minutes," but the word "wait" suggests something that requires fixing, where answers one and four are the only ones associated with a need to pause. (You can map the sentences to physics, where a computer is a lot better.)

[@PersonalPower]

Could a robot pass most bachelors degrees? Yes, because you only have to memorize a lot of phrases to pass. Might struggle at the bachelor thesis, but otherwise I dont see a Problem. I probably learned more from fellow educational KZbin channels than from my 2 semesters at university.

[@BagoGarde]

If Ai can write news article , i believe they can write a thesis.And with some physical aid, even collect information.

[@s.vincentparish8168]

What university did you attend? UT specifically aimed to measure student understanding by gearing exams towards critical thinking rather than memorization.

[@PersonalPower]

I'm currently studying in a smalltown in Germany. We have some sort of critical thinking but most of the time its memorizing I think.

[@yanusanders7686]

Depends completely on the school system. In the UK all tests are made in a way so that just memorisation gets you a maximum of a C or D.

[@Sdhector21]

Bago Garde The whole point of the talk is that AI cannot "write", rather summarize existing text. Writing involves creativity, which implied understanding abstract ideas. Until AI can understand meaning it won't be able to "write".

[@Afterlifesinner]

My only gripe with this talks is that she could have elaborated her conclusion a little more. It was worth listening to.

[@aposslex]

Even the subtitles are struggling. Think I got the jist though

[@HakunaKun]

Probably it's because the old evaluation systems are probably outdated and the teaching process requires an important review urgently 😔...

[@pigeonwizard]

If its problem is understanding meaning in written languange that might be the reason why AI tends to develop its own language (as it happened with the Facebook AI tests) when it has to "talk" to other AI. The human brain is wired in a way that our language is the best way to communicate for us, but for a machine built differently this might not be the case. Therefore it doesn't perform perfectly in the human communication field, but if it were allowed to develop its own communication system without being shut down it could outperform humans in all fields.

[@Sdhector21]

Jessica Cremonese Can you point me in the direction of those Facebook AI tests? I'm skeptical an AI could develop their own language before even replicating an existing one. I'm curious what exactly is meant by saying it developed it.

[@pigeonwizard]

Hi! Yes, here is a couple of articles from supposedly respectable sources of information: www.telegraph.co.uk/technology/2017/08/01/facebook-shuts-robots-invent-language/ www.independent.co.uk/life-style/gadgets-and-tech/news/facebook-artificial-intelligence-ai-chatbot-new-language-research-openai-google-a7869706.html I cannot find the original website that provided the info to me first... sorry for that. But I read these articles and they explain it quite well. Long story short, the supervisors of the project noticed that there was a change in the pattern of words which did not make sense at first. However it seems that there was a pattern in the "new" language, so they reprogammed the AI with a new rule: stick to English. This is very fascinating!

[@TheArtPerspective]

already passed Berkeley.

[@sarinvictims]

She misunderstood the concept of standardized test. We don’t have any standardized test for college entrance examinations in Japan. We only have common subjects based test. I hope she can be a little bit more accurate.

[@quyetthango2848]

Little UK take me come here !!

[@Axphey007]

She should also start lullaby yt channel

[@editorjohn8803]

As an English professor, I know that there is no way for an AI to write a B+ or above essay (based on the standards of the top 50 universities in the world). There may be a great deal of excitement over an AI program's potential, but I have yet to see even one sample circulating anywhere, including the Internet, that shows that AI can write an essay that reveals a semblance of wisdom and critical thinking, which are necessary for any reason for a student to write an essay.

[@92000]

Who else thought that "Todai" robot was an text error (Today)? :p

[@tadashiise2845]

The student finished their schooling doesn’t studied difficult subjects but University of Tokyo giving hard exam almost the Mathematics and studying English is late. How about when you graduated in Hibiya High School and Kasei Academy?

[@akritiarora1904]

This robot will easily get into Indian universities

[@MRLONG758]

How so?

[@akritiarora1904]

MRLONG758 because the education system sucks that's why I cleared my entrance by mistake and got a rank of 18 all over India thousands of people register for it it was MCQ based and I barely studied

[@akritiarora1904]

MRLONG758 I also gave other entrances like of computer application course and business administration I don't even know maths science or accounts I got 500 rank in computers which could easily get me into best college and 3000 rank for business it is kind of a hit and trial

[@suryasharma9410]

Akriti Arora : Hey Akriti , In which entrance exam did you get AIR 18 ? , because in country like India . It is very tough.

[@akritiarora1904]

Surya Sharma GGSIPU

[@desiredditor]

Hey did u allow the kids to Google just like the ai

[@mohsonkhalafjabbar9655]

about that issue i think ai like notebook (smart notebook)

[@TheReject182]

The answer is yes because University = Learn things and copy past them from your mind to your sheet of paper :/

[@Sarruji]

Well I would pass any exam if i had the internet as a tool during said exam.

[@TheRealmDrifter]

Did she say 'Fucktard questions'?

[@DheerajBhaskar]

TheRealmDrifter factoid questions. I heard it in a weird way as well 🙂

[@smoldogbigbobs]

factoid

[@alexjackson4103]

Lmao, at 8:12 she does the body language equivalent of "Please Clap"

[@MarkVanReeth]

Can someone please develop an AI that can translate whatever she's saying into human speech? I tried listening to the podcast version in the car, but wasn't able to understand half of it. :(

[@Xbotto]

video is 13:37 aswell

[@holycow6954]

これ好き

[@uditrockx]

She made me worry.

[@Erikulum]

Wait, you can't use the internet for university entrance test right? That robot is cheating!

[@suryasharma9410]

Erikulum ; but they are talking about the capabilities; even if humans are allowed to use internet, they will not be performing that well.

[@Sdhector21]

Surya Sharma Yes they will. If students were given Google I doubt the AI would be better than 10%, maybe even none of them.

[@suryasharma9410]

Hector Sanchez ; I totally agree with you , but if we take time duration in consideration, then they are far better than us. I BELIEVE humans are great and having incredible powers . If human brain is able to create AI , then it is already very far in that race with AI.

[@5even5eals]

The one who is cheating here is university. Test answers should not be googleable in the first place.

[@suryasharma9410]

5even5eals ; all the questions are from our discovered and invented knowledge , how can you prevent that ? Google is a search engine and not a helping hand in the examinations.

[@anhsinhvienKyThuat]

I have an idea for a machine that moves objects through space and time using quantum theory and mathematical coding ....... I am in need of advice and suggestions from the you ...........STM

[@user-bc3ym9xp7o]

何このかわいらしい声📣

[@BJ-xh8tg]

11:11 "...failed to answer (to) this question.." u BOT no hUm@N !!

[@thammarongp]

Arai san, please translate your latest book in English. A Thai translator recommend me to read. I think it's useful. Thank you.

[@danthetravelman9113]

The AI got an open book open source test while the students did not, more and more universities are expanding their student base for profit, and standardized tests aren't necessarily the best tellers of knowledge. We have all of the same capabilities and resources as AI and that is why universities should teach how to reach solutions with the given resources and technology. More tests should require abstract thinking in order to reach a new idea or solution, and for this reason that is why the computer failed the language questions. Any one of us can find the answer to simple bits of history or math problem, but lets say I ask you. . . How can we effectively change the undesirable situations that await us in the future? You must include a plan that deals with the monetary system, government, international relations and everyday lifestyle. . .

[@Leandro80446]

She is robot!

[@aleksandersuur9475]

That's a really good video on the incredible power and debilitating limitations of the AI technology as it is today. Very few people understand much of anything about what we today call AI and get their impressions of what it is mostly from science fiction genre. Reality has little to do with fiction however, in this case - almost nothing at all.

[@joehealy1136]

Please say it's no

[@Sobieski526]

I think the study is flawed as the computer has access to infinite information & solutions available on the internet. A smart person equipped with a smartphone and a good understanding of search engine capabilities can do the same job in general knowledge and even essay writing. Math might be harder but still doable using online tools for assistance. It would be interesting to see the AI that can compartmentalise and store information as humans do without having access to the worldwide web. Yet she has a good point - we urgently need to teach people how to read, process, and access this wide pool of information availiable to us!

[@drpporto]

Ridiculous questions so far. I am really upset about this. What is the point in an test about this?

[@kanwaldeeppuri8761]

This thought could be sheared without involving the country

[@Sdhector21]

I wonder how much better would the students do if they are allowed to Google their answers...

[@suryasharma9410]

Hector Sanchez ; remember one thing . Only those students will answer perfectly which are sharp and having orientated brains . I believe, nothing can match human intelligence, but if we are using our brain, only then. Rather Al would be far better than us.

[@user-kr3wy6bd5o]

I've read her book before. I think we have to bring up the future generation(and ourselves¯\_(ツ)_/¯).

[@mohsonkhalafjabbar9655]

many thanks for arab translate شكرا للترجمه بالغه العربيه

[@alph4966]

暗記型人間≒エキスパートシステム ロボットで十分代替可能なホワイトカラー職が危ない

[@lloydgush]

Don't get all alarmed when robots do what they are good at. Now go ask the same robot to load the truck on a unspecified location.

[@Kongolox]

educators has been saying this for ages...

[@basharatullah285]

Google could you the same all you have to do is put the information into a system not a big deal

[@fleXcope]

Needless to wait till 2020, when university campuses will no longer be necessary.

[@tobiasuretheworld1434]

Lol . congrats of predicting the future

[@hermansterdal3587]

She does not give enough proof for me to beleve her. Where is the 600 word essay it wrote?

[@danielsteel5251]

It didn't 'write' anything; it plagiarized various sources on the Internet, or in its database or whatever. (Incidentally, so too do many human students, until they're able to think for themselves.) There wouldn't have been anything generated by the machine that was both _appropriate to the situation_ and _novel and innovative._ But those characteristics (i.e., the hallmarks of creative self expression through natural language) would not have been necessary for an essay that would satisfy a grader. So why are you suspicious? Most of the KZbin commentary I come across reads like it was generated by a bot (whose programmers' English skills were about as strong as the TED speaker's). I'm willing to believe the essay was 'good enough' to pass.

[@GumRamm]

I'm sure you could look it up somewhere, but also it's in Japanese as you can see in the clip at the beginning

[@RidiculousThisIs]

AI isn't thinking, its still doing a very narrow, specific task which is written by a human. This is not AI yet.

[@crazyprayingmantis5596]

Pretty sure this woman is a robot

[@KennethWuAU]

Here's the problem. Why is the way human solve problems being used as the standard for intelligence here? The method one chooses to answer a question will always be judged by humans in this case, despite the fact that it has reached a correct answer. What if the 'machine' has solved it using it's own method that is counter intuitive to everything we understand? Are we deliberately nerfing the capabilities of AI to solve problems by giving it human problems and are understandable to us? Do solutions always need to be short and concise for us to understand?

[@sushloura1536]

Strongly no

[@novotnybrandon]

It shouldn't have taken an AI to point this out. Any educator who has been contempt with the current system should be ashamed.

[@theCUBE403]

Will AI speak in bad engrish too?

[@Myrslokstok]

But let the kids surf the web as the AI did, and then compere. In reality you have google at your fingertips.

[@BagoGarde]

What is a good to have when the AI will be doing and replacing everyone?

[@meamzcs]

Sensei why that? I don't think software engeneers are irreplaceable but i think if you are one you will have a job at least almost until there are no more jobs. Because if AI can program, it can basically replace every job with it's software.

[@drewkavi6327]

Bago Garde Then you don't have to work and can live a life of leisure forever why are you complaining about not needing a job

[@kinsmed]

+zomge Yeah, get rid of all those uninformed judgmental people, right?

[@BroCactus]

Programmers will be among the first or second class of employment to be lost. They are literally coding their replacements.

[@meamzcs]

Eric, maybe but as i said, it could write software so fast, that it wouldnt take long for it to replace every other job with its software

[@kennoquick]

But it is dependent on humans

[@kennoquick]

But honestly it will create jobs

[@F0r3z0r]

Yeah, even the robots that are created for manual labor needs to be controlled, they also need a security maintenance and another jobs.

[@jasondads9509]

and it will create less jobs than the number it replaces

[@rahb3rt]

Or it shows how useless these entrance exams are.

[@pzrkstinger]

Robert Davis explain

[@rahb3rt]

Youre taking an exam that does not show any real life application just how fast you can recite things from memory. Real world doesnt require you to memorize facts but instead apply knowledge and be able to learn on the run new concepts and adapt. The education system is a scam sold to students thinking that if a then b then c, but sometimes that intermediate step b is unnecessary to accomplish any of it. Its an imposition of impracticality. The robot was able to memorize facts and look them up, thats easy. But interpreting meaning or having any depth application is what makes it still a robot. If society wants to raise robots thats fine, but they will all get out paced by real robots that are better at performing these sorts of tasks. Its like asking a student to memorize the multiplication table when they have a calculator, sure its a great skill to have but why not teach students to learn about the theories of mathematics and why numbers can be multiplied or proofs than something that is easily doable by a machine.

[@qncsc]

*you missed the whole point: the exam is to identify a distribution.* and that distribution is reflecting some amount of rigor that will be coincident to the material that will be covered in future higher education. those who can accommodate more complex forms of data, analysis will reflect that and be able to reflect similarly in future domains. education is about (a) selecting those who are cognitively capable (and to a lesser extent willing), and the (b) curriculum that deconstructs the basic principles, allowing for future efforts of complexity and abstractions upon which to build greater competency. what the robot shows is that even rudimentary competency can be achieved where vast numbers of humans have great inability. 1 million robots will easily displace those 1 million humans; the economics of a robot are all superior to the human. and therefore, the humans will have no compelling economic argument to advance their economic purpose and agenda. the expansion of artificial intelligence can then scale up the economic value chain overtaking more and more human functionality. at every turn the robot will offer a more compelling economic expression, so long as it outcompetes the human on task. this is what the lecturer in the video sees -- but is only implied. this was an excellent video!

[@rahb3rt]

Education has nothing to do with being "cognitively capable" because there are plenty of people who are. You cannot translate a subject like english or social studies to be a numbers game when inherently the subject itself is not a numbers game. Something like physics and chemistry sure to some extent you can translate to numbers. But it makes no sense that a subject like English can be translated to a functional number where subject matter is opinion based and grammar is not a definitive reflection of anyone's ability to speak, read or write.

[@nofanfelani6924]

The entrance exams is useless for most of us, yes. But one in a thousand of student who pass the test will become a great scientist, another one will become a physicist or another great person who will invent a new things for humanity. We wouldn't know who is that "one person" if we only being thought the basics and practical knowledge at school. Who do you think invented the smartphone you have now? It's one of the student who passed that "useless" entrance test

[@jahdeedmanlungat8143]

It depends on how many terabytes and information loaded on a robot.

[@nofanfelani6924]

Everything is going well, until the AI started asking "what's the importance of human if we can learn better than them"

[@Gribbo9999]

I wrote essays and didn't understand much too. Now 50 years later I now understand a bit (maybe) . The real issue is I took 50 years to get here. Maybe our Japanese robot friend will get there in 50 minutes or 50 seconds if it goes quantum. We might be right to be worried. Very soon the decision to pull the plug on AI will be gone and we won't necessarily be AI's first priority.

[@ddeemmoonn]

But whats the point of this experiment? The AI and the students aren't on the same terms. Connection to the internet of the AI is the only thing that makes it better. Its like comparing a student that does the exam from memory and a student which allowed to use his phone. The entire process of learning its basically manually filling up memory in your brain, of course the AI would win because it has alot of memory and it is allocated so basically there's no really a chance to beat it in this particular area. Thats why exams should be about opinions, creativity and problem solving. But what will happen when you'll add some aspects of life? And life is full of chaos. Humans learn from trail and error. So if for example there's a question with data missing for example a typo , the AI wouldn't know how to deal with it, you can give him a spell checker that will try to do it, but the AI cant grab the context of the rest of the sentence. But us? We can decipher information with minimal data given, because our brains already understand what happened, there was a typo, so we should consider the context of whats before and after and choose the appropriate correction and thats happening instantly in our brains without a need of the internet. Basically what im trying to tell that humans not takin emotions and deeper understanding of things into consideration, we also have some kind of built in "debugger".. on a daily basis we solve real life "bugs" . We have the ability to bend the rules or break them. And reprogram our brain. So a real AI would be that one which could reprogram his own code in a way that it is 100% different from the original code and would have a debugger that could grasp unordinary situations

[@BenTajer89]

"How was this robot able to outperform our students?" - Robot has access to the internet and unlimited memory... The task that the students have is fundamentally harder than that of the robot, give the students a smart phone and then compare the scores.

[@overwrite_oversweet]

The robot could simply load say, 1 TB into cache. And, no, 1 TB is not larger than the capacity of the human brain, whatever that might be, though yes, it's a different type of memory. Computer hardware uses a fundamentally different paradigm to wetware, on which information stored is too lossy for truly accurate reproduction even for "well memorised" items. Humans truly have the advantage in some cases, but in certain tests, "consciousness" and the ability to comprehend is a disadvantage, not an advantage. Maybe we ought to look at optimising tests to measure something more useful.

[@revdcdon970]

NO - Instructions - Turn in your test at the point you cannot answer any question. Second question - What is your Blood Type?

[@TreesPlease42]

This video is 1337.

[@seanmcguire7974]

??

[@RoyMuffinStar]

This speech brought an attention to me, personally in my studies. Throughout my diploma learning I've made used of countless memory techniques to ace my exams, combined with true understanding I can bring out the answer that no AI can, however memory was one of the lacking thing, memorisation. If AI could produce meaning and understanding, the end of humanity will be close. I doubt it will happen any time soon though.

[@ashleymathura]

Key lesson - our kids need to read more and understand better. The benchmark now just happens to be smart machines..

[@VictorGallagherCarvings]

So "educators" will have to shift from indoctrinating students to teaching critical thinking and logic.

[@andrewclover1462]

"Can a robot pass a university entrance exam?" I'm not sure. Thankfully, if it cannot pass the entry exam, the government will give it low interest loans so it can waste everyone else's time being nursed through classes in college it should have passed in high school.

[@ichweissnicht9608]

i dont know, but 1337

[@llivezey]

Are real students allowed to Google their answers and plagiarize to write their essay? What if the robot had to do it without help from the outside world, like students do?

[@ZiemniakPospolity]

Why should we think that AI is smarter than a human when AI can use google to find correct answers during the test and a human can't? Give google to the human and he will probably score better.

[@fedswithful]

Is she AI...

[@SoulxLucifer]

I'll be honest, I watched this at 1.25x speed...

[@ys-xl9ft]

When Japanese speak English, it takes longer time.

[@MI-ns5vn]

But you cannot speak this topic in Japanese at natural speed at all.

[@justrandom6751]

Todai robot ?

[@DheerajBhaskar]

Just Random oh I heard it as todai Robert

[@user-rk8bn4rs5v]

Just Random She says "Todai robot" It means Tokyo Daigaku (university in Japanese) Robot. Many Japanese call the university of Tokyo "TODAI". That is the most famous university in Japan.