Information entropy | Journey into information theory | Computer Science | Khan Academy

Finally we arrive at our quantitative measure of entropy
Watch the next lesson: www.khanacadem...
Missed the previous lesson? www.khanacadem...
Computer Science on Khan Academy: Learn select topics from computer science - algorithms (how we solve common problems in computer science and measure the efficiency of our solutions), cryptography (how we protect secret information), and information theory (how we encode and compress information).
About Khan Academy: Khan Academy is a nonprofit with a mission to provide a free, world-class education for anyone, anywhere. We believe learners of all ages should have unlimited access to free educational content they can master at their own pace. We use intelligent software, deep data analytics and intuitive user interfaces to help students and teachers around the world. Our resources cover preschool through early college education, including math, biology, chemistry, physics, economics, finance, history, grammar and more. We offer free personalized SAT test prep in partnership with the test developer, the College Board. Khan Academy has been translated into dozens of languages, and 100 million people use our platform worldwide every year. For more information, visit www.khanacademy.org, join us on Facebook or follow us on Twitter at @khanacademy. And remember, you can learn anything.
For free. For everyone. Forever. #YouCanLearnAnything
Subscribe to Khan Academy’s Computer Science channel: / channel
Subscribe to Khan Academy: www.youtube.co...

[@vandanachandola322]

I remember my teacher in high school defined entropy as "the degree of randomness". I decided it was an abstract concept that I don't get. Now learning about information entropy in my master's class, I found this video and I'm so glad I did!! Thanks, it's very well explained :)

[@itsRAWRtime007]

good video. i like the way it shows the intuition behind the concept, that is the reason why the concepts actually exists rather than plainly defining it and then showing its properties.

[@youngsublee1102]

couldn't agree more

[@amirizaiah7179]

I guess Im kind of randomly asking but do anybody know of a good site to watch newly released series online?

[@andrewrandall9989]

@Amir Izaiah Flixportal :D

[@amirizaiah7179]

@Andrew Randall Thank you, I signed up and it seems like a nice service :) I appreciate it !!

[@andrewrandall9989]

@Amir Izaiah you are welcome xD

[@bobbobdog]

at 1:24, i would argue the 3rd question (ie, the question on the right of 2nd hierarchy) should be "Is it C?" (or "Is it D?") rather than "Is it B?" (i think this is so because, as the 1st machine answered "No" to the 1st question ["Is it AB?"], it essentially rules-out both A and B, leaving only C(or D) as the possible outcome; hence no role for "B" anymore)

[@marcuschiu8615]

yea, I agree with you. dam, so many mistakes in this video, 1:24 and 3:50. makes me question their reliability... good video though

[@dien2971]

I thought I understood wrong lol. Thank you!

[@muhaymenulislam1942]

But here the probability of D is 25% which Is more than 12.5%, so in the second they ask is it D? .

[@pedrogorilla483]

I have asked several professors in different universities and countries, why we adopted a binary system to process information and they all answered because you can modulate it with electricity, the state on or off. This never satisfied me. Today I finally understand the deeper meaning and the brilliance of binary states in computing and its interfacing with our reality.

[@kempisabel9945]

this video blew my mind away. Thank you! I love these intelligent yet fun videos!

[@hrivera4201]

previous lesson: kzbin.info/www/bejne/jaqkpYKnm6iceNk next lesson: kzbin.info/www/bejne/iqnOcmiLjZmen9U

[@phycosmos]

thanks

[@mathaha2922]

This is one of the most informative -- and I use that term advisedly -- videos I have ever seen. Thank you!

[@MaryMary-ep4hd]

Ingenious interpretation! I applaud!

[@salrite]

What a Beautiful Explanation!!!

[@someshsharma6683]

Awesome explanation with a very intuitive example.Thanks a lot...

[@antoinecantin1780]

What a formidable way of visualizing and introducing information entropy. Your contributions are deeply appreciated

[@boredomgotmehere]

Makes it all so super clear and easy to follow. Love this.

[@boredomgotmehere]

Just a tiny error at 3:50 - the final calculation shld be 0.25*2.

[@Dhanush-zj7mf]

1:24 You are asking same question twice because you already asked "Is it A or B" in the root if the answer is no that means "it will be either C or D" but you are asking again whether it is B, or not in the sub branch. It should be either "Is it C" or "Is it D".

[@MohdFirdaus-fk6no]

yes, you are correct

[@suryacharan5184]

What a video!!....This is how education should be.

[@twoplustwo5]

Kudos for linking number of bounces -> binary tree -> log. And overall very nice explanation. That's like 3rd explanation for info entropy i liked.

[@YYchen713]

This is such a great way to explain information entropy! Classic!

[@daihung3824]

I have one question: Say p(A)= 0.45, p(B)=0.35, p(C)=0.15, p(D)=0.05, then 1/p(D)=20, log base 2 of 20 = approx 4.3, however, number of bounces should remain 3 is it? Would anyone mind explaining this possible difference? Thanks a lot!

[@Ewerlopes]

Perfect explanation! :)

[@waylonbarrett3456]

I found a few errors. Am I the only one seeing this?

[@youngsublee1102]

Wonderful idea of "bounce" that express the amount of information. It's so exciting.

[@raultellegen5512]

Amazing video. Seldom seen a better explanation of anything. Thanks!

[@BambiOnIce19]

Perfectly well explained. The best video on information entropy I’ve seen so far

[@karrde666666]

why can't textbooks or lectures be this easy

[@Hopemkhize-d2i]

Tell me about it😢

[@csaracho2009]

I have an answer for that: The zen pupil asks the master, is the flag moving with the wind? The master replies: neither the flag or the wind move, it is your mind that moves.

[@SAGEmania-q8s]

Thank you so much. It explains the entropy so well.

[@YuriValentines]

This video has explained entropy better than any teacher I've had in my entire life. It makes me so angry to think of all my time wasted in endless lectures, listening to people with no communication skills.

[@miketor2011]

Great video but is it just me or there is an error on 3:49. The correct calculation for the number of bounces should be 0.5*1+0.125*3+0.125*3+0.25*2 = 1.75 instead the video shows 0.5*1+0.125*3+0.125*3+0.25*4 =2.25? Any thoughts?

[@musicmaker33428]

I was just thinking this. Thank you for pointing it out. I thought maybe I misunderstood something fundamental.

[@rkiyanchuk]

Yep, that's a typo, number of bounces for P(D) = 2.

[@bouzouidjasidahmed1203]

Very comprehensible thank you !! it very helpful

[@potatocoder5090]

Brilliant explanation. So simple yet so profound. Thanks!

[@tythedev9582]

Yessss I finally got the concept after this video.

[@vlaaady]

The most intuitive explanation

[@gaofan2856]

The most beautiful explanation of entropy

[@mostafaomar5441]

Thank you. Explains the intuition behind Entropy very clearly.

[@jonathan.gasser]

Wow, what a presentation!

[@kartikbansal6439]

Loved the piano bit towards the conclusion!

[@nomann5244]

you are truly a genius.

[@yudong8820]

Really good one, thanks!

[@osobliwynick]

Great explanation.

[@edwardjurkowitz1663]

Excellent video. I think one point mistakenly refers to "information" when the author means 'entropy.' Machine 2 requires fewer questions. It produces more information and less entropy. Machine one produces maximum entropy and minimum information. Information is 'negative entropy.'

[@tingwen524]

Great video! I totally understood entropy!

[@FrancescoDeToni]

Isn't there a mistake at 3:50? Shouldn't it be 0.25 x 2 instead of 0.25 x 4?

[@philtrem]

+Francesco De Toni yup!

[@sighage]

Yes, it's 2.25 I guess

[@rah2023]

It's indeed a mistake

[@nikhilsrajan]

@@sighage no it's 1.75, just there was a typo. you get 1.75 with 0.25 x 2

[@argha-qi5hf]

I can't imagine how someone could ever come up with such abstract ideas.

[@swazza9999]

This should have more likes!

[@ГримМорген]

The concept had been presented to me on some online course, but until this video I didn’t really understand it. Thank you!

[@russianescapist5262]

I loved this surreal music and real life objects to move in a grey 60s like atmosphere.)

[@lovenishkavat46]

My problem has been sold 10 years ago you are superman

[@Puneethmypadi]

Now I understand decision tree properly

[@shelendrasharma9680]

Best explaination , salute ....

[@mohammadrezamoohebat9407]

It was perfect. thx

[@daviddeleon292]

Huh??? Why am I finding out that information entropy was a concept. MIND BLOWN!!!

[@science_electronique]

قوة في الشرح و وضوح

[@chandragupta2828]

awesome video!

[@Chrls5]

Nice!

[@n0MC]

this is wonderful. thank you

[@jayrar6645]

so just to clarify, is the reason the decision tree for machine B is not the same as for A as you ask less questions overall? and how do you ensure that the structure of the decision tree is such that it asks the minimum number of questions?

[@FARHANSUBI]

Can someone please explain why machine 2 is producing less information? Shouldn't it be more information because we're asking fewer questions? Or is it the more number of questions we ask, the more information we receive?

[@souvikmajumdar5604]

Thank You

[@malevip]

Another way to look at entropy: Measure of distribution of probability in a probability distribution.

[@AhmedKMoustafa2]

great explanation bro :)

[@leoyuk-tingcheung3587]

Could anyone help explain why less uncertainty means less information (Machine 2)? Isn't it the other way round? Many thanks.

[@TheOmanzano]

there is less certainty in machine 2 because on "average" there will be less questions...meaning after many trials on average there will be 1.75 questions needed to get right result meaning there is less variety, randomness, chaos in machine 2 due to the fact that "A" will be occurring alot more than other letters

[@hirakmondal6174]

Think of it as a Hollywood film..where a police inspector interrogates a criminal and he must speak truth each and every time. After 175 questions the inspector found out that he knows no more than that, where as when he interrogated other criminal in an adjacent cell he found out that after asking 175 questions he can still answer 25 more.. Now U Tell Me Who Has More Information? . . . . U are welcome!! 8)

[@Exhora]

HIRAK MONDAL That was a great example! Thank you so much!!!

[@salrite]

Less uncertainty means your chances to predict the outcome is more aka predictability increases and hence it refers to less information, means you don't require as many bits (say: to represent outcome) as you would when uncertainty was high or the outcome was pure random. Make Sense?

[@rcgonzalezf]

I also have this question. I guess we need to define information in this context, for the answers and the video itself I think they're referring to data and as I posted below, data is different than information. More questions = more data to get the same information (the output), but I might be missing something.

[@ahmedelsharkawy1474]

just awesome

[@btsandtxtloverstraykidzfan3486]

What are some good books on this topic ?

[@shepbryan4315]

Why is the number of bounces the log of the outcomes?

[@juanpablovaca-lago5659]

Is there a direct analogy for the second and third law of thermodynamics and the information entropy?

[@jorgeleirana-alcocer5642]

The equation in 3:48 should result in 2.25 not 1.75 (0.5*1)+(0.125*3)+(0.125*3)+(0.25*4)= 2.25 I think it should have been (0.5*1)+(0.125*3)+(0.125*3)+(0.25*2)

[@ChusKon1]

Beautiful

[@hiteshjambhale301]

Hey there is one mistake at timestamp 1:26 ..... the question should be "is it C?" instead of B

[@dissdad8744]

Good explanation! If I wanted to calculate the entropy with log2, which calculator can do this? Is there an online calculator for this? What would be the best approach?

[@ElectricChaplain]

Hans Franz Too late now, but log2 b = ln b / ln 2 or more generally log 2 b = log base a of n / log base a of 2.

[@FGNiniSun]

Hello please why does the number of outcomes at a level equal to 1/probability ?

[@hingaglaiawong7815]

at @3:15 I think there's a typo? The last term should be 0.25*2 instead of 0.25*4 I guess.

[@hirakmondal6174]

Why outcome is 1/p?

[@anirbanmukherjee4577]

Possibility of outcome=1/number of possibility

[@alexhsia9510]

What do they mean by number of outcomes? Can someone give me an example using the ABCD examples they used?

[@temenoujkafuller4757]

Yes, I asked myself this question and watch it twice. (5:45) Count the number of branches at the bottom The number of final outcomes = 2^(number of bounces) Therefore, the inverse function of exponent is logarithm >>> The number of bounces = = the number of questions = log_2 (number of outcomes)

[@_crispins]

nice!

[@YTBxd227]

still confused why #outcome=1/pi

[@CZRaS]

because you need to "build" a binary tree to simulate bounces. E.g. you have probability p=1/2 (50%). From that outcome = 1/1/2 = 2. If you have p=1/8 (12,5%), you get outcome = 8. From which you can get the log2, which is basically the level on which the value is in the binary tree.

[@suliu2933]

Great video! I can follow it but I have trouble understanding the problem statement. Why "the most efficient way is to pose a question which divides the possibility by half"?

[@vandanachandola322]

Too late, but maybe because we're trying to ask the minimum no. of questions (and therefore going with the higher probability first)?

[@xxxxxx-wq2rd]

is it valid to say less entropy = less effort required?

[@최로봇]

if it makes us ask less questions, doesn't it mean it provides more information?

[@mikibellomillo]

note: number of bounces - entropy is maximum when all outcomes are equally likely . when introduce predictability the entropy must go down. thanks for sharing this video! God bless you!🎉

[@gustavomartins007]

Muito Bom

[@kawaikaede2269]

cool

[@sholi9718]

can someone explain in #bounces=p(a)x1+p(b)x3+p(c)x3+p(d)x2 at 3:44 , how numbers 1,3,3 & 2 came?

[@achrafamrani2730]

x1 is the number of bounces needed to get to point A which is equal to 1 (steps needed for the disc to fall in case A) , x2=x3 is the number to get to points C and B separately and it equals to 3. For x4 it takes 2 bounces to fall in D

[@sanadarkia2724]

can't we just ask one question? is it abc or d ? edit: nevermind, i just figured that 1 bit removes uncertainty of 1/2

[@pablobiedma]

So if I recall correctly, the one with the highest entropy is the least informative one, then the, if a machine generates symbols, and apply the formula for each symbol, which symbol provides the most information? the one with the least amount of bits? how does that make sense, isn't it the one with the highest amount of bits? calculated by p log( 1/p)

[@divermike8943]

What if machine 1 had 6 letters ABCDEF? How many questions on average would you have to ask? Using the Bounce analogy I get 16/6 = 2.6667. But -Log2(1/6) =2.585. What am I doing wrong? Is it ABC? If yes then Is it A or B? If yes then Is it A? for A 3 questions. 3 questions for B. . But if it's C only 2 questions : Is it ABC, Y. Is it AB, N, then it's C. Likewise if it is D or E or F? "3 Questions for D & E. Is it ABC? No. Is it D or E? Yes Is it D? 2 questions if it's F. 3+3+2 + 3+3+2 = 16. 16/6=2.6667 on average not 2.585 =log2 (1/6).

[@monicarenas]

In minute 3:51, I guess there is a mistake, for p_D, the value is 2 instead of 4, does not?

[@sanjayrakshit8797]

Heckin Shanon

[@assylblog]

Cool beat

[@samhe331]

I think the math at 3:52 is wrong.. should be 0.25 x 2 instead of 0.25 x 4 but the result is right 1.75

[@mikechristian-vn1le]

Language is a much more powerful invention than the alphabet, and written language -- Chinese and the Japanese syllabary don't use alphabets -- are more powerful than the alphabet. And written language includes numbers and mathematical symbols . . .

[@mansigupta]

An excellent excellent excellent video. I finally get it.

[@abdelrahmangamalmahdy]

I don't understand why he always divides by 2 !!

[@mansigupta]

Because the way the questions are framed, allow for only two possible answers - Yes or No

[@tag_of_frank]

Why is entropy and information given the same symbol H, and why does the information formula given in video 5 of playlist include an "n" for the number of symbols transmitted, but this does not?

[@betbola5209]

como se calcula a entropia de um texto? e o que podemos fazer com isso?

[@poisonpotato1]

What if we used ternary ?

[@zainulabydeen2809]

Can anyone explain ,how the answer become 3/2 in solved example ? Any help will be appreciated

[@youssefdirani]

4:45 Markoff or Markov ?

[@infernocaptures8739]

4:36 **less** information?

[@betoib1504]

!Órale!

[@zkhandwala]

Not to knock this, but I do want to voice an issue that I have with it and every other video I've found on the topic: They always use probabilities that are an integral power of 1/2, which greatly simplifies the explanation, but doesn't generalize well to understanding the majority of real-world scenarios, for which things are not adequately covered by this simplified exposition. I worry, then, that people come away thinking they understand the topic better than they actually do. Of course, I'm open to the perspective of others here...

[@daihung3824]

I agree with your statement. I try to have a go at changing the probabilities, say p(A)= 0.45, p(B)=0.35, p(C)=0.15, p(D)=0.05, then 1/p(D)=20, log base 2 of 20 = approx 4.3, however, number of bounces for D should still remain 3 is it?

[@ignatutka6202]

how come machine's two entropy is more than one? if entropy's maximum is one

[@sammyz1128]

Why can't we ask whether it is AB, for the second distribution, same as the first distribution?

[@sammyz1128]

OHH i get it. If doing so, the average number of questions we ask will be bigger

[@ArthurShelby-y8s]

👏🏻

[@phuocnguyenlethanh3104]

the number of bounces is not equivalent to the number of questions asked