Information entropy | Journey into information theory | Computer Science | Khan Academy

[@vandanachandola322]

I remember my teacher in high school defined entropy as "the degree of randomness". I decided it was an abstract concept that I don't get. Now learning about information entropy in my master's class, I found this video and I'm so glad I did!! Thanks, it's very well explained :)

[@itsRAWRtime007]

good video. i like the way it shows the intuition behind the concept, that is the reason why the concepts actually exists rather than plainly defining it and then showing its properties.

[@youngsublee1102]

couldn't agree more

[@amirizaiah7179]

I guess Im kind of randomly asking but do anybody know of a good site to watch newly released series online?

[@andrewrandall9989]

@Amir Izaiah Flixportal :D

[@amirizaiah7179]

@Andrew Randall Thank you, I signed up and it seems like a nice service :) I appreciate it !!

[@andrewrandall9989]

@Amir Izaiah you are welcome xD

[@XavierGlenKuei]

at 1:24, i would argue the 3rd question (ie, the question on the right of 2nd hierarchy) should be "Is it C?" (or "Is it D?") rather than "Is it B?" (i think this is so because, as the 1st machine answered "No" to the 1st question ["Is it AB?"], it essentially rules-out both A and B, leaving only C(or D) as the possible outcome; hence no role for "B" anymore)

[@marcuschiu8615]

yea, I agree with you. dam, so many mistakes in this video, 1:24 and 3:50. makes me question their reliability... good video though

[@dien2971]

I thought I understood wrong lol. Thank you!

[@muhaymenulislam1942]

But here the probability of D is 25% which Is more than 12.5%, so in the second they ask is it D? .

[@pedrogorilla483]

I have asked several professors in different universities and countries, why we adopted a binary system to process information and they all answered because you can modulate it with electricity, the state on or off. This never satisfied me. Today I finally understand the deeper meaning and the brilliance of binary states in computing and its interfacing with our reality.

[@kempisabel9945]

this video blew my mind away. Thank you! I love these intelligent yet fun videos!

[@mathaha2922]

This is one of the most informative -- and I use that term advisedly -- videos I have ever seen. Thank you!

[@Dhanush-zj7mf]

1:24 You are asking same question twice because you already asked "Is it A or B" in the root if the answer is no that means "it will be either C or D" but you are asking again whether it is B, or not in the sub branch. It should be either "Is it C" or "Is it D".

[@MohdFirdaus-fk6no]

yes, you are correct

[@salrite]

What a Beautiful Explanation!!!

[@MaryMary-ep4hd]

Ingenious interpretation! I applaud!

[@someshsharma6683]

Awesome explanation with a very intuitive example.Thanks a lot...

[@boredomgotmehere]

Makes it all so super clear and easy to follow. Love this.

[@boredomgotmehere]

Just a tiny error at 3:50 - the final calculation shld be 0.25*2.

[@antoinecantin1780]

What a formidable way of visualizing and introducing information entropy. Your contributions are deeply appreciated

[@suryacharan5184]

What a video!!....This is how education should be.

[@YYchen713]

This is such a great way to explain information entropy! Classic!

[@YuriValentines]

This video has explained entropy better than any teacher I've had in my entire life. It makes me so angry to think of all my time wasted in endless lectures, listening to people with no communication skills.

[@twoplustwo5]

Kudos for linking number of bounces -> binary tree -> log. And overall very nice explanation. That's like 3rd explanation for info entropy i liked.

[@raultellegen5512]

Amazing video. Seldom seen a better explanation of anything. Thanks!

[@SAGEmania-q8s]

Thank you so much. It explains the entropy so well.

[@BambiOnIce19]

Perfectly well explained. The best video on information entropy I’ve seen so far

[@youngsublee1102]

Wonderful idea of "bounce" that express the amount of information. It's so exciting.

[@daihung3824]

I have one question: Say p(A)= 0.45, p(B)=0.35, p(C)=0.15, p(D)=0.05, then 1/p(D)=20, log base 2 of 20 = approx 4.3, however, number of bounces should remain 3 is it? Would anyone mind explaining this possible difference? Thanks a lot!

[@potatocoder5090]

Brilliant explanation. So simple yet so profound. Thanks!

[@hrivera4201]

previous lesson: kzbin.info/www/bejne/jaqkpYKnm6iceNk next lesson: kzbin.info/www/bejne/iqnOcmiLjZmen9U

[@phycosmos]

thanks

[@bouzouidjasidahmed1203]

Very comprehensible thank you !! it very helpful

[@tythedev9582]

Yessss I finally got the concept after this video.

[@mostafaomar5441]

Thank you. Explains the intuition behind Entropy very clearly.

[@kartikbansal6439]

Loved the piano bit towards the conclusion!

[@Ewerlopes]

Perfect explanation! :)

[@waylonbarrett3456]

I found a few errors. Am I the only one seeing this?

[@argha-qi5hf]

I can't imagine how someone could ever come up with such abstract ideas.

[@miketor2011]

Great video but is it just me or there is an error on 3:49. The correct calculation for the number of bounces should be 0.5*1+0.125*3+0.125*3+0.25*2 = 1.75 instead the video shows 0.5*1+0.125*3+0.125*3+0.25*4 =2.25? Any thoughts?

[@musicmaker33428]

I was just thinking this. Thank you for pointing it out. I thought maybe I misunderstood something fundamental.

[@rkiyanchuk]

Yep, that's a typo, number of bounces for P(D) = 2.

[@gaofan2856]

The most beautiful explanation of entropy

[@jonathan.gasser]

Wow, what a presentation!

[@vlaaady]

The most intuitive explanation

[@swazza9999]

This should have more likes!

[@nomann5244]

you are truly a genius.

[@daviddeleon292]

Huh??? Why am I finding out that information entropy was a concept. MIND BLOWN!!!

[@tingwen524]

Great video! I totally understood entropy!

[@ГримМорген]

The concept had been presented to me on some online course, but until this video I didn’t really understand it. Thank you!

[@karrde666666]

why can't textbooks or lectures be this easy

[@Hopemkhize-d2i]

Tell me about it😢

[@csaracho2009]

I have an answer for that: The zen pupil asks the master, is the flag moving with the wind? The master replies: neither the flag or the wind move, it is your mind that moves.

[@FrancescoDeToni]

Isn't there a mistake at 3:50? Shouldn't it be 0.25 x 2 instead of 0.25 x 4?

[@philtrem]

+Francesco De Toni yup!

[@sighage]

Yes, it's 2.25 I guess

[@rah2023]

It's indeed a mistake

[@nikhilsrajan]

@@sighage no it's 1.75, just there was a typo. you get 1.75 with 0.25 x 2

[@yudong8820]

Really good one, thanks!

[@russianescapist5262]

I loved this surreal music and real life objects to move in a grey 60s like atmosphere.)

[@edwardjurkowitz1663]

Excellent video. I think one point mistakenly refers to "information" when the author means 'entropy.' Machine 2 requires fewer questions. It produces more information and less entropy. Machine one produces maximum entropy and minimum information. Information is 'negative entropy.'

[@lovenishkavat46]

My problem has been sold 10 years ago you are superman

[@osobliwynick]

Great explanation.

[@shelendrasharma9680]

Best explaination , salute ....

[@Puneethmypadi]

Now I understand decision tree properly

[@mohammadrezamoohebat9407]

It was perfect. thx

[@malevip]

Another way to look at entropy: Measure of distribution of probability in a probability distribution.

[@science_electronique]

قوة في الشرح و وضوح

[@n0MC]

this is wonderful. thank you

[@Chrls5]

Nice!

[@leoyuk-tingcheung3587]

Could anyone help explain why less uncertainty means less information (Machine 2)? Isn't it the other way round? Many thanks.

[@TheOmanzano]

there is less certainty in machine 2 because on "average" there will be less questions...meaning after many trials on average there will be 1.75 questions needed to get right result meaning there is less variety, randomness, chaos in machine 2 due to the fact that "A" will be occurring alot more than other letters

[@hirakmondal6174]

Think of it as a Hollywood film..where a police inspector interrogates a criminal and he must speak truth each and every time. After 175 questions the inspector found out that he knows no more than that, where as when he interrogated other criminal in an adjacent cell he found out that after asking 175 questions he can still answer 25 more.. Now U Tell Me Who Has More Information? . . . . U are welcome!! 8)

[@Exhora]

HIRAK MONDAL That was a great example! Thank you so much!!!

[@salrite]

Less uncertainty means your chances to predict the outcome is more aka predictability increases and hence it refers to less information, means you don't require as many bits (say: to represent outcome) as you would when uncertainty was high or the outcome was pure random. Make Sense?

[@rcgonzalezf]

I also have this question. I guess we need to define information in this context, for the answers and the video itself I think they're referring to data and as I posted below, data is different than information. More questions = more data to get the same information (the output), but I might be missing something.

[@chandragupta2828]

awesome video!

[@hiteshjambhale301]

Hey there is one mistake at timestamp 1:26 ..... the question should be "is it C?" instead of B

[@souvikmajumdar5604]

Thank You

[@jorgeleirana-alcocer5642]

The equation in 3:48 should result in 2.25 not 1.75 (0.5*1)+(0.125*3)+(0.125*3)+(0.25*4)= 2.25 I think it should have been (0.5*1)+(0.125*3)+(0.125*3)+(0.25*2)

[@jayrar6645]

so just to clarify, is the reason the decision tree for machine B is not the same as for A as you ask less questions overall? and how do you ensure that the structure of the decision tree is such that it asks the minimum number of questions?

[@ahmedelsharkawy1474]

just awesome

[@hingaglaiawong7815]

at @3:15 I think there's a typo? The last term should be 0.25*2 instead of 0.25*4 I guess.

[@juanpablovaca-lago5659]

Is there a direct analogy for the second and third law of thermodynamics and the information entropy?

[@AhmedKMoustafa2]

great explanation bro :)

[@mikibellomillo]

note: number of bounces - entropy is maximum when all outcomes are equally likely . when introduce predictability the entropy must go down. thanks for sharing this video! God bless you!🎉

[@ChusKon1]

Beautiful

[@sholi9718]

can someone explain in #bounces=p(a)x1+p(b)x3+p(c)x3+p(d)x2 at 3:44 , how numbers 1,3,3 & 2 came?

[@achrafamrani2730]

x1 is the number of bounces needed to get to point A which is equal to 1 (steps needed for the disc to fall in case A) , x2=x3 is the number to get to points C and B separately and it equals to 3. For x4 it takes 2 bounces to fall in D

[@shepbryan4315]

Why is the number of bounces the log of the outcomes?

[@monicarenas]

In minute 3:51, I guess there is a mistake, for p_D, the value is 2 instead of 4, does not?

[@alexhsia9510]

What do they mean by number of outcomes? Can someone give me an example using the ABCD examples they used?

[@temenoujkafuller4757]

Yes, I asked myself this question and watch it twice. (5:45) Count the number of branches at the bottom The number of final outcomes = 2^(number of bounces) Therefore, the inverse function of exponent is logarithm >>> The number of bounces = = the number of questions = log_2 (number of outcomes)

[@btsandtxtloverstraykidzfan3486]

What are some good books on this topic ?

[@_crispins]

nice!

[@FARHANSUBI]

Can someone please explain why machine 2 is producing less information? Shouldn't it be more information because we're asking fewer questions? Or is it the more number of questions we ask, the more information we receive?

[@kawaikaede2269]

cool

[@sanjayrakshit8797]

Heckin Shanon

[@sanadarkia2724]

can't we just ask one question? is it abc or d ? edit: nevermind, i just figured that 1 bit removes uncertainty of 1/2

[@suliu2933]

Great video! I can follow it but I have trouble understanding the problem statement. Why "the most efficient way is to pose a question which divides the possibility by half"?

[@vandanachandola322]

Too late, but maybe because we're trying to ask the minimum no. of questions (and therefore going with the higher probability first)?

[@ArthurShelby-y8s]

👏🏻

[@dissdad8744]

Good explanation! If I wanted to calculate the entropy with log2, which calculator can do this? Is there an online calculator for this? What would be the best approach?

[@ElectricChaplain]

Hans Franz Too late now, but log2 b = ln b / ln 2 or more generally log 2 b = log base a of n / log base a of 2.

[@gustavomartins007]

Muito Bom

[@최로봇]

if it makes us ask less questions, doesn't it mean it provides more information?

[@assylblog]

Cool beat

[@hirakmondal6174]

Why outcome is 1/p?

[@anirbanmukherjee4577]

Possibility of outcome=1/number of possibility

[@betoib1504]

!Órale!

[@samhe331]

I think the math at 3:52 is wrong.. should be 0.25 x 2 instead of 0.25 x 4 but the result is right 1.75

[@FGNiniSun]

Hello please why does the number of outcomes at a level equal to 1/probability ?

[@youssefdirani]

4:45 Markoff or Markov ?

[@mansigupta]

An excellent excellent excellent video. I finally get it.

[@abdelrahmangamalmahdy]

I don't understand why he always divides by 2 !!

[@mansigupta]

Because the way the questions are framed, allow for only two possible answers - Yes or No

[@YTBxd227]

still confused why #outcome=1/pi

[@CZRaS]

because you need to "build" a binary tree to simulate bounces. E.g. you have probability p=1/2 (50%). From that outcome = 1/1/2 = 2. If you have p=1/8 (12,5%), you get outcome = 8. From which you can get the log2, which is basically the level on which the value is in the binary tree.

[@pablobiedma]

So if I recall correctly, the one with the highest entropy is the least informative one, then the, if a machine generates symbols, and apply the formula for each symbol, which symbol provides the most information? the one with the least amount of bits? how does that make sense, isn't it the one with the highest amount of bits? calculated by p log( 1/p)

[@zkhandwala]

Not to knock this, but I do want to voice an issue that I have with it and every other video I've found on the topic: They always use probabilities that are an integral power of 1/2, which greatly simplifies the explanation, but doesn't generalize well to understanding the majority of real-world scenarios, for which things are not adequately covered by this simplified exposition. I worry, then, that people come away thinking they understand the topic better than they actually do. Of course, I'm open to the perspective of others here...

[@daihung3824]

I agree with your statement. I try to have a go at changing the probabilities, say p(A)= 0.45, p(B)=0.35, p(C)=0.15, p(D)=0.05, then 1/p(D)=20, log base 2 of 20 = approx 4.3, however, number of bounces for D should still remain 3 is it?

[@MNKPrototype]

Did anyone else notice the DEATH NOTE music at 4:40.

[@GiuseppeRomagnuolo]

I was wondering what was that music, I really like it. Do you have any link I found this kzbin.info/www/bejne/nnzJfIyml8Zjmqc following your comment. Is that it? If so can you point me to the right minute? Tnx

[@zainulabydeen2809]

Can anyone explain ,how the answer become 3/2 in solved example ? Any help will be appreciated

[@betbola5209]

como se calcula a entropia de um texto? e o que podemos fazer com isso?

[@infernocaptures8739]

4:36 **less** information?

[@xxxxxx-wq2rd]

is it valid to say less entropy = less effort required?

[@sammyz1128]

Why can't we ask whether it is AB, for the second distribution, same as the first distribution?

[@sammyz1128]

OHH i get it. If doing so, the average number of questions we ask will be bigger

[@mikechristian-vn1le]

Language is a much more powerful invention than the alphabet, and written language -- Chinese and the Japanese syllabary don't use alphabets -- are more powerful than the alphabet. And written language includes numbers and mathematical symbols . . .

Phụ đề Tiếng Việt ở 4:34 sai rồi, máy 2 sản xuất ít thông tin hơn máy 1

[@tag_of_frank]

Why is entropy and information given the same symbol H, and why does the information formula given in video 5 of playlist include an "n" for the number of symbols transmitted, but this does not?

[@Apolytus]

In 3.50 you mistakenly have written 0.25*4 instead of 0.25*2.

[@gregorywinter4132]

身份证原件及复印件三张，每张附件是一张身份证正反面清楚的复印在a4纸同一面，千转夜海粤海街道办中国银行借记卡一张借记卡复印件一份需将借记卡与身份证正反面复印复，复印在a4纸同一面，并手抄卡号，身份证号签名不限制办卡地点中国银行借记卡即可。员工信息登记表，请将附件一中国员工信息登记表含样地打印后，参照样例手工填写一份表中涉及日期内容的，必须确认到日，请务必粘贴登记照。