Lecture 1: Introduction to Information Theory

[@prajwolgyawali6770]

Fundamental Problem: Reliable communication over unreliable channel Binary Symmetric Channel: 9:35 Disk Drive Problem: 11:30 Redundancy: 23:00 Repetition Code: 24:35 Decoding is inference Inverse Probability: 31:33 Forward Probability: 40:30 Hamming Code: 47:10 (kzbin.info/www/bejne/m5rbdHqnmdF6rrM%29) Capacity of channel: 58:10 Shannon Noisy Channel Coding Theorem: 58:45 The Weighing Problem 1:00:50

[@wampwamp1458]

thank you :)

[@fjs1111]

Unreliable channel = Unreliable information

[@mohamedrabie4663]

RIP prof David, you were, and still a great inspiration to us

[@eddiehazel1259]

ah man sad to hear 😔

[@iwtwb8]

I was saddened to read that David MacKay passed away earlier this year.

[@JTMoustache]

what a great teacher.. he will live on :)

[@wunanzeng7051]

He was a very great teacher! Very Articulate!

[@yousify]

I was shocked when I read your comment. I'm following his lecture and his book on this course, it is a sad news.

[@jimmylovesyouall]

what a great teacher.. he will live on

[@bayesianlee6447]

RIP For great teacher of human beings. His passion and endeavor for tech would remain for descendant

[@lesliefontenelle7224]

I am not involved in information technology but this lecturer is making a difficult subject like information theory look so easy. You really must see this...

[@shellingf]

kinda boring though

[@JerryFrenchJr]

How am I just now discovering this lecture series??! This is awesome!

[@JakobFoerster]

better late than never! glad you are finding it useful

[@anantk2675]

i got it now bro, i am latter than ya ; )

[@siweiliu9925]

@@anantk2675 I'm later than you, hhhh

[@trueDeen911]

@@siweiliu9925 i am later than you

[@the_anuragsrivastava]

One of the best lecture of " information theory and coding " I have ever seen....love from India 🇮🇳🇮🇳🇮🇳🇮🇳

[@eul3rr]

As a math student who is interested in information theory and neural networks, i discovered this gem of a lecture series when i was looking for videos to fall asleep to! In fact i've finished the lectures when i was sleeping :D Now I decided to start it properly and just finished watching this lecture and taking notes. I would love to send David a mail when i finish the course. Thanks for leaving this behind my man, rest in peace.

[@IrfanAli-jl7vb]

Thank you, Thank you, Thank you for sharing these excellent video lectures. Dr Mackay is amazing at teaching complicated topics. These lectures are great supplement to his excellent book on information theory which has so many excellent plots and graphs that enables one to visualize information theory. Information theory comes alive in pictures. Thank you for sharing these.

[@linlinzhao9085]

Dr. Mackay is a great explainer. Anyone interested in machine learning and Bayesian statistics can also read his doctoral thesis.

[@dragonfly3139]

Thank you for these great lectures your memories will live on ... RIP

[@baganatube]

I don't mind the slowness. With some decoding, my brain is receiving a cleaner signal with that extra redundancy.

[@sadimanesadimane6746]

Bagana also does he have to write EVERYTHING

[@AvindraGoolcharan]

This may be the longest blackboard I've ever seen

[@hyperduality2838]

Repetition (redundancy) is dual to variation -- music. Certainty is dual to uncertainty -- the Heisenberg certainty/uncertainty principle. Syntropy (prediction) is dual to increasing entropy -- the 4th law of thermodynamics. Randomness (entropy) is dual to order (predictability) -- "Always two there are" -- Yoda. Teleological physics (syntropy) is dual to non teleological physics. Duality: two sides of the same coin.

[@mahmoudehab8627]

Absolutely one of the gems that are just there on the internet but hidden by some bullshit courses. We should keep digging to find more of these!

[@a_user_from_earth]

what an amazing lecture and great teacher. May you rest in peace. You will not be forgotten.

[@fireflystar5333]

This is a great lecture. The design of case problem is really helpful here. Thanks for the lecture.

[@AlexandriaRohn]

00:30 Information Theory invented by Claude Shannon to solve communication problems. Fundamental problem: Reliable communication over an unreliable channel. e.g. [Voice->(Air)->Ear], [Antenna->(Vacuum)->Mars Rover], [Self->(magnetized film)->Later Self] 04:00 Received signal is approximately equal to the transmitted signal because of added noise. 05:45 What solutions are there for having received and transmitted signal be the same? Either physical solutions or system solutions. 07:30 Source message -> [Encoder] -> Coded transmission -> [Channel (noise introduced)] -> Received message -> [Decoder] -> Best guess at original source 08:45 Encoder is some system that adds redundancy. Decoder makes use of this known system to try to infer both the source message and n.

[@TheNiro87]

This is great, thank you! The lecture is far more entertaining than just reading the book.

[@HeMan-tm8wl]

whihc book is it?

[@george5120]

So nice to watch a video like this that does not have music.

[@monazy11]

It was amazing, Finally I learned Shannon noisy channel theorem: there is exist an encoding and decoding system that could reach to the capacity of channel. so error correcting and detecting course is about to learn these encoding and decoding system. wow Amazing lots of thanks to the teacher

[@jedrekwrzosek6918]

I looove the lectures! Thank you for the upload!

[@oscarbergqvist4992]

Thank you for a great lecture, looking forward to follow the rest and study the book!

[@kikirizki4318]

Thankyou very much prof David MacKay, Your lecture helps me understand information theory. Btw I live in the country where toilet is not so common

[@the_anuragsrivastava]

You are from where??

[@leduran]

These lectures are great. Thanks for sharing.

[@abhishekpal5871]

this really helps. I really wanted to learn information theory. this video series is really easy to understand and awesome.

[@Handelsbilanzdefizit]

I would use SUDOKUS for communication. Only few numbers have to be transmitted correctly, and the other numbers/information can be restored by the decoder :-D

[@Hastur876]

The problem is that only the sudoku numbers that you transmit count as information: the rest of the numbers are constrained by having to follow the rules of Sudoku, and can't be any numbers you want, thus they can't be information. So you're still having to get 100% data transmission.

[@palfers1]

Genius camera work

[@rafaelespericueta734]

Indeed so. It's so frustrating and irritating when the camera focuses on the lecturer when you really want to look at the slide with plots and equations.

[@oyindaowoeye467]

I wonder if he was making a joke when he said "Mr. Binomial" or if he actually meant to say "Bernoulli".

[@vik24oct1991]

he is being sarcastic i think , he finds it silly that distributions are named after people who discovered them while it would have been more logical if they were named after what they represent (which binomial distribution does).

[@PierrePark]

@@vik24oct1991 he doesn't find it silly, he is himself just being silly, because so many things are named after someone he's joking and extending that to binomial

[@Omar-th5vv]

I'm new to information theory. At 4:24 why is the received signal is "approximately" and not "equal" to the transmitted signal + noise? like what else is there other than the transmitted signal and the noise? Thanks for sharing such helpful lectures.

[@caribbeansimmer7894]

It gets extremely difficult trying to model everything that affects the signal, but it's relatively easy to add noise to the signal, but not just noise but an assumption that the noise is additive (hence the plus sign), white, and gaussian( normally distributed). As you would know from statistics, a normal distribution has nice properties for estimation. This stuff gets really complex but I guess you get the idea. Note, we can make other assumptions about the characteristics of noise

[@nishantarya98]

Received = Transmitted + Noise makes a *lot* of simplifying assumptions! In the real channel, the noise might be multiplied, not added. The transmitted signal might go through weird transformations, like you might receive log(Transmitted), or it might be modeled as a filter which changes different parts of your Transmitted signal in different ways!

[@reyazali2768]

awesome example of teaching style

[@ridwanwase7444]

the link of getting free book is not working,can anybody tell from where i can get that free book? Thanks in advance

[@gadepalliabhilash7575]

Could anyone explain how is the flip pobability is 10^-13 and and 1% failure is 10^-15. Video reference is at 22:10

[@RippleAnt]

Ah! thanks, was looking for a good series... this is just the one. A great cliffhanger at the end to be precise... ^_^

[@cupteaUG]

Any idea about the final puzzle? My answer is 3. :-)

[@papatyavanroode2329]

3/8 3/8 2/8 cake and siblings That's the most equal way to share In the 3rd cake they are most close to equal 4/82/8 2/8 is far behind equal

[@christopheguitton7523]

J'ai compris pourquoi répéter 3 fois la même information de suite à mes enfants n' était pas forcément efficace :-)

[@deepkushagra]

at 46:50, what is rate? i guess it is (1/no of repetitions) but what does it mean in layman terms

[@bambitsunami4165]

for anyone who sees this in the future, i believe “rate” is the rate of information transfer. so if i send the message “hello” maybe it takes me 1 second, but if i send repetitions to protect against errors, eg “hello hello hello”, it takes me 3 times as long because i had to send the message 3 times. so the rate is 1/3 of what it was before. its more “reliable”, but its also slower.

[@spring74light]

Cheering, in these days of advanced educational gizmolgy, to see Professor MacKay making extensive and effective use of a stick of chalk and a polychromatically absorbent surfaced board. {Tyneside, England.]

[@vedantjhawar7553]

Hi, I just wanted to ask what was meant when at 24:20 when he says that "1 is the same as 5, and if there was a 4, there would be a 0."

[@MN-sc9qs]

1 and 5 are both ofd so assigned 1, and 4 is even so assigned 0.

[@vedantjhawar7553]

@@MN-sc9qs Thanks.

[@qeithwreid7745]

This is fantastic I love it.

[@arthurk7270]

I'm a bit confused. If we're trying to estimate the mean of the Binomial distribution, wouldn't we use the sigma/sqroot(n) formula for the +- bound? In other words, the mean +- the standard deviation of the estimator: 1000 +- 30/sqroot(10000) = 1000 +- 0.3? My statistics is a bit rusty.

[@JakobFoerster]

Arthur K thanks for the comment. Which part of the lecture are you referring to?

[@arthurk7270]

Hi. I was referring to the discussion at 14:00.

[@JakobFoerster]

Arthur K I believe you are confusing the standard deviation of the sample mean rate with the standard deviation of the mean total count. The standard deviation of the mean rate does indeed drop as 1 / (N)^0.5, while the standard deviation of the total count increases by (N)^0.5. Since we care about the total number of bits flipped, it's about the total count rather than the rate. You can see that multiplying the standard deviation of the rate, 1 / (N)^0.5, with the total number, N, results in a standard deviation of the total ~(N)^0.5. Please let me know if that clarifies.

[@derekcrone4679]

is dis how u mak a plane

[@turkiym2]

Was Mr. Bionomial a joke that fell on dead ears or was it a genuine confusion with Bernoulli?

[@MarkChimes]

+omniflection I laughed when I heard it. I take it was just a very dry joke.

[@yltfy]

+Mark Chimes Hahah. Me too. This is when I clicked the pause and check the comments...

+Mark Chimes Well hello Mr. Chimes :)

[@AtharvSingh-vj1kp]

I'm a bit curious about when these Lectures were recorded. Was it 2003??

[@terrythibodeau9265]

I am curious to know how Shannon would have interpretted the internet as part of his theory... noise perhaps

[@aritraroygosthipaty3662]

38:19 I am unsure of the 1/2 in the denominator According to my calculations P(r=011) = (1-f)f^2+f(1-f)^2 This is done by the sum rule. The numerator should have 1/2 due to the P(s=1) term My final answer is P(s=1|r=001) = (1-f)/2 Could anybody help me with my concerns?

[@dlisetteb]

When you add up the probability of r given s=1 and given s=0, you must include the probability of each of those events. It results as P(r=011) = P(r=011, s=0) + P(r=011, s=1) P(r=011) = P(r=011/s=0) * P(s=0) + P(r=011/s=1) * P(s=1) P(r=011) = (1-f)f^2 * 1/2 + f(1-f)^2 * 1/2

[@dr.alaaal-ibadi8644]

I like this channel. I'm already teaching this topic for student in Iraq. In Arabic.

[@driyagon]

can someone explain how to solve the homework problems?

[@ozgeozcelik8921]

awesome! thanks for sharing

[@sahhaf1234]

towards 56:00 he uses the term "bit error" and "block error" but doesnt define them properly..

[@keshavmittal1077]

hey is there any pre-requisite of it

[@ciceroaraujo5183]

Thank you professor

[@artmaknev3738]

After listening to this lecture, my IQ went up 20 points!

[@Hussain1Salman]

Thanks for the lectures. I am just curious about what happened to lecture 2. It says it was deleted.

[@JakobFoerster]

Hi Hussain, thanks for catching this. I have reached out to youtube support to find out. Hopefully will be resolved soon.

[@JakobFoerster]

Hi Hussain, Apparently there was a bug in the youtube system and they deleted it by accident. The video is back online now.

[@mustafabagasrawala7790]

I'm fairly new to this. What is a "flip" ?

[@aritraroygosthipaty3662]

a flip is to change the bit from a 1 to 0 or a 0 to a 1.

[@deeplearningpartnership]

Thanks for these.

[@pauldacus4590]

You know he's a jokster cuz at 1:00:13 the slide says his textbook weighs "~35 lbs".

[@rarulis]

XD, that's the price. 35 british pounds.

[@carloslopez7204]

What are the requirements to understand this lecture?

[@AndreyAverkiev]

As it says the only piece of Mathematics is binomial distribution 16:08

[@ncckdr]

very great theory

[@motikumar1442]

Helpful lecture

[@stevealexander6425]

Good lecture except for the highly distracting camera work.

[@Rockyzach88]

"People need 20 GB drives nowadays" Laughs in Call of Duty

[@Abu_khalid1]

Lol

[@GOODBOY-vt1cf]

9:50

[@izzyr9590]

im learning this at school... yet I'am here watching a lecture on KZbin ... I dont know why ... I should pay attention in my class I guess.

[@javatoday5002]

you're definitely exercising inference

[@minglee5164]

I have read the extraordinary book.

[@GSSIMON1]

the fact is the received signal is not identical to sent signal due to corruption and distortion in the signal, in a process , so how much of the the original signal is received ,what would be the measurement in what unit ,,,,,thats why i do drugs ,and dont give a damm !

[@javatoday5002]

measurement would be in bits and I am not joking

[@brandnatkinson5981]

K

[@MDAZHAR100]

Binomial is more related to probability topics. Bernoulli is about hydraulics.

[@JakobFoerster]

en.wikipedia.org/wiki/Bernoulli_distribution

[@dharmendrakamble6282]

Information theory

[@afarro]

I was able to achieve f=0.1 for this video with x1.5 speed ...

[@pandasworld4168]

hehe if you re like me, then you have that exam in one week

[@SphereofTime]

2:00

[@circlesinthenight3141]

rip david

[@alyssag8099]

8:22

[@tonewreck1]

Profesor McKay is certainly extremely competent in the subject but this really is the most un-intuitive way of introducing information theory. We are given the answer right from the start and work our way backwards to see that it is an effective system, instead of trying to understand the problem and find the adequate solution. We are never trying to understand the nature of the problem but instead made to test the effectiveness of the solution. Typical of classical academic philosophy. Let's make knowledge as boring and abstruse as possible so the riff raff is kept out of our little club!

[@vedantjhawar7553]

Do you know of any sources to check out that might teach the content in the method you are talking about? Thank you.

[@tonewreck1]

@@vedantjhawar7553 try this ...kzbin.info/www/bejne/hqXQq5mmi5t2eqc

[@vedantjhawar7553]

@@tonewreck1 Thank you. I took a look at it and felt that it described the act of measuring information very thoroughly. Are there any videos you recommend for learning about system solutions for transmission errors?

[@tonewreck1]

@@vedantjhawar7553 I am glad you found it helpful. I suggest to see the other videos in this series. Once you get a feel for what information theory is really about, you can go back to MacKay for error correction and all the nitty gritty, he does the details very effectively.

[@renduvo]

@tonewreck1 For some reason I'm unable to see the reply in which you've suggested the source that @Vedant Jhawar has requested. Could you please share the source again?

[@dharmendrakamble6282]

🎉

[@PseudoAccurate]

This is painfully slow. I'm sure the information is fantastic but you have to watch him write everything he says on the chalkboard. When I got to him telling the class to discuss how much 10% of 10,000 is I couldn't take it anymore. Can anyone suggest a similar lecture that moves more quickly?

[@Meeran]

use the youtube speed feature, set it to 2x, done

[@PseudoAccurate]

Lol, nice, good idea.

[@unorthodoxresident7532]

Lectors goal ultimatly is to share information and for audience to absorb as much as possible. There are different ways how people can do that and for each it's different. Some people are better at listenening others better absorb visual representation (either looking at chalkboard or writing notes themselves) :)

[@PseudoAccurate]

That's completely understandable - that's what it was like when I went to school. It's just that most lectors now give you the notes themselves so they don't have to take the time to write much and can spend the lecture explaining the material.

[@MlokKarel]

Now the question wasn't aimed at the 10% of 10k but, rather at the +- part, i.e. std.dev, IMO. Did you get that correctly as well?

[@박재우학부재학전기전]

channel immigration

[@seweetgirlnay]

Where in the fock am I?

[@jabbatheplutocrat1074]

There is no information here and when will it be realized?Never!!!

[@GOODBOY-vt1cf]

15:52

[@GOODBOY-vt1cf]

13:24

[@GOODBOY-vt1cf]

32:13

[@GOODBOY-vt1cf]

34:43