How an HTTP Request Gets Served - In Great Detail

[@fairalbion]

Each of these steps has a huge number of background operations going on to make it happen. I'm a retired DNS guy, and that process alone could occupy one of Dave's videos.

[@johntoe6127]

Computing is a perfect example of the Dunning-Kruger effect. The average person completely under estimates the complexity behind things that have become common.

[@brodie3088]

@@johntoe6127 definitely. I know a guy who is like a 10x dev wizard type dude. His motto is "all I know is that I know nothing". Personally I gave up on being a web dev about a year and a half after I became competent enough to do some cool stuff. The realization that just because you made it work doesn't mean you didn't build it in a way that somebody smarter than you would consider stupid is a tough one to stomach.

[@Bunny99s]

@@johntoe6127 The Dunning-Kruger effect has nothing to do with underestimating the complexity but only with over estimating the own knowledge. Those are two very different things. Explaining how DNS roughly works is perfectly fine, even when you don't know every little detail. Telling a lay person that it's essentially a telepone book for the internet is enough to get an idea what it does or what it's there for. As long as you don't claim you know everything about DNS, there's nothing wrong with only having partial knowledge. I for myself have written my own DNS server from scratch in C# which currently runs on my raspberry pi with mono :P Though I've only implemented what I needed / wanted and it was more a proof-of-concept / learning project. I've read through most of the RFCs. I didn't implement any DNSSEC stuff. Though my server supports both, UDP and TCP requests. It does not act as a resolver for other zones than my own as I only serve my own records. Currently every request and response is written as a hex dump to the console :) Runs stable for over a year non-stop now. So I don't consider myself as a DNS expert, by no means, but I have a fairly good understanding of the basic protocol, even though I don't implement every extension (DoT, DoQ, DoH, TSIG, ...). Though it's funny to see how many requests come in to check the "bind" version as most would assume that a DNS server is some variant of bind. Berkeley has set quite a few standards over the time. A lot of people who "work" with DNS don't really understand the fine details of the protocol either. Most work only on the configuration level. DNS has a fascinating "name compression" system for requests as each part of a domain name is split into it's individual labels and when multiple things are requested or returned, the protocol can use back references to labels that already existed. Those can even be recursive. Quite fun to implement a parser and encoder for that. I often miss many concept of the "old days" where they tried to spare every bit without the need to make it overly complicated. Modern day development is more about: "we do have the CPU power, memory and bandwidth, so f-it". The "old" protocols were all designed to be easily parsable by hardware implementations. Most modern stuff is just a mess with tons of overhead.

[@mikko-penttieinarieronen]

@@johntoe6127 I like to adapt the Coastline paradox into computing and technology or any other process-related subjects. The more you dig into each component or step, the more questions you'll end up having :D And often you forget some of the old stuff you have learned, so it's impossible for one person to handle/understand everything.

[@jcasetnl]

Do you sometimes still volunteer to translate names to IP addresses in your spare time?

[@RichardCollette42]

Hey, Dave. As someone who has been teaching this stuff for nearly 30 years, I can say without hesitation that you are excellent at it. I found myself saying out loud, "Dave, you forgot to mention ..." only for you to say it five seconds later. I recommend your channel to my students. On a side note, how about a follow-up video on TCP vs. UDP and, more importantly, QUIC? I'm amazed at how little info there is on QUIC in the traditional educational resources I (must) use. Perhaps more Layer 3 info on how IP works as well.

[@Richardincancale]

Yes - maybe another follow up video on how a KZbin video of Dave gets to you - with distributed network of YT servers and use of QUIC to overcome the round trip time that was a problem video over TCP - especially over 4G and / or satellite - my domain!

[@PeterBachman]

QUIC makes the certificate key exchange simpler than TCP, fewer round trips, better suited for streaming

[@fieryweasel]

He forgot the step of hurriedly closing the tab when your boss walks over.

[@michaeltyniec7010]

Tetris had that feature - a certain keystroke would switch it to Lotus 1-2-3 (DOS mode) if your boss was coming over.

[@robertbelanger2834]

@@michaeltyniec7010 That was the BOSS key, the first one to learn when playing on company computers...

[@daniellundqvist5012]

I did the Tetris thing in reverse, I incorporated a sourcecode version of tetris into the system that would start with the Break key, and stop with the Escape key😂

[@FlameStrykeShadowDark]

.....or when the tab inexplicably begins blaring music.....

[@batteries76]

Your boss lives in your house?!

[@jholloway77]

I built myself a simple HTTP 1.0 server in C as a little project. Learned so much about how the protocols work, how sockets work, and how to build something according to documentation & specs. One of my favorite projects that I've done for fun.

[@julianocardarelli5106]

Any specific library to handle sockets and open tcp port on server side?

[@jholloway77]

@@julianocardarelli5106 @julianocardarelli5106 It was a very, very simple server just using the standard POSIX libraries built into FreeBSD. Nothing external required. I'd share a link to the github, but KZbin keeps filtering out comments with links

[@brianjuergensmeyer8809]

Nothing teaches you how something works until you try to implement it to spec.

[@jholloway77]

@julianocardarelli5106 it was a basic application and just use the libraries found in any POSIX system. I made mine on FreeBSD

[@fakedevdutt]

TF nerd! nice work tho

[@yoman912567]

These are my favorite types of videos! You do such a good job and explain a level deeper than so many other channels. I learn more in these type of vids than years of KZbin IT osmosis.

[@ionamygdalon2263]

Nobody can explain these things better than Dave! I often send these videos to people I know who are struggling with these concepts and they all tell me how much it helped!! WELL DONE DAVE 👍👍👍

[@zoltannemeth8864]

Back before the internet existed, this process was all done manually, I used to hate the three-way handshakes, but I did enjoy the cookies! Seriously, another great video Dave! Thank you for sharing your experience and knowledge with us!

[@ArsNoble]

I first held a phone tat used to dial ..... I always wondered why internet was like dialing a phone number .

[@Sir_Uncle_Ned]

As an IT networking guy I can confirm that this does paint the general picture of what happens when you request a web page; But every element of that process is a seperate rabbit hole of processes, servers, math, and ungodly gallons of caffeine to keep it all running. I can see why you didn’t go down those rabbit holes for this basic overview

[@alexanderboehm7473]

I thought he explained things pretty thoroughly - of course, there will always be more details.

[@DetectiveConan990v3]

what do you do as an IT networking guy

[@Sir_Uncle_Ned]

@@DetectiveConan990v3 mainly look after switches and routers. If I do my job well the people I work with don’t notice.

[@ArsNoble]

So ya guysknow i watch cat videos 😱

[@grottyboots]

Love these deep dives into technical subjects. Your explanations are so damned good! I recall reading Comer's "Internetworking with TCP/IP Volume One" and being blown away at the brilliance of TCP/IP. Keep up the great videos, sir!

[@nomadic_shadow]

If you started an “in great detail “ series dave, I don’t think anyone would be mad.

[@zoltannemeth8864]

This would be awesome!

[@steamer1]

I think you'll find I second this motion.

[@alex-m-jones]

Dave I think a SYN is an empty packet with the SYN flag set.

[@turkishmaid]

an "in great detail" series for every one of the steps, please... Maybe 2 hrs per step as a starter...

[@dailydose7772]

Subscribed after just 2 minutes of your awesomeness. Thanks for your time and knowledge

[@floriansalihovic3697]

Double you tee eff. That’s as entertaining as educational - with a delivery worth an audiobook. Well done - subscribed after 12 seconds. Great work

[@Atsumari]

I am a web tech and manage servers for websites I am glad you went over this. There are so many people who think this is magical and that the internet just works; no it doesn't there is a lot fo work, systems, deamons and configuration files involved. The internet is fascinating and humbling if you understand all the work that goes into making it work and keeping it up.

[@subbastionbastion2167]

The protocols doesn't explain how it works at all honestly. Like how the ip address knows where it is in the world and how a device connects to it. Dns to ip ok but what does the ip do

[@alexanderboehm7473]

@@subbastionbastion2167 Off the top of my head - Internet Protocol is assigned by a group called IANA. They help manage this because public IP addresses must be unique - unlink private/internal IP addresses. Your ISP is essentially responsible for routing your traffic through different hops (sometimes managed by the ISP, or sometimes when allowed, through other Autonomous Systems). These hops communicate with each other using BGP - an exterior routing protocol that allows for dynamic IP address routing over the internet.

[@dooza]

@@subbastionbastion2167 An IP address does not know where it is. Routers and ISPs figure out where the IP packets go. Each ISP has bought a set of IP addresses that they assign their customers. When your router has sent the packet to the "Internet highway" (as it saw it was not LAN applicable), the ISPs have their own routing table to know where to send it. "Hmm, this looks like an IP from an Swedish ISP I know. I'm going to send it that way".

[@jacobm1190]

@@subbastionbastion2167 The packet is sent through many routers as it makes its way to its destination. First to your home router, then to a router belonging to your ISP, and so on. Each router looks at a huge table in its memory to determine in which direction (that is, to which router) it should send the packet next. These tables are updated by the Border Gateway Protocol (BGP) which operates globally. That's as far as I know because I am only a CCNA.

[@komraa]

​@@subbastionbastion2167well every single key word there is a large field in itself. You can find books on these pretty easily

[@christophermcdonald8732]

Awesome explanation Dave! Fascinating to see the details, keep it coming!

[@johnneyland3334]

You’re the best Dave !! Love your videos and knowledge content !

[@javabeanz8549]

Great explanation Dave! For how complex the process is, I think that you broke it down quite well, to be digestible for the general public.

[@JV-bw2fw]

Excellent video sir; thank you for all the effort, time and dedication that goes into each of the videos you make. Please trust me when I say, there are some of us who truly value and appreciate it.

[@MrSunDevil23]

Where were you in the early 90’s when I was just learning networking and getting my MCSE… oh yeah, you were working on the OS I was learning!!! Great video as always. While this is mostly old school info for an old engineer like myself it is a great refresher and will be very VERY helpful to newbies who want to learn.

[@toby9999]

It's all new to me... kind of. I've been struggling to understand this stuff for years. I've been a C++ Windows application developer for 28 years, but anything rated to networks does my brain in.

[@jimsubtle886]

Awesome video yet again Dave ! I was very pleased with part 1.. I cannot wait to see part 2 !!

[@FlightEagle]

The best explanation I've seen, thank you. Included correct and appropriately complete detail of the process.

[@surabhsawant]

such a great video! had a great time watching it probably because I had enough context of the web flow but still it had me hooked up till the end in one go.

[@bw6378]

Apache and nginx but no IIS? Oh, the irony is not lost on me!!! LOL. Thanks for the video!

[@anon_y_mousse]

He's just admitting what we all know, IIS sucks.

[@abhidube2928]

😂😂

[@electrictrojan6719]

No one cares about windows, not unless paid to care.

[@kevinL5425]

You really appreciate how much work the computer and network is doing when implementing it with RFC 1149. Personally I think that implementation is for the birds but some people have used it.

[@larryjanus596]

I'd tell you a UDP Joke, but you might not get it.

[@DavesGarage]

You could just resend it, though.

[@Sobioytccc]

​@@DavesGaragelet TCP handle it for you 😅

[@XORBob]

I will be using this video for my Web Dev class next semester. Thanks Dave!

[@markmonroe7330]

Excellent presentation. Thank you.

[@larrycleeton]

It frightens me that I have intimate knowledge of everything you described. Kudos on leaving out the really boring details. One of my minor claims to fame as a former employee is I personally removed the gopher protocol from Internet Explorer that became version 7. 😎

[@sdrc92126]

In the olden days, it took decades to earn these things

[@wxfreak]

Thanks Dave

[@quambafu]

Great videos Dave! My 16 year old enjoyed the Hello World video and learned quite a bit.

[@davidjsutherland]

Great content Dave. Missed the Friendly Giant ending though.

[@TonyGingrich]

I would be interested if Mr. Plumer has any sentiments to offer regarding VB Script and ActiveX?

[@NathanaelNewton]

This is my new favorite series on all of KZbin, please make more😮❤🎉

[@FootsieAU]

Great vid Dave, but what about SNI? That usually exposes the hostname of the request before the TLS handshake is complete.

[@LotharLindinger]

Excatly what I wanted to add. I can't believe Dave doesn't know about it but maybe his mind is already TLS 1.3 only.

[@geepeezee5030]

Agreed. We need more sites implementing ECH. Otherwise, does no good to obscure dns with DoH/Q/T/etc.

[@davidjowett8195]

Once again informative and enjoyable. Thank you.

[@Noam-Bahar]

This is the clearest explanation I've heard so far

[@MikeD-tf7dk]

As someone that’s been in this world for almost 30 years, you remind me of why it’s important to be open to learning new stuff about subjects, you already have a good understanding of. extra bonus geek question as someone that’s been in this world for almost 30 years. You remind me of why it’s always helpful to be open to stuff you may not know. Extra bonus geek question tho. I didn’t hear you make a more detailed explanation as to what a “static” page is. I’m wondering if you’re so used to it that it seems self-explanatory. The only reason I realized it was I was about 3/4 of the way through when you said it again without explaining what it was.

[@DavidWonn]

We too easily take for granted how quickly the overall process happens these days. But in the dial-up days and much slower processors, you could almost witness several of the steps happening in real-time if you knew what to monitor.

[@sueterra3911]

GOT IT! Gotta run; my landline is ringing.

[@mcwolfbeast]

03:00 And yet, Google is pushing using UDP for QUIC/HTTP3 instead of TCP. I'm agreeing with you here, I just wonder how a giant like that can get it so wrong.

[@thehauntedstream7206]

Incredible brother thank you

[@IAmUsingAndroid]

Thanks dave for such a detailed explanation.

[@ThunderChasers]

Love these videos. You'll have that gold play button in no time!

[@flowerofash4439]

so dense of information in one video, incredible

[@gerakore8948]

back in the day i used to make my own web servers that added background functionality before web 2.0 was a thing. also used this to circumvent cell phone web access limitations.

[@manzoor5554]

Exceed Dave please keep posting more detailed Deep Dives ..😊😊😊❤❤

[@411sev]

Well explained. Thank you very much.

[@fritzmusic]

I have a static IP address so this is not relevant to me. Yes, I'm being facetious. Love your informative videos Dave and always watch and upvote and comment.

[@DavesGarage]

I have 5 gigabit fiber that I paid to have installed and they STILL won't give me a single static IP :-(

[@toby9999]

What difference does a static IP make within the context of this video? I'm clueless 😊

[@pete3897]

@@DavesGarage RFC1918, brother- make you own! :)

[@kevinL5425]

My problem with HTTP servers is I try to get them to serve me coffee with my cookies, but I keep getting a 418 status code back. Used to happen a lot with Russian web sites. When I finally do find a coffee pot the picture usually shows it is empty (sigh).

[@billj5645]

And it's amazing that for many websites that are responsive all of this can happen in under 1 second.

[@JT-kz7kq]

Excellent video as always. What about a follow-up video as to how data actually gets moved through networks and the internet (mac addresses, subnets, gateways etc)?

[@DragonRides75]

another great video Dave thanks for this video

[@LivvieLynn]

Browser address bar is just one big manual GET request with some rendering. Outside of browser features for searches, local resources, or code etc.

[@scratchanitch]

Hey Dave, love these full end-to-end discussions. When you were working at Microsoft did you ever have lunch at Crossroads? If so, we probably crossed paths 🙂. More importantly - an idea for a video is for a deep dive on memory management in Windows and how it compares to Apple (and their unified memory architecture).

[@bramnet]

This is a question I’ve had for a long time. What prevents 3rd parties from listening for the SSL setup commands which would allow them to know the encryption (I think)?

[@schulmastery]

Love your content! Always wondered though, when you say " 'mostly' in it for the subs and likes", what's the rest??

[@DavesGarage]

Well, I do list a book for sale. Before the book, I'd say "I'm just in this for..." but figure since I have that other incentive, I should allow for it!

[@kimtyson9101]

Very nicely done!

[@breakoutgaffe4027]

Super nice explanation 🔥 🔥

[@scottstefanatz]

Good video, only you you didn’t mention the network socket layer that breaks the data into sized packets. Each packet contains header info about origin, destination, last network component to handle it, how many hops it has taken and time to live, and sequencing info to know how to put the data back together.

[@notreallyme425]

I just watched an interesting video from Computerphile how difficult it is to end the connection. It turns out it’s the same as the two generals problem, how do you acknowledge that you’ve disconnected?

[@shantanusapru]

Anther great video, Dave!!

[@AngeloMartino-z8f]

5:15 Actually when you type a website's name in your browser, your device sends a DNS request to translate the domain name into an IP address and your ISP often handles these DNS requests, so they can log the domains you're accessing even if you're visiting HTTPS websites

[@chuckistheman2000]

I was asked this question during an interview. Pretty much answered the same way

[@MattLaubhan12]

Dave: Any thoughts on NIST SSDF (Secure Software Development Framework)? Many tech folks working for companies that wish to ever do business with the US government or the EU in general moving forwards are now held to pretty extreme security standards. Sounds good on the surface...but that comes with challenges... No security == Optimal Performance. Max security == No product. Somewhere in between is a balance.

[@tubeDude48]

Well done Dave. Here's a thought. How about covering tags. I have quite a few at the top of my web page.

[@brodie3088]

This goes further than the request, because HTTP request is a separate concept from the browser rendering the HTML (although I guess you could say the concept's are closely tied due to the rendering process usually resulting in a waterfall of additional HTTP requests). Not that I am complaining though! I really liked the video

[@youmasajp]

Before name resolution, proxy detection is performed. In many cases, computers behind a proxy cannot resolve external names.

[@siwiecministro1994]

There is lot of talk about telemetry/spying by Windows, but my impression is that using firewall (like simplewall or Net Peeker or even frontend to Windows Firewall called Windows Firewall Control) basically we can "disable" all of it. Is it correct impression or has Microsoft this "covered" and can establish connections even against firewall rules? Perhaps, you could make a video about Windows Filtering Platform or how "Firewalling" works in Windows. :)

[@rajakvk2837]

where will you find browser's cached IP and domain name mapping ?

[@petervandebeek5980]

Happy to see you included the Host file, but did not put to much attention into it. If people are going to play with it and forget, it can generate a lot of t-shooting.

[@professor-viewsalot]

Is that all it does, so why does it take so long? The bit I don't understand is how Dave gets to the end without having to add an ammusing sarcastic smile. Great video presentation Dave.

[@davidadams421]

Fantastic! A great follow-up video would be explaining website performance e.g. Google Pagespeed Insights, how DOM complexity, resource sizes e.g. images, largest contentful paint, javascript blocking etc. effect the performance of the website.

[@envoycdx]

Hi Dave, nicely put together! will be helpful for some junior techs just getting into things (and possibly users..) sidenote - is there anywhere you can get your book other than amazon? Thanks.😎

[@MagicMan123ification]

am not ready for this , its major

[@jemdeweare6432]

Interesting dave , thank you

[@LM-ek2hb]

Hmmm..? Step 8, Sending the HTTP response has always been a mystery to me. A large response is broken up into packets. Depending on the server and number of packets, some may take different routes (hops) than others. How do they all get to the their destination at the proper time or assembled properly before render time?

[@byronwatkins2565]

I learned a great deal; however, I also noticed that sound was not considered and that scroll bar windowing was omitted.

[@IsrealSmith32465]

This has nothing to do with the video, but can someone please tell me what that flame spiral thing in the background is? It looks so cool and I want one.

[@David_Best]

It's amazing that all this is as reliable as it is. Boggles the mind. Now, imagine having to modify and adapt a mainframe Cobol program to fit into all this structure. No wonder the UPS online billing system is siloed and doesn't coordinate with the tracking system. Dave - thank you for this, it's always important to be reminded what's going on behind the blue door.

[@vernonturner3113]

What is the protocol for the data packet copy that goes to the NSA ?

[@liaminwales]

That's the most clear description of HTTP iv seen.

[@carlosfeliciano8687]

Great video! Thanks

[@stevebrine4324]

Reminds me of when I wrote a dhcp server on a 16bit microcontroller for a project about 15 years ago. All the back and forth of packets. So many nice libraries now.

[@kunhuh]

The single take video style is becoming his signature now. 🙂

[@bshingledecker]

Another important key in the transporting of all the itty bitty bits, is the network, or I believe they are called telco class routers set along the path that routes each block of data to where they need to go. Juniper, Cisco, Alcatel, Ciena, are a few of the makers of these monster routers. Costs of these beasts can hit a million in no time, and give a hint why the internet is not free. I would love to hear your take on networking.

[@Lardzor]

@1:46 I believe first the system checks the 'windows\system32\drivers\etc\HOSTS' file. A shout-out to NAT Network Address Translation would have been nice.

[@matttownsend7119]

Thanks Dave. Please do the same with how packets get to and fro, and how windows security works.

[@BRBTechTalk]

8:22 I don't know that I would call advertising services "enrichment".

[@DavesGarage]

That's why I did the air quotes :-)

[@mds000123]

This was soooo good.

[@GeorgeValkov]

Dave, would you be curious to explore the internals of my web server?

[@HughsMakersCorner]

All this in the time it takes for me to have a sip of my morning coffee!

[@digantakoner]

I know this is a stupid question but is active directory code available somewhere?

[@luckybutunlucky8937]

People will never know the satisfaction of flipping through a yellowpages and collecting the coupons in the back of it.

[@brettspicer6463]

I still remember and how they hurt over the top of the head.

[@NotSoMuchFrankly]

I thought it was to make prank calls more efficiently when not every possible phone number was used. I wonder if younger folks would understand what it means when someone would say in a movie before leaving another character they just met, "I'm in the book." I always wanted the other person to respond, "You're in the Bible?"

[@zoltannemeth8864]

“Hi, I’m an appliance repair technician, I’m calling to see if your refrigerator running?”… etc… Or calling a bar “I’m looking for a friend named Amanda… last name Huggenkis…”… life was so simple back in the day… Ah the good ‘ole days of prank calling…

[@ChitChat]

Fun fact: Try not to think of UDP as "unreliable." UDP packets crossing the same virtual circuit as TCP packets have the same speed and "reliability." Better to think that UDP doesn't provide receipts of delivery like TCP does.

[@zonegamma8197]

I teach this subject this winter, wery useful thanks

[@robumf]

Remember that the Internet was design with redundant in mind. All the security was added later and on top and backwards compatible to the existing protocols.

[@LtShifty]

You should do a video on the part between steps 3 and 4, how a packet is routed across the internet from the local machine, to the gateway, to the ISP, across the internet, to the server. I'm more than familiar with networking, and even today I'm convinced it's still IT voodoo.

[@stargazer7644]

You didn't say which version of HTTP you're explaining. The latest versions are now using UDP instead of TCP.

[@notaras1985]

Why is this

[@stargazer7644]

@@notaras1985 They're switching to UDP because so many devices are mobile now. When you're mobile your network and IP address can change. This breaks TCP based connection oriented protocols. With UDP it doesn't matter which IP the request is made from and it hands off seamlessly.

[@swittman9123]

I don't think this is the 12 step process my therapist had in mind...

[@fragalot]

But how does it work when you have multiple web tabs open, or you're behind a NAS with multiple devices on the same web site.. how aren't data packets getting confused where to go? Is it SYN/ACK part that does that?

[@ilovecatsandsynths9702]

I think you meant to say NAT, not NAS. The NAT translates the IP addresses between your internal network and internet, and translates the port numbers (those numbers Dave mentions during the TCP handshake) between the two sides. Each connection uses a different source port number.

[@fragalot]

@@ilovecatsandsynths9702 Yes NAT sorry about that

[@fragalot]

@@ilovecatsandsynths9702 So it uses a different ports? Wouldn't that all be on port 80 for the web? I know ports aren't hard-coded in to every protocol used (80 for web) but i figured it had to be some sort of unique identifier in the tcp/ip packet that the NAT router sees and knows where it goes.

[@ilovecatsandsynths9702]

@@fragalot Every TCP (and UDP) packet has a source port and a destination port. The request from your browser to the server goes out on a random source port and a set destination port (generally 80 for HTTP, 443 for HTTPS). The response packets have the source and destination ports swapped so the browser knows what connection the packet belongs to.

[@YoutubeBorkedMyOldHandle_why]

Dave ... even more amazing than this incredibly complex series of steps to display a web page, is that you managed to rattle off the entire sequence without skipping a beat. Unbelievable.