How an HTTP Request Gets Served - In Great Detail

[@fairalbion]

Each of these steps has a huge number of background operations going on to make it happen. I'm a retired DNS guy, and that process alone could occupy one of Dave's videos.

[@johntoe6127]

Computing is a perfect example of the Dunning-Kruger effect. The average person completely under estimates the complexity behind things that have become common.

[@brodie3088]

@@johntoe6127 definitely. I know a guy who is like a 10x dev wizard type dude. His motto is "all I know is that I know nothing". Personally I gave up on being a web dev about a year and a half after I became competent enough to do some cool stuff. The realization that just because you made it work doesn't mean you didn't build it in a way that somebody smarter than you would consider stupid is a tough one to stomach.

[@Bunny99s]

@@johntoe6127 The Dunning-Kruger effect has nothing to do with underestimating the complexity but only with over estimating the own knowledge. Those are two very different things. Explaining how DNS roughly works is perfectly fine, even when you don't know every little detail. Telling a lay person that it's essentially a telepone book for the internet is enough to get an idea what it does or what it's there for. As long as you don't claim you know everything about DNS, there's nothing wrong with only having partial knowledge. I for myself have written my own DNS server from scratch in C# which currently runs on my raspberry pi with mono :P Though I've only implemented what I needed / wanted and it was more a proof-of-concept / learning project. I've read through most of the RFCs. I didn't implement any DNSSEC stuff. Though my server supports both, UDP and TCP requests. It does not act as a resolver for other zones than my own as I only serve my own records. Currently every request and response is written as a hex dump to the console :) Runs stable for over a year non-stop now. So I don't consider myself as a DNS expert, by no means, but I have a fairly good understanding of the basic protocol, even though I don't implement every extension (DoT, DoQ, DoH, TSIG, ...). Though it's funny to see how many requests come in to check the "bind" version as most would assume that a DNS server is some variant of bind. Berkeley has set quite a few standards over the time. A lot of people who "work" with DNS don't really understand the fine details of the protocol either. Most work only on the configuration level. DNS has a fascinating "name compression" system for requests as each part of a domain name is split into it's individual labels and when multiple things are requested or returned, the protocol can use back references to labels that already existed. Those can even be recursive. Quite fun to implement a parser and encoder for that. I often miss many concept of the "old days" where they tried to spare every bit without the need to make it overly complicated. Modern day development is more about: "we do have the CPU power, memory and bandwidth, so f-it". The "old" protocols were all designed to be easily parsable by hardware implementations. Most modern stuff is just a mess with tons of overhead.

[@mikko-penttieinarieronen]

@@johntoe6127 I like to adapt the Coastline paradox into computing and technology or any other process-related subjects. The more you dig into each component or step, the more questions you'll end up having :D And often you forget some of the old stuff you have learned, so it's impossible for one person to handle/understand everything.

[@jcasetnl]

Do you sometimes still volunteer to translate names to IP addresses in your spare time?

[@RichardCollette42]

Hey, Dave. As someone who has been teaching this stuff for nearly 30 years, I can say without hesitation that you are excellent at it. I found myself saying out loud, "Dave, you forgot to mention ..." only for you to say it five seconds later. I recommend your channel to my students. On a side note, how about a follow-up video on TCP vs. UDP and, more importantly, QUIC? I'm amazed at how little info there is on QUIC in the traditional educational resources I (must) use. Perhaps more Layer 3 info on how IP works as well.

[@Richardincancale]

Yes - maybe another follow up video on how a KZbin video of Dave gets to you - with distributed network of YT servers and use of QUIC to overcome the round trip time that was a problem video over TCP - especially over 4G and / or satellite - my domain!

[@PeterBachman]

QUIC makes the certificate key exchange simpler than TCP, fewer round trips, better suited for streaming

[@jholloway77]

I built myself a simple HTTP 1.0 server in C as a little project. Learned so much about how the protocols work, how sockets work, and how to build something according to documentation & specs. One of my favorite projects that I've done for fun.

[@julianocardarelli5106]

Any specific library to handle sockets and open tcp port on server side?

[@jholloway77]

@@julianocardarelli5106 @julianocardarelli5106 It was a very, very simple server just using the standard POSIX libraries built into FreeBSD. Nothing external required. I'd share a link to the github, but KZbin keeps filtering out comments with links

[@brianjuergensmeyer8809]

Nothing teaches you how something works until you try to implement it to spec.

[@jholloway77]

@julianocardarelli5106 it was a basic application and just use the libraries found in any POSIX system. I made mine on FreeBSD

[@fakedevdutt]

TF nerd! nice work tho

[@IainNitro]

Hi Dave: Great video. Been working with the web since around 1993. I learned a lot working at an ISP (we had dial-up then) and learned a lot of this. I loved the reference to the DNS system as being the "phone book" for the Internet! I am definitely old enough to remember using those big paper books. Thanks for your info.

[@yoman912567]

These are my favorite types of videos! You do such a good job and explain a level deeper than so many other channels. I learn more in these type of vids than years of KZbin IT osmosis.

[@TechnoTim]

Thanks Dave for a great explanation without all of the hype

[@fieryweasel]

He forgot the step of hurriedly closing the tab when your boss walks over.

[@michaeltyniec7010]

Tetris had that feature - a certain keystroke would switch it to Lotus 1-2-3 (DOS mode) if your boss was coming over.

[@robertbelanger2834]

@@michaeltyniec7010 That was the BOSS key, the first one to learn when playing on company computers...

[@daniellundqvist5012]

I did the Tetris thing in reverse, I incorporated a sourcecode version of tetris into the system that would start with the Break key, and stop with the Escape key😂

[@FlameStrykeShadowDark]

.....or when the tab inexplicably begins blaring music.....

[@batteries76]

Your boss lives in your house?!

[@johnneyland3334]

You’re the best Dave !! Love your videos and knowledge content !

[@JV-bw2fw]

Excellent video sir; thank you for all the effort, time and dedication that goes into each of the videos you make. Please trust me when I say, there are some of us who truly value and appreciate it.

[@christophermcdonald8732]

Awesome explanation Dave! Fascinating to see the details, keep it coming!

[@yorkaturr]

Every single web developer needs to know this as well as ways of debugging every step. Well done!

[@nomadic_shadow]

If you started an “in great detail “ series dave, I don’t think anyone would be mad.

[@zoltannemeth8864]

This would be awesome!

[@steamer1]

I think you'll find I second this motion.

[@alex-m-jones]

Dave I think a SYN is an empty packet with the SYN flag set.

[@turkishmaid]

an "in great detail" series for every one of the steps, please... Maybe 2 hrs per step as a starter...

[@omanshsharma6796]

I third this motion

[@zoltannemeth8864]

Back before the internet existed, this process was all done manually, I used to hate the three-way handshakes, but I did enjoy the cookies! Seriously, another great video Dave! Thank you for sharing your experience and knowledge with us!

[@ArsNoble]

I first held a phone tat used to dial ..... I always wondered why internet was like dialing a phone number .

[@Atsumari]

I am a web tech and manage servers for websites I am glad you went over this. There are so many people who think this is magical and that the internet just works; no it doesn't there is a lot fo work, systems, deamons and configuration files involved. The internet is fascinating and humbling if you understand all the work that goes into making it work and keeping it up.

[@subbastionbastion2167]

The protocols doesn't explain how it works at all honestly. Like how the ip address knows where it is in the world and how a device connects to it. Dns to ip ok but what does the ip do

[@alexanderboehm7473]

@@subbastionbastion2167 Off the top of my head - Internet Protocol is assigned by a group called IANA. They help manage this because public IP addresses must be unique - unlink private/internal IP addresses. Your ISP is essentially responsible for routing your traffic through different hops (sometimes managed by the ISP, or sometimes when allowed, through other Autonomous Systems). These hops communicate with each other using BGP - an exterior routing protocol that allows for dynamic IP address routing over the internet.

[@dooza]

@@subbastionbastion2167 An IP address does not know where it is. Routers and ISPs figure out where the IP packets go. Each ISP has bought a set of IP addresses that they assign their customers. When your router has sent the packet to the "Internet highway" (as it saw it was not LAN applicable), the ISPs have their own routing table to know where to send it. "Hmm, this looks like an IP from an Swedish ISP I know. I'm going to send it that way".

[@jacobm1190]

@@subbastionbastion2167 The packet is sent through many routers as it makes its way to its destination. First to your home router, then to a router belonging to your ISP, and so on. Each router looks at a huge table in its memory to determine in which direction (that is, to which router) it should send the packet next. These tables are updated by the Border Gateway Protocol (BGP) which operates globally. That's as far as I know because I am only a CCNA.

[@komraa]

​@@subbastionbastion2167well every single key word there is a large field in itself. You can find books on these pretty easily

[@grottyboots]

Love these deep dives into technical subjects. Your explanations are so damned good! I recall reading Comer's "Internetworking with TCP/IP Volume One" and being blown away at the brilliance of TCP/IP. Keep up the great videos, sir!

[@MrSunDevil23]

Where were you in the early 90’s when I was just learning networking and getting my MCSE… oh yeah, you were working on the OS I was learning!!! Great video as always. While this is mostly old school info for an old engineer like myself it is a great refresher and will be very VERY helpful to newbies who want to learn.

[@toby9999]

It's all new to me... kind of. I've been struggling to understand this stuff for years. I've been a C++ Windows application developer for 28 years, but anything rated to networks does my brain in.

[@MikeD-tf7dk]

As someone that’s been in this world for almost 30 years, you remind me of why it’s important to be open to learning new stuff about subjects, you already have a good understanding of. extra bonus geek question as someone that’s been in this world for almost 30 years. You remind me of why it’s always helpful to be open to stuff you may not know. Extra bonus geek question tho. I didn’t hear you make a more detailed explanation as to what a “static” page is. I’m wondering if you’re so used to it that it seems self-explanatory. The only reason I realized it was I was about 3/4 of the way through when you said it again without explaining what it was.

[@larryjanus596]

I'd tell you a UDP Joke, but you might not get it.

[@DavesGarage]

You could just resend it, though.

[@Sobioytccc]

​@@DavesGaragelet TCP handle it for you 😅

[@titrecords2294]

I guess that proves it sends cuz I got it 😂😂😂

[@SajadKarami-hi4cd]

just keep telling it till I get it😂

[@dailydose7772]

Subscribed after just 2 minutes of your awesomeness. Thanks for your time and knowledge

[@javabeanz8549]

Great explanation Dave! For how complex the process is, I think that you broke it down quite well, to be digestible for the general public.

[@kevinL5425]

You really appreciate how much work the computer and network is doing when implementing it with RFC 1149. Personally I think that implementation is for the birds but some people have used it.

[@patric_forreal]

Human kind can build great things 🎉 What a time to be alive.

[@floriansalihovic3697]

Double you tee eff. That’s as entertaining as educational - with a delivery worth an audiobook. Well done - subscribed after 12 seconds. Great work

[@bw6378]

Apache and nginx but no IIS? Oh, the irony is not lost on me!!! LOL. Thanks for the video!

[@anon_y_mousse]

He's just admitting what we all know, IIS sucks.

[@iabhimanyudwivedi]

😂😂

[@electrictrojan6719]

No one cares about windows, not unless paid to care.

[@rodsmade]

ijust wanna hug you bc of this. tysm it makes knowledge more accesible ❤

[@jimsubtle886]

Awesome video yet again Dave ! I was very pleased with part 1.. I cannot wait to see part 2 !!

[@FlightEagle]

The best explanation I've seen, thank you. Included correct and appropriately complete detail of the process.

[@larrycleeton]

It frightens me that I have intimate knowledge of everything you described. Kudos on leaving out the really boring details. One of my minor claims to fame as a former employee is I personally removed the gopher protocol from Internet Explorer that became version 7. 😎

[@sdrc92126]

In the olden days, it took decades to earn these things

[@surabhsawant]

such a great video! had a great time watching it probably because I had enough context of the web flow but still it had me hooked up till the end in one go.

[@NosAltarion]

OK, this one is too good to ignore. I may translate most of that and present it to my students. Very good content

[@ionamygdalon2263]

Nobody can explain these things better than Dave! I often send these videos to people I know who are struggling with these concepts and they all tell me how much it helped!! WELL DONE DAVE 👍👍👍

[@davidjsutherland]

Great content Dave. Missed the Friendly Giant ending though.

[@XORBob]

I will be using this video for my Web Dev class next semester. Thanks Dave!

[@markmonroe7330]

Excellent presentation. Thank you.

[@quambafu]

Great videos Dave! My 16 year old enjoyed the Hello World video and learned quite a bit.

[@NathanaelNewton]

This is my new favorite series on all of KZbin, please make more😮❤🎉

[@invalidred]

The level of details is tooo good

[@flowerofash4439]

so dense of information in one video, incredible

[@brodie3088]

This goes further than the request, because HTTP request is a separate concept from the browser rendering the HTML (although I guess you could say the concept's are closely tied due to the rendering process usually resulting in a waterfall of additional HTTP requests). Not that I am complaining though! I really liked the video

[@Noam-Bahar]

This is the clearest explanation I've heard so far

[@davidjowett8195]

Once again informative and enjoyable. Thank you.

[@ThunderChasers]

Love these videos. You'll have that gold play button in no time!

[@thehauntedstream7206]

Incredible brother thank you

[@wxfreak]

Thanks Dave

[@Sir_Uncle_Ned]

As an IT networking guy I can confirm that this does paint the general picture of what happens when you request a web page; But every element of that process is a seperate rabbit hole of processes, servers, math, and ungodly gallons of caffeine to keep it all running. I can see why you didn’t go down those rabbit holes for this basic overview

[@alexanderboehm7473]

I thought he explained things pretty thoroughly - of course, there will always be more details.

[@DetectiveConan990v3]

what do you do as an IT networking guy

[@Sir_Uncle_Ned]

@@DetectiveConan990v3 mainly look after switches and routers. If I do my job well the people I work with don’t notice.

[@ArsNoble]

So ya guysknow i watch cat videos 😱

[@manzoor5554]

Exceed Dave please keep posting more detailed Deep Dives ..😊😊😊❤❤

[@IAmUsingAndroid]

Thanks dave for such a detailed explanation.

[@omanshsharma6796]

Wonderful content, thank you!!

[@billj5645]

And it's amazing that for many websites that are responsive all of this can happen in under 1 second.

[@sueterra3911]

GOT IT! Gotta run; my landline is ringing.

[@scratchanitch]

Hey Dave, love these full end-to-end discussions. When you were working at Microsoft did you ever have lunch at Crossroads? If so, we probably crossed paths 🙂. More importantly - an idea for a video is for a deep dive on memory management in Windows and how it compares to Apple (and their unified memory architecture).

[@Lardzor]

@1:46 I believe first the system checks the 'windows\system32\drivers\etc\HOSTS' file. A shout-out to NAT Network Address Translation would have been nice.

[@breakoutgaffe4027]

Super nice explanation 🔥 🔥

[@chuckistheman2000]

I was asked this question during an interview. Pretty much answered the same way

[@411sev]

Well explained. Thank you very much.

[@YoutubeBorkedMyOldHandle_why]

Dave ... even more amazing than this incredibly complex series of steps to display a web page, is that you managed to rattle off the entire sequence without skipping a beat. Unbelievable.

[@H4R4K1R1x]

Cool video, an in detail series would be well received.

[@DutchShaggy]

Happy to see you included the Host file, but did not put to much attention into it. If people are going to play with it and forget, it can generate a lot of t-shooting.

[@DavidWonn]

We too easily take for granted how quickly the overall process happens these days. But in the dial-up days and much slower processors, you could almost witness several of the steps happening in real-time if you knew what to monitor.

[@kimtyson9101]

Very nicely done!

[@DragonRides75]

another great video Dave thanks for this video

[@fritzmusic]

I have a static IP address so this is not relevant to me. Yes, I'm being facetious. Love your informative videos Dave and always watch and upvote and comment.

[@DavesGarage]

I have 5 gigabit fiber that I paid to have installed and they STILL won't give me a single static IP :-(

[@toby9999]

What difference does a static IP make within the context of this video? I'm clueless 😊

[@pete3897]

@@DavesGarage RFC1918, brother- make you own! :)

[@David_Best]

It's amazing that all this is as reliable as it is. Boggles the mind. Now, imagine having to modify and adapt a mainframe Cobol program to fit into all this structure. No wonder the UPS online billing system is siloed and doesn't coordinate with the tracking system. Dave - thank you for this, it's always important to be reminded what's going on behind the blue door.

[@MagicMan123ification]

am not ready for this , its major

[@shantanusapru]

Anther great video, Dave!!

[@carlosfeliciano8687]

Great video! Thanks

[@kunhuh]

The single take video style is becoming his signature now. 🙂

[@HughsMakersCorner]

All this in the time it takes for me to have a sip of my morning coffee!

[@gerakore8948]

back in the day i used to make my own web servers that added background functionality before web 2.0 was a thing. also used this to circumvent cell phone web access limitations.

[@liaminwales]

That's the most clear description of HTTP iv seen.

[@vamsikrishna-mx4kn]

Thanks Dave

[@jemdeweare6432]

Interesting dave , thank you

[@davidadams421]

Fantastic! A great follow-up video would be explaining website performance e.g. Google Pagespeed Insights, how DOM complexity, resource sizes e.g. images, largest contentful paint, javascript blocking etc. effect the performance of the website.

[@JT-kz7kq]

Excellent video as always. What about a follow-up video as to how data actually gets moved through networks and the internet (mac addresses, subnets, gateways etc)?

[@manuelcheta]

At step 10 the browser also creates the Accesibility Object Model, a tree mirroring each element on the page with details like name, role, value. The accessibility tree is created so that accessibility tools interact with it every time when those tools interact with a webpage. Basically the accessibility tools will interact with the accessibility tree, not with the web page directly.

[@DavesGarage]

Did not know that, thanks for adding!

[@brianjuergensmeyer8809]

@@DavesGarage I was involved with this in a previous existence: the gov't has legal requirements about maintaining accessibility for both desktop apps as well as web pages. There are some JS libraries that will help, but there's a lot of info that goes into cueing things like JAWS that you're in a table vs. a frame. Unfortunately, a lot of (non-government) sites tend to gloss over the accessibility aspect, leaving accessibility technology users with an unintelligible mish-mash.

[@ronm6585]

Thanks Dave.

[@kwazar6725]

Dave and I are similar age and done similar things. Its fun to see you tell our story

[@zonegamma8197]

I teach this subject this winter, wery useful thanks

[@stevebrine4324]

Reminds me of when I wrote a dhcp server on a 16bit microcontroller for a project about 15 years ago. All the back and forth of packets. So many nice libraries now.

[@Frisky0563]

Thank you!

[@AngeloMartino-z8f]

5:15 Actually when you type a website's name in your browser, your device sends a DNS request to translate the domain name into an IP address and your ISP often handles these DNS requests, so they can log the domains you're accessing even if you're visiting HTTPS websites

[@mds000123]

This was soooo good.

[@tubeDude48]

Well done Dave. Here's a thought. How about covering tags. I have quite a few at the top of my web page.

[@professor-viewsalot]

Is that all it does, so why does it take so long? The bit I don't understand is how Dave gets to the end without having to add an ammusing sarcastic smile. Great video presentation Dave.

[@scottstefanatz]

Good video, only you you didn’t mention the network socket layer that breaks the data into sized packets. Each packet contains header info about origin, destination, last network component to handle it, how many hops it has taken and time to live, and sequencing info to know how to put the data back together.

[@robumf]

Remember that the Internet was design with redundant in mind. All the security was added later and on top and backwards compatible to the existing protocols.

[@LtShifty]

You should do a video on the part between steps 3 and 4, how a packet is routed across the internet from the local machine, to the gateway, to the ISP, across the internet, to the server. I'm more than familiar with networking, and even today I'm convinced it's still IT voodoo.

[@bshingledecker]

Another important key in the transporting of all the itty bitty bits, is the network, or I believe they are called telco class routers set along the path that routes each block of data to where they need to go. Juniper, Cisco, Alcatel, Ciena, are a few of the makers of these monster routers. Costs of these beasts can hit a million in no time, and give a hint why the internet is not free. I would love to hear your take on networking.

[@LivvieLynn]

Browser address bar is just one big manual GET request with some rendering. Outside of browser features for searches, local resources, or code etc.

[@DNHarris]

You've been served!

[@mcwolfbeast]

03:00 And yet, Google is pushing using UDP for QUIC/HTTP3 instead of TCP. I'm agreeing with you here, I just wonder how a giant like that can get it so wrong.

[@johnburgess2084]

Thanks for the great explanation Dave. You missed the steps of announcing to Google, Facebook, TikTok and all the other nosy trackers out there what you're doing.

[@lindadoune]

Oh he did. That info is included as the smirkable "relevant data".

[@matttownsend7119]

Thanks Dave. Please do the same with how packets get to and fro, and how windows security works.

[@FootsieAU]

Great vid Dave, but what about SNI? That usually exposes the hostname of the request before the TLS handshake is complete.

[@LotharLindinger]

Excatly what I wanted to add. I can't believe Dave doesn't know about it but maybe his mind is already TLS 1.3 only.

[@geepeezee5030]

Agreed. We need more sites implementing ECH. Otherwise, does no good to obscure dns with DoH/Q/T/etc.

[@toby9999]

Wish I'd seen this five years ago. Never made any sense. I'll have to watch this a few times to absorb it.

[@Latrocinium086]

Man I love these videos. Feel like I’m back at devry w a teacher who actually knows something 😂 Any chance you can cover we assembly?

[@byronwatkins2565]

I learned a great deal; however, I also noticed that sound was not considered and that scroll bar windowing was omitted.

[@ChitChat]

Fun fact: Try not to think of UDP as "unreliable." UDP packets crossing the same virtual circuit as TCP packets have the same speed and "reliability." Better to think that UDP doesn't provide receipts of delivery like TCP does.

[@alastairnewman1233]

A Question. Is the Browser communicating directly with the GPU or is the Browser communicating with the GPU via the Operating System? And Dave. You managed to do this in 16 minutes! You packed it in and knocked the explanation out of the park. Thanks very much.

[@DavesGarage]

Glad you liked it! I'm not up to speed on the latest browsers, but the image compositing allows them to render elements to different GPU surfaces and then merge them in the GPU, so it seems like the GPU is being used directly.

[@meisievannancy]

Wow. What a memory you have.