Latency Numbers Programmer Should Know: Crash Course System Design #1

[@ByteByteGo]

Correction: The label at 4:46 should be inter-zone networking, not intra-zone. It is the network round trip between availability zones. The closed caption and verbal explanation are correct.

[@donaldsouillet2307]

Still a very good metrological read, thankyou.

[@munteanionut3993]

Thanks a lot for the video! Also for the correction! Very video useful indeed! It puts things into perspective. As a man who writes code myself, I agree with you that the more important idea of the video is "Developing a sense of the relative orders of magnitude difference" instead of knowing exact numbers (probably in ~5 years time these numbers will change again, but the magnitude between them will pretty much stay the same unless some scientifical&technological breakthrough occurs) Just started seeing all of the videos you have in the System Design Fundamentals. Very useful! I hope everything related do systems design from your channel is in this playlist and I am not missing any of them, as they are pure gold haha! Starting seeing you guys as sort of a second freecodecamp channel. This is great and I hope it will stay free. Kudos to you again and keep it up! God bless!

[@sumatepanich-xo6nx]

@@donaldsouillet2307

[@OidhcheMhath]

Another interesting intuition is to put those times on a human scale by setting 1ns CPU time = 1s human time. At this conversion ratio, a clock cycle is ~0.1-0.2s, which is more or less instantaneous for humans. This could be, for example, the time to add 2 small numbers that you already have in memory. If you need to read them off of a page in front of you (L1/L2 cache), that could take a few seconds. Going further, a system call takes a few minutes, context switching is an hour or two, HTTP request takes several hours, you need to wait a day to read something from SSD, and when writing it's gonna be in the mail for around a week. Seek time of HDD is like 2 months, if you need data over the network from another continent that's a few years, and a TLS handshake is like a decade. And God forbid you need to transfer 1GB over network in same cloud region - that's like 3 centuries.

[@willemvdk4886]

Can I somehow spotlight this comment? This is exactly the kind of perspective most people need to get a grasp of these numbers. I usually do a simmilar thing when teaching kids about the age of the universe and how long people have been on our planet.

[@dottn]

The simile I've heard for data access puts you in the shoes of a craftsman on a work site. Register is the tools in your hands, Cache your toolbelt, RAM your toolbox, SSD your car, HDD back at the warehouse and so on.

[@quickmana]

Awesome analogy, thanks!

[@PanicbyExample]

alternately, i was thinking it could be helpful to realize the screen refresh rate as the upper limit of that time scale... 60 Hz makes the screen visible every 6 ms (doubting my math now😬)

[@festivebear9946]

Shows just how incredibly fast computers have made humanity

[@JosuaKrause]

in the UI if something takes longer than 100ms it becomes noticeable by the user. like if you click a button and it takes more than 100ms to react to the click the action doesn't feel instantaneous anymore

[@biskitpagla]

3+ sec page load times are problematic too

[@diablo.the.cheater]

@@biskitpagla WDYM? My users accept anything as long as it is bellow one minute

[@meze2095]

@@biskitpagla might aswell be offline then

[@seraphina985]

Indeed the brain is pretty tolerant at simply correlating the signals together bellow that threshold. It has to be as it has to deal with both the physical lag of the input itself and the fact that your senses each have their own neural processing latency that varies depending on which sense is being activated ~20-40ms for Visual or ~8-10ms for auditory for example. Of course this does all assume you are not dealing with a user with sensory processing disorder as one of the symptoms is breaking this ability to correlate sensory input from different sources.

[@paulgoogol2652]

As a programmer when I click on one of my buttons, I know it is time to get a new coffee because it will take a while.

[@nccamsc]

This channel deserves to be a lot more popular. It is head and shoulders above almost all other IT channels I have encountered on KZbin.

[@TechDiffuse]

Your content is awesome. The amount of knowledge per time is fantastic. Only facts no bs. And the visualizations are great. Thank you

[@snk-js]

truly amazing

[@willemvdk4886]

This is a sheer jewel. Although I knew most of this by experience and interest, having a summary like this is incredibly eye opening. This should be standard curriculum in the first year of computer science education.

[@QuantumImperfections]

I literally love "working with you". Your videos and newsletters (and books) make my data science programming so much more effective and significantly easier to deploy to production. Thank You - you've taken my career to an entire new level.

[@vlkf]

Have you consider to record videotutorial (or maybe whole course) about how to create such amazing visual effects for presentation? Your team very good at it, and those knowledges would be priceless

[@swattertroops-yaaa]

that doesn't make sense for him

[@ccc7421]

Communication, pitching is super important task of architect, so I concur about the idea for the visual effect empowered presentation sharing clips

[@rajdeepdutta1655]

yes I was also wandering how do you make such a wonderful videos

[@StephenGillie]

GIMP animations, with face overlay and pan & zoom effects from Shotcut? You might learn more about video production from video production channels.

[@scarlatum]

> Latency Numbers Programmer Should Know Average Python guy who use openCV for croping images: Thanks, it's so helpful!

[@CalifornianViking]

Another great video. Have you created content that shows the impact of this latency on system designs? Examples: * Accessing memory by reference instead of cloning the content. * Single threaded, vs multi-threaded applications * Async-await vs multi-threaded applications * Impact of microservices and containerization * Impact of serverless I often feel that latency and the cost of communication is forgotten in the scale-out discussion. I have personally seen examples where a containerized microservice architecture was 40-200 times slower than a non-containerized approach. Have you seen the same?

[@JosephLuklukkyjoe]

I am still astonished that content of this quality can be *free. It gives me a lot of hope for the world. Thanks you much and keep it the great work

[@ayporos]

The many round trips of the TLS handshake is something that is actively being improved upon. TLS 1.3 currently supports 1-RTT handshakes and initial support for 0-RTT handshakes.. which would basically remove the delay of the TLS handshake all together as it can be performed simultaneously to an actual content request call.

[@randypenajimenez3893]

This is one of the best channels ever.

[@dragorosson3271]

main memory 10-100us - Network proxy HTTP request routing - Read 1 MB sequentially from main memory - Read 8 KB page from SSD 100-1000us - SSD write - Intra-zone networking round-trip 1-10ms - Memcache/Redis get round trip (~1 ms) - Inter-zone networking round-trip (~5 ms) - HDD seek time (~5 ms) 10-100ms - US east to west coast or Europe network round-trip - Reading 1 GB from main memory 100-1000ms - bcrypt a password (~300 ms) - TLS handshake (~250-500 ms) - US west coast to Singapore network round-trip - 1 GB sequential read from SSD >1s - 1 GB network transfer in same cloud region (~10 s)

[@alias2587]

❤

[@MrInsanepotato]

This channel is incredibly handy

[@VadimTsarkov]

The knowledge density of content on this channel is through the roof. The books are great too. Impatiently waiting for more videos! Keep it up!

[@chaoluncai4300]

indeed! stuffs like branch mispredict penalty, MD5 Hash etc. are complete brand new concepts for a CS bachelor like me, gonna take at least half an hour to digest all details he mentioned in a single video. This channel is like the low-level design version of Fireship while both channel rock the same dense and style loll

[@nezbrun872]

I've been lucky enough to enjoy two simultaneous careers in the 35 years I've been working professionally, one in enterprise database design & troubleshooting, and the other I'm an RF mixed signal electronics engineer: I go all the way down to pico & sometimes 100s of femto seconds for the electronics side. One of the most useful facets of having experience in two apparently largely mutually exclusive skillsets is when it comes to performance & troubleshooting, whether it's knowing what looks about right, or knowing what's reasonably achievable. In my database work, I usually talk in terms of orders of magnitude improvements, but in my electronics work it's more often just ~10% or 20% improvements, as we're already hitting the state of the art. The same dependencies on both experience and underlying understanding are required. In terms of analytical skills, it's pretty much identical: you split a given challenge into manageable chunks, identify the area(s) where a problem lies, re-engineer accordingly., and put it back together. The two are similar in so many ways, even down to Heisenberg uncertainty, such as sticking in instrumentation into code or probing a high speed interface: both can substantially interfere with the result.

[@martincurtis7121]

"...such as sticking in instrumentation into code or probing a high speed interface: both can substantially interfere with the result..." - Guilty!

[@shutanovac]

I love the way each of the videos is short and rich with useful info that helps to expand your software engineering knowhow 👌

[@HackSawSees]

Useful information. A bit of connected data: 10 milliseconds of difference between two notes sounds fine, and imparts a certain "feel", but 30ms starts to sound "off", like one is early or late.

[@TomLeg]

I learned these things in the 1990s, so nice to have an update. Wish you had mentioned cost of an ordinary subroutine call.

[@danielreed5199]

Glad you had enough time available to make this video.

[@distrologic2925]

Your content is highest quality! And very insightful!

[@Carewolf]

Also important 10-100ms the time between two image in animated image. For a game you should typically aim at getting everything done in 16ms to hit 60fps. Or finish all calculations in the 1-10ms range to be able to scale to higher frequency displays. ~100ms is also the maximum amount audio and video can be out of sync before people start noticing it. Already at ~200-300ms it becomes difficult to hold an audio conversion, as people start missing queues about when it is their turn to talk.

[@gnanesharva4998]

Decades of knowledge in 5 minutes ❤

[@sloanNYC]

It always amazes me how few design based on these factors that can affect performance at such huge levels.

[@Mauryarohit1]

You're doing good work. I think this video will be helpful for anyone in computer science whether a student, professional or someone who is just curious. Also the visualisations are great. Thanks.

[@oscareriksson9414]

I love the last one, takes 10 sec to transfer 1GB over network within same region .. then some webpages take several seconds to load a little bit of text and a few pictures.. I mean there are of course many things at play here, it's not only the transfer, which is considered here, but still we have a lot to improve on in general when it comes to web. And I don't only mean the programming I mean choice of server OS and so on. Some of the extra latency can be due to hundreds and thousands of processes being run concurrently on the same servers . So if the kernel interrupts every n mikroseconds, it takes the amount of nano/microseconds for every process to go from userland to kernel land which adds up until the scheduler gets back to the process again.. And storing and loading the process state in it self also requires cyckles aka time to do every time. Then there are more stuff going on as well.. (I didn't remember the numbers presented in the video but there all there in the video :) ) We need less bloaty things even though hardware gets better. Sometimes today surfing the web is almost the same experience as the early 2000-s..

[@OatmealTheCrazy]

Most of that is taking bids to sell ads by now lol

[@oscareriksson9414]

@@OatmealTheCrazy lol yeah! There's also that!

[@ManagementManagement-vu1mk]

trap -> kernel space -> trap -> user space, not accounting for processing time of the system call itself) - MD5 hash of 64-bit number 1-10us - context switch between Linux threads (does not account for bringing large # of pages for the new thread, which would extend time further) - copy 64 KB memory to new location 10-100us - typical amount of time for network proxy to process a http request (nginx) - read 1 MB of sequential main memory data - read 8K page of SSD 100-1000us - SSD write latency for 1 page - intra-zone networking latency [100us] - memcache/redis get operation as measured by client [1ms] 1-10ms - inter-availability zone of modern cloud provider networking latency [5ms] - hard disk seek time [5ms] 10-100ms - us-west coast to us-east coast latency, us-east coast to europe latency - read 1 GB of sequential main memory 100-1000ms - bcrypt a password [300ms] - TLS handshake [250 - 500ms, depends on distance between machines] - us-west coast to singapore latency - read 1 GB of sequential SSD data 1s - transfer 1 GB over cloud network within same region [10s]

[@jdmji]

Fascinating to put this into perspective! Specially access speed differences between registers, cache and memory. Also context switching for threads

[@probablypablito]

These videos are cool! I miss the videos where you explain more basic concepts such as TLS or other protocols. I found those the most interesting

[@ByteByteGo]

More to come! Do you have any topics in mind?

[@Piupiumacher]

@@ByteByteGo mqtt would be quite interesting

[@georgiiperepechko9321]

@@ByteByteGo a video about cache hierarchy (L1 to L7) might be really cool :)

[@mark-6572]

@@ByteByteGo RabbitMQ, ampq, celery.

[@mnchester]

@@ByteByteGo Push vs. Pull model for getting events

[@DrDiabolical000]

Awesome video. This was important to get the time intuition of different processes.

[@EbonySeraphim]

Excellent video. I definitely have intuitively known some of these numbers for a long time being just a super nerdy programmer with a gamedev hobby. However, I came up a while ago not knowing a lot of the cloud numbers, and L3 cache didn't exist in CPUs and maybe not even L2 in the earlier days. No surprises really exist other than SSDs being a lot faster than HDDs. I've known intuitively that network transfer rates (bandwidth, not latency), but now even latency of network access is faster than an HDD. In otherwise, a locally connected HDD is _slower_ than an intrazone networking transfer if the server is reading from main memory or significantly faster IO storage.

[@RoelofaCoetzee]

From 1s down to 1nano second would have made much more sense. Starting from something 'relateable' down to something almost "abstract" instead of something we have no direct "feeling" for and working back to something we know.

[@alaminshuvo3815]

This video really helped me. Thanks a lot!

[@smilin_dominator]

Amazing video! Loved the visuals and the explanation!

[@peterstedman6140]

I share your videos with all of my developer friends. Great for a refresher, summary, or introduction to these concepts. Keep up the excellent work!

[@ercanmamur]

Thank you. Very important and helpfull context

[@MRApht]

Greate video, thanks for making it! I think this knowledge is useful when reasoning about software, for instance if you have a simple task that takes a second, you might have made a mistake somewhere, like a n+1 query. If you don't have this intuition you might think: Oh, it's just a second. But a second is literally an eternity for a computer. If your operation takes a second, you better know why!

[@boringmanager9559]

You really picked up publishing pace ☺️👍🏻

[@ByteByteGo]

Same pace since the beginning in June. Once a week on Tuesday.

[@gus473]

@@ByteByteGo It has become a highlight of my early week! Thank you! 👍🏼😎✌🏼

[@MikkoRantalainen]

Great summary of meaningful latencies. I think it would have been nice to include human reaction to in 100-200 ms range. This is the minimum time humans need to actually react any new information, not matter how little semantics that information contains. An example would be ability to press a mouse button after something appears on the screen. Another good latency to include would have been 10-20 ms for the minimum time to get something on the screen on most computers.

[@arsenylosev1784]

Awsome video, both explanations and animation are fabulous! Thanks a lot!

[@viveksmenon123]

Thank you! I have bought both your books!

[@randomstuff1669]

simple yet elegant.

[@JohnSmith-jq9xx]

This is awesome content !!! Thanks.

[@_chip]

I would have loved to see gpu read and write latencies on here but I really loved this video

[@Maouww]

This is great - going straight to my bookmarks, thankyou :)

[@nhanNguyen-wo8fy]

Sound like a full course for me!

[@xdyps]

Would have loved to see a similar video with the comparison to how tech has evolved , anyway great video 🎉

[@yuriyv5812]

Very nice 👍 Helpful to have this kind of intuition while debugging performance issues in complex distributed systems

[@SiliceIO]

Wow greate explanations, thank you

[@LionKimbro]

100ms -- the time at which a human being regards a response from a computer as "instantaneous" -- for example, if the user presses down on a key "X", and then an "X" appears on the screen within 100 milliseconds, then the human being judges that the computer responded "instantaneously" -- VERY IMPORTANT for programmers to know!

[@danchoatanasov1611]

Your videos are amazing. Thanks!

[@BritishBeachcomber]

I'd be interested to see thread switching and kernel call latency comparisons between Linux and Windows.

[@snk-js]

good point 🤔. but I think Linux is some 10% - 30% faster

[@PimpoDiabolo]

Very nice video! Thanks!

[@kriegeadler]

Simply thank you very much for your videos!

[@higiniofuentes2551]

Thank you for this very interesting video!

[@fitybux4664]

100 to 1000ms: User perception of a "fast response" on the internet is around this range. (Google says 200ms.) That means if you want your page to feel responsive, a response better be delivered and rendered to the user before 200ms have passed after a mouse click or screen tap. 1s to 10s: Range of spindown time of spinning hard drive. 10s to 100s: TikTok and other short format videos are in this range. User attention becomes this 15s limit, meaning users get bored super quickly. (Especially young users.) Also, interestingly, if a system does not respond in this amount of time, users might think your program has locked up and try force closing/rebooting/etc. Along the same lines, most UNIX and Linux systems send SIGTERM to all process, followed by 5 seconds wait, followed by SIGKILL. (Your program has 5 seconds to clean things up before it's forcefully killed.) 100s to 1000s: TCP maximum retransmission time is 120 seconds by default. If DRAM is cooled, the contents can be retrieved in this amount of time after power is removed.

[@michaelnurse9089]

Human visual reaction time is about 200ms. An interface response faster than this has diminishing returns.

[@xxRAP13Rxx]

Shouldn't reading 1 GB sequentially take about 50 milliseconds if reading 1 MB sequentially takes about 50 microseconds? Also, does an intra-zone networking round trip take 100 microseconds (according to 4:16) or 5 milliseconds (according to 4:46). Or is 4:46 referring to intra-zone networking latency instead of an intra-zone networking round trip?

[@ByteByteGo]

Thank you for the feedback. At 4:46 it should be inter-zone networking instead of intra-zone. This is the round trip time between availability zones. The illustration is wrong, and the caption and verbal explanation are correct. For reading 1GB sequentially we are on the same page, right?

[@xxRAP13Rxx]

@@ByteByteGo At 5:50, it was stated that reading 1GB sequentially takes 100-1000ms. I thought reading 1GB sequentially takes 50 milliseconds instead.

[@ByteByteGo]

At 5:50 it was reading 1GB sequentially from an SSD. Reading 1GB from main memory is at 5:10, which is in the 10-100 ms bucket.

[@xxRAP13Rxx]

@@ByteByteGo Thanks for the clarification! Great video!

[@sandeepnegi4939]

amazing videos It must took you a lot of effort to combine the resources and put it in a simple way , for viewers they need to take notes and then understand each things mentioned in the video :)

[@bossgd100]

I liked the video. See you in 10 years for the new version !

[@TheMicstep]

Another good important point PCI-E devices roundtrip is about 500ns

[@batchrocketproject4720]

Fascinating, thanks. It had not occurred to me before that a (relatively) slow hash algorithm has the advantage regarding resistance to brute force trials. I'd love to see a comparison of server response latencies; I'm guessing a liteSpeed server using ssd storage located a continent away would respond faster than a local apache server with a moving disk.

[@alamelu85]

Awesome one - Thanks a lot

[@WilliamLeStrange]

Great content and presentation. Keep up the good work!

[@imiebaka]

The beauty of computer science in one video

[@Konservator69]

It is worth mentioning 100ms border of a human perception. Any operations faster then this we treat as "instant". Anything longer - as having some delays.

[@imranhussain8700]

Great content as usual, and amazing presentation to explain the same. Loved it 👍

[@pankajchaudhari2588]

Just amazing stuff. Excellent!!!

[@RonJohn63]

35-40 years ago, the latency hierarchy on a PC was: 1) Registers 2) RAM 3) HDD 4) Floppy drive 5) Monitor 6) Printer

[@KelvinMeeks]

Great video!

[@sharwariphadnis1298]

Amazing video once again!

[@GlobalYoung7]

thank you 🙏 ❤

[@Matlockization]

What a very fascinating thing to talk about.

[@ethanmye-rs]

Hey, nice plaque!

[@ByteByteGo]

Hey, thanks! 🤣

[@orafasistemas]

Nice videos ...thank you.

[@mikej9062]

A good sister video would be to review these numbers for an embedded system, maybe an ARM microcontroller.

[@Aethid]

Coming from a game dev background, I'm used to thinking of latency in clock cycles, and anything more than a few hundred is "too long".

[@wannabelikegzus]

That redis cache read number is pretty surprising. I suppose that's better than having to get a DB to get a result, but network latency for intra-zone is only 300 micros, so if you can just have a fast database, you won't lose a bunch of performance there. Honestly, if you're not using a cache system within your app, you're leaving a ton of gains on the table.

[@r0sh4n0]

thanks for the info good representing and good editing

[@IFearlessINinja]

I am very impressed with your script writing. It is more clear and efficient than any other I've seen on this entire website. Extremely effective.

[@jeffchoate8712]

I would add that at 10-100ms is the range which many consider to be "real time", you should have latency under 15-30ms if you want to attain 60-30fps due to some complex command like computer vision.

[@nsuid7499]

Great video! I'm curious about how you got the estimations for these operations.

[@zeroregretsgiven]

As always great video

[@thejswaroop5230]

Me switching tabs when someone walks in : 0.000001 ns

[@ByteByteGo]

That a good one. Great dexterity. 🤣

[@yurangtian]

Thank you for sharing!

[@NursultanBegaliev]

Thank you! It is really helpful)

[@pcfreak1992]

What blew my mind was that accessing main memory on an M1 Mac is in the same order of magnitude as accessing the L3 cache 😳

[@aravindpallippara1577]

M1 is using lpddr5, faster ddr5 is now starting to become available for desktops that has faster access speeds

[@snk-js]

your vids are so perfect

[@MrFleit]

I would find it cool to add perceived latency on the ui. there are papers on this, like when does a ui feel "laggy". For example a button has some highlight effekt on mouse over and if the effekt takes too long to start it feels laggy. I have in the back of my head the mark of 50ms for those things, but could be wrong.

[@seraphina985]

It's typically closer to 100ms in most people, bear in mind there is around a 20-30ms difference in neural processing delay between visual and auditory signals so a normal brain needs to be able to integrate at this level. Touch signals can be even slower you are looking at closer to 40ms (Which is at the slow end of visual processing time) for a signal from the toes to reach the brain before processing can even begin for an adult (~2m @ 50-60 m/s). If it wasn't able to do it over windows at least in the 50ms range or so then coordinating anything involving multiple senses would be significantly hampered eg using vision and proprioception to manipulate objects with precision etc.

[@seshasaivenkat]

Brilliant Many Thanks

[@MechMK1]

Minor nitpick: bcrypt has a variable workload parameter, which allows one to freely configure how long it will take to calculate a hash for a given amount of data on a given hardware. Thus, saying a slow hash-function takes between 100 and 1000 ms isn't really correct. It can take however long it is configured to take.

[@ByteByteGo]

Good point. Appreciate the clarification.

[@raghavamorusupalli7557]

How minute your discussions have been, Sir

[@stevenbliss989]

Thanks, nice to hear common sense from pros. Sometimes I train junior programmers, and their ignorance of time and size of things is beyond GARGANTUANT. Even simple things like string to integer conversion calls, simple but expensive (relatively speaking) are ABUSED like crazy in code that has no business doing that i the first place. IGNORANCE IGNORANCE IGNORANCE.... so THANK YOU! :):):)

[@bitti1975]

"Network latency between countries stay pretty consistent because they have to obey the laws of physics". Interesting, so the other things don't have to obey the laws of physics? What you probably mean is that this latency is already near the physical limit.

[@HL65536]

You have to put into perspective the distance light travels in these times. In 1 ns light only travels ~30cm. With a speed of electricity of ~2/3 c that means it is physically impossible to access information further than 10cm from the CPU in 1ns. That means anything the CPU has to do in 1 clock cycle of let's say a 5 GHz CPU has to be physically within 2cm (a bit less than an inch for the americans). More realistically, it has to be within 1cm as any logic also takes time and electrical connections are rarely point to point. And this might soon or already now need a 100ps-1000ps range as clock speeds are getting faster again.

[@Noobificado]

I really love your content

[@HM55-77]

great video

[@kyrylosovailo1690]

Starting Internet Explorer: ...this tiny maneuver will cost us 1000 years

[@enitalp]

The day we will invent communication faster than light, will be great for space, but also for computers / internet