Blame Me: I Worked on the Windows Progress Dialog!

As one of the devs that worked on it at Microsoft, Dave explains how the Windows progress dialog works, why it's so often wrong, and why it's such a hard problem to solve in the first place!
Secrets of the Autistic Millionaire: amzn.to/3diQILq
Discord: / discord
PS: Measurements is spelled wrong, as is "acruate", which is almost funny. Almost.

[@dino6627]

It is frustrating when it gets stuck at nearly completed and doesn't recalculate, but a real progress bar these days is a rarity and much better than the endless spinning circles.

[@DavesGarage]

True! I show the opposite in this video, where it surprises itself with an early completion!

[@MrJC1]

yes why do we keep using spinning circles these days? it isn't good design. its streamlining for streamlinings sake.

[@esenel92]

@@MrJC1 But it's soooo much easier. ;) And of course many applications these days get data from the web, and might not even know how much data there is before it finishes, it just keeps reading the server's output as it's generating it. And of course there's a whole lot of lazyness these days with companies and/or developers. Everything has to be done in a hurry, so why even bother trying to get it right?

[@DeanWilliams1987]

@@MrJC1 it shows that something is doing something and hasn't just froze, without it you could be thinking "has it crashed"

[@DeanWilliams1987]

@@esenel92 agree this is exactly why a lot of things don't get progress because it's not possible to figure out. I wouldn't say it's lazy developers though...

[@gashnal]

Sir, we used to call it "microsoft time" and was a bit of dry humor when clients would ask how long things would take. "Its Microsoft time, it could take 10 minutes, 4 days, or 30 seconds." I am glad your doing these. It shines light on things i experianced learning technology as a kid and why things are they way they was.

[@tharagz08]

Oh boy, wait till you became an Azure or Exchange Online administrator. The saying "Microsoft Time" is alive and well

[@adammanning5250]

We called it Microsoft Minutes so it sounded like a deprecated Office program. Amazing video! Many admins can learn a thing or two from content like this! Reminded me of when I realized copy/paste is most of the simplified logic behind anything you do on a computer

[@londonmeantime7123]

In a parallel universe we did exactly the same😂😂😂

[@GamingXperience]

And at larger timescales its called valve time ... xD

[@photoniccannon2117]

@@adammanning5250 Yep I was taught “Microsoft minute” as well. Brings me back to my days as a aspiring programmer in childhood.

[@insidetrip101]

Maybe I'm weird, but I've actually never found it odd that the estimated time bounced around. It always made sense to me that the estimated time is precisely what it says--an estimate. Given that it's using the current transfer speed, its probably mainly using that to estimate the time, so if I see the transfer speed go down, I should see the estimated time go up--which is exactly what we see. So I never really understood how people can complain about this sort of thing, its always seemed so obvious to me.

[@ernestnatiello]

The problem is that it's laughably useless, and may as well not exist.

[@De-M-oN]

@@ernestnatiello Its useful when the speed is relative constant. It can be constant on a large .rar file, or slower USB Sticks etc.

[@EvilTim1911]

@@ernestnatiello In my experience it's been pretty useful. Even if there's a ±20% margin of error that's still an okay rough estimate for a lot of use cases, at least for me. I'd much rather have a large transfer say "45 minutes remaining" and be off by 10 minutes than not having an estimate at all

[@PaLaS0]

@@ernestnatiello everyone crying about it not existing on epic games launcher so

[@XDSDDLord]

I'm in the same boat. In fact, I'm going to go one step further and say that it is so predictably bad; as a human, I was able to spot a pattern, which made it far more helpful. Generally speaking, I know what I'm copying, I know when I'm copying things with a lot of small files versus a lot of big files, and knowing that and what other io operations I am running, along with the capabilities of my hardware, I can use the first few bad estimations to see what the system thinks it will take for any given file type and then extrapolate a more accurate ETA. Over the years, I've become so confident I use this system to determine what AFK tasks I can do while it is copying without going so long the computer sits idle but not finishing quickly enough that I have to sit there and wait. This is important because I'll often copy large directories in batches of similar file types and sizes. After all, that is more efficient than the system switching between large and small files.

[@adamsfusion]

I like to say that "It's much easier to design a specialty product than a general purpose one. Sure we might have tighter requirements, but we get a lot less phone calls." When you have to support literally every possible workload, you have to throw all assumptions out the door. Things like the progress dialog may not be perfect, and sometimes it's barely good, but in most cases it's good enough.

[@kabochaVA]

Thew same goes with development estimates... Project Manager: "How long will it take to develop an accurate Progress Dialog?" Developer: "About 3 days... oh no, 4 weeks... or maybe 3 months... I mean, 2 weeks... no, 13 years... or 3 weeks... ok done"

[@pierrec1590]

Do you want it to be accurate, quick or affordable? Can't have both.

[@johnshaw6702]

That reminds me why I don't like making estimates on things I have not done before. If I've done something similar, the answer is how long I think it should take x 3. That is usually accurate for me, I can't be so sure when someone else is doing the work. For something new, you would first have to do some research. Of course now they want to know how long that will take too. Every estimate is ultimately just an educated guess. The bean counters want to save money in the short term, while seeming to ignore the long term cost of not doing it right in the first place.

[@kencollins1186]

@@pierrec1590 I presume you meant, "you can have two, but not all three."

[@RiversJ]

Additionally if you want the first option, it won't be quick nor affordable usually

[@pierrec1590]

@@kencollins1186 In this context, I meant that you may (if lucky) get only one of the three, forfeiting two elements of the set. "Both" refers to the two that will have to be dispensed of. ;-)

[@JamesPotts]

It was brilliant when they added the graph of throughput. With this "explanation" for the estimate changes added, it's much easier to forgive the estimate. Instead of griping about the estimate, we gripe about the throughput. 😉

[@Roxor128]

I wish that was an option back when I was still on Windows (last version was 7). Now I want it on Linux. I wonder if anyone has implemented it yet?

[@taiiat0]

yeah it's a pretty nice addition. more useful than the past types of strictly text Data that was provided. one of those things that you don't know you want until after you have it.

[@Jimmy_Jones]

@@Roxor128 Yeah. Not having that graph was a real hold back for me making Linux my main OS. I got over it in the end but it really bugged me.

[@Roxor128]

@@Jimmy_Jones I only found out that graph existed when I watched this video, but damn do I think it's a good idea now that I've seen it. I'm all for giving the user feedback on what the computer is doing. Especially if it can help figure out if something has gone wrong, and if so, what.

[@Hackanhacker]

feedback

[@jobbld]

Dave: Presents dozens of cogent solid reasons why this is a hard problem to approach and an impossible one to solve based on experience and understanding. Describes many approaches and caveats to them all. Does so in a perfectly comprehensible simple manner. Some comments: "Yeah, but if you add it up and divide then all good right?" *sigh*

[@yellowrose0910]

Very true, but this shows this problem is indicative of bad user interface design. If the time can't be reasonably predicted, even one-sidedly (ie always under estimate or always over estimate), then don't show it: it just annoys people and (back when it mattered) makes people think your system is crap. Design it differently, like maybe show only the % of files copied (with a fidget spinner -- I know, an entirely separate problem -- so you know it's still working), or a files copied / total files, or something you can reasonably predict, so your system doesn't look like crap. Or maybe take a little room from all that personal data you're collecting and keep a history or hash of previous copies to that (type of) drive so you can predict better. Or something that will misdirect or redirect people and keep them happy and impressed with your system. Not the programmers' direct fault; overpaid management should have hired design engineers. And if the OS is "really bad" with serial multiple small files then fix the OS.

[@chaosxepsilon6833]

Is putting two times a solution? One is the shortest possible time and the other is the longest probable time.

[@cube2fox]

He doesn't mention any reason why adding up and dividing wouldn't work better.

[@Drummerx04]

​@@yellowrose0910your closing comment: you think a bunch of small file transfers taking a long time is indicative of bad OS design? File systems are not operating systems, but maybe consider that if this was an easy problem to solve (correctly and consistently storing millions of independently created, deleted, resized or modified files which are accessible to hundreds of simultaneously operating applications that can be owned by multiple users and attacked by malicious actors)... it would have been solved in the last 50 years of computing. File copies are way more involved then you probably think especially when you look at features offered by Windows during or after a copy. Did you know you can cancel a copy halfway through? What about a Ctrl-Z after the copy or delete is finished? If you you have a more efficient method for turning a block device into a usable and consistent reliable file system, Microsoft will happily pay out the ass for your file system which outperforms theirs by a wide margin.

[@Drummerx04]

​@@cube2foxDave mentioned many reasons that a simple sliding average is unlikely to be accurate. In addition to some of the challenges I mentioned to the other guy, file copies can also go over a network with packet losses, other applications can access the disk, while writing you may run into issues where the underlying file system needed to move files around to make room. An HDD or SSD isnt like a cup you pour water into until it fills up. It's more like a giant sheet of graph paper where you need to manually keep an index and table of contents while writing down millions of independent sets of information. If someone asks you to copy 22000 entries while a hundred other people are requesting information and changes... your time estimate is probably going to be a little off.

[@punboleh7081]

I've seen the XKCD comic back then, and it's surreal to years later see the abstract person come to life..... I always assumed there was a team of people working at, well, file explorer, not that the progress bar was actually done by one person as in the comic, and especially not that this person would start producing videos about their work at Microsoft. I can't wait to see more :)

[@SirHackaL0t.]

The one thing that always gets me is why do I need an estimate of how many files I’m deleting from a folder? I just want to delete the files, which the cmd prompt does. Windows Explorer on the other hand, wants to find out how many files and how large they are first which slows up the operation for large folders massively.

[@ncot_tech]

"Discovering files..." 😴 I think it does that so it can show a progress bar. It needs to know what to delete to tell you how much it has deleted. Clearly someone decided UI was more important than marking a load of directory entries as unallocated. Especially when sometimes it says "recycling" and not "deleting" by mistake, making you think it's moving the files into the recycle bin.

[@GeorgeFoot]

It's like when you're decluttering your garage and insist on looking at all the things you're getting rid of instead of just throwing them all away.

[@MikelNaUsaCom]

use the command line.

[@drozcompany4132]

Right? By the time it's discovered everything, the command line is already halfway done...

[@DrHarryT]

@@graealex Open a command prompt in the directory you want to clean out... del *.* then cd.. then rd "directory name"

[@AshleyJColeman]

I'd love to hear a story about your most difficult to diagnose problem that you came across during your career. I'm sure every software developer has atleast one horror story about the bug that would never go away.

[@DavesGarage]

I've had a couple! But it'd be hard to make a store out of them, unless something comes to me!

[@Spero_Hawk]

@@DavesGarage if you can remember any of them well enough and have some of the code then I'd like to see your thought process as you go through the solution. I think it'd be neat if you could come up with a tricky bug yourself as well, maybe even challenge the viewers to see if they can think of a solution.

[@nottheengineer4957]

Take a look at the task manager series, there's a nice story about a very weird bug in there.

[@charleshines2506]

Dave does have a lot of interesting stories most of which don't seem like a horror story. Even for people like me who don't know the first thing about writing a program, it is interesting to hear the things he says.

[@glasser2819]

@@charleshines2506 very true! Dave's mind is trained to present knowledge in many different languages from x86 mnemonics to English. He can build jokes just as easily as code trees. I can squash overhead to boost all I/O's by 25% Internally transfers are mixed batches of smaller packets. Fix that and core bottlenecks disappear! Old-hands know where underdesigned kernel code is... opportunities for radical improvements available ! Our weaknesses give us some of our strength. 👍

[@AdamEmond]

XKCD: Makes a Joke Dave: And I took that personally

[@AdamEmond]

Jokes aside, really interesting insight! Subbed! Looking forward to more (& exploring the back catalogue!)

[@imjustsomeguy]

I prefer seeing the transfer speed than the overall estimated time. Since Windows 8, I love how the Windows progress dialog shows a graph of the transfer speed in real time. Last year, I bought my first Mac and its progress dialog only shows the estimated time :/

[@degru4130]

Heh, better than ChromeOS where there is ONLY an estimated time, no progress indication at all.

[@timwhitman]

The more I hear Dave recount his tales at Microsoft, the more I realize that everything I do in my job as an IT tech over the last 20 years has been Bless or Cursed by his mighty hand... Thank you!

[@sarkybugger5009]

It's people like Dave that drove me to Linux, so he has done something good. I'm surprised he admits to this shit. I'd be in hiding. ;o)

[@liesdamnlies3372]

@@sarkybugger5009 Meh. He’s a smart guy. Even as a Linux user (btw) I’m here because he knows his shit.

[@remasteredretropcgames3312]

@@sarkybugger5009 Lol this comment.

[@robnauticus]

From the same background. I love hearing this stuff, at least it gives me a reason behind the years of chaos dealing with Windows lol.

[@OmniscientWarrior]

It has been blursed by Dave; among many others.

[@coorbin]

In my experience, most of this is due to the unpredictable nature of the computational complexity of on-access virus scanners. On Linux, without an OAVS running, the predictions made by GNOME or KDE file copy status dialogs are pretty accurate.

[@Atixtasy]

EVERY time I do any of these large scale file transfers / writes, I disable the motherf&*ker! WAY too much overhead lol

[@Roxor128]

@@Atixtasy And shouldn't it have already scanned the files, anyway? You know, when they were first put onto the computer? Or if this is getting them on for the first time, to wait until the copy operation is done and _then_ scan them? I mean, when in the process of copying is a virus going to get the opportunity to run and infect the machine?

[@mk72v2oq]

@@Roxor128 no. At least for Windows Defender. When you are copying large amounts of small files, you can see how its process drains CPU significantly. Also disabing the defender speeds up such file operations a lot.

[@liesdamnlies3372]

@@Roxor128 Just because it was clean when it came in doesn’t mean it’s clean going-out, in their defense. Scanning them after moving them also doubles how many IOPS that drive will be doing; for SSDs in particular this would just be patently irresponsible, since these reads can have a marginal effect on drive lifetime (writes are the real killer). And you don’t know that drive will still be there once you’re done with the copy, or that you’ll even have access to the files. It’s a weird wacky world and I can see the logic for doing the scan as you’re doing the copy, even if I have to thank my stars I got a 5600X that can actually handle it. One shouldn’t need that much horsepower for this but…welp.

[@sasas845]

Yeah, the problem isn't with the algorithm of the time estimation itself; the core underlying problem is that Windows slows down *that much* (> 3 orders of magnitude on the shown system) when encountering small files, which craps over any attempt to predict future speed.

[@theftking]

I don't know what that fire thing is in the background, but it's awesome.

[@clementeguillen3767]

I love that you share your experiences and anecdotes as a participant and witness to history. I was the designer of the COMDEX, Networld+Interop and CES conventions (trade show look and feel) and I feel like I have so many great stories and nobody to tell them to. The way you present your stories is fascinating and doesn't come across as vane at all. You are a great story teller Dave. Thank you for doing this.

[@JaleGameSharing]

Dealing with the shell copying small files over the past 20 years has always been painful. Over networking, it seems to be much more painful. When possible and dealing with many small files over networks, I compress locally, copy, and then decompress at destination.

[@jakobstengard3672]

I guess io operations on network drives involves opening a tcp connection for each file. Or, i hope not, but i suspect that can be the case.

[@Alexagrigorieff]

@@jakobstengard3672 It doesn't, because an SMB connection stays for long time, but it still involves a few TCP roundtrips for each file, which could really have been done in parallel, instead.

[@RainBoxRed]

Sounds like a good candidate for a file transfer protocol to do automatically…

[@MrHerbalite]

Same here. However once I've had a bizarre experience. After buying a new NAS, that finally also had 100MB/s network transfer rate I wanted to transfer some large amount of files that couldn't be compressed or put into an archive. Still it should have taken a few minutes, at most, to transfer the data. That was how it was with the previous NAS, who only had a 10MB/s network adapter, but everything else the same (including the OS on the NAS). The transfer took however hours. Much slower than before. I wondered if that new NAS had some hardware problems. Before I went back to the shop however, I investigated the router. It was one which only one port had 100 MB/s the other ports were all still 10MB/s. Putting PC and the NAS on a 10 MB/s port, things speed up, but still too slow for my taste. So I decided it's time to buy a new shiny fast router, before bothering the shop to check out that NAS that I just bought.. After changing to the new router the results were how it should have been. Turns out the Linux Samba and the Windows network had some issues working together when on different port speeds. I could see that during the very first test with the new NAS. For a huge file it took ~10 seconds to negotiate, and then the copying part was fast, small files also about ~10 seconds negotiationg and no measurable time to transfer data. Those ~10 seconds were also noticable when I accessed the NAS web server through a browser from the PC. I learned a lesson that day. Never mix network speeds if it can be avoided.

[@Vaelosh466]

When available I've found robocopy with the multithread option is a pretty good way to copy a lot of small files.

[@IanBPPK]

I'm feeling that LTT will reference this in the near future for some content.

[@DavesGarage]

I'm flattered, but would appreciate a mention if they do :-)

[@IanBPPK]

@@DavesGarage They're good at crediting info sources, especially if they can tie it do another yt channel. I definitely see this popping up on TechQuickie or the like.

[@dunxy]

@@DavesGarage You sure do deserve a shout out, that's for sure.

[@gotbordercollies]

Your skill for explaining things is amazing! Thank you

[@luisdlcz]

I just recently discovered this wonderful person. I am absolutely in love with all you have posted (that I have been able to see so far). I loved the miniature chairs at the end of some videos. I love your video introductions. You are so delightful to hear. Thank you so much for deciding to share and making all these videos. Thank you for your book.

[@nkronert]

Great explanation Dave. Next question I'd love to hear the answer to would be why Windows still can only mention that "this file has been locked by another process", not which bleeding process it is that holds the lock. The OS should know about these things, right? Oh and thanks Microsoft for finally having options in newer versions of Windows to retry or skip errors while copying huge trees of files. It used to be hair-pullingly frustrating to have an abort after hours of copying and having to figure out which files were still missing on the destination drive.

[@craigmjackson]

One of my biggest beefs with Windows. On Windows, it will refuse to allow even an Administrator to remove a file even if it is only opened for *reading*. Very frustrating and sometimes only solved with a reboot or a killing of the process that is using that file (maybe windows explorer), if you can find it, and you can usually only find out which process is using that file with 3rd party tools such as Process Explorer (good tool by the way). In Linux, if you are a user with rights to a file, you can do anything you have rights to on that file at any time. Even if the file is open by another process, your program may be able to detect that the file is open, but if you choose to blow it away, it won't stop you.

[@ItsGravix]

yess

[@Acorn_Anomaly]

@@craigmjackson API design limitation prevents open files from being deleted. In particular, drivers can ask for the path of any open file handle. What happens if the file is gone? How many things would break if they change something like that from "always returns a valid path" to "sometimes returns nothing"?

[@debug9424]

@@craigmjackson linux (and unix in general) is able to remove opened files because it actually keeps the files alive until the last program using one releases it. The implementation is a tad complex to explain, but the general idea is that a file is actually stored as an "inode", which is referenced by the filesystem's directory tree. This allows the kernel to give a program a file descriptor that points to that inode. There's even stories of old timeshare unix systems where an accidental deletion of the entire contents of the system program partition could be recovered from because some users still had critical programs like shells and archive tools running The on Linux the opened file descriptors of all running programs can be directed accessed via '/proc/[pid]/fd/'

[@craigmjackson]

@@Acorn_Anomaly or how about the application developer does its own due diligence? Take a program like Notepad++, it gets around this annoying Windows behavior by releasing the file handle right after opening, and keeping its own working copy in memory or temp space. The state of a file handle might be different later, and that should be ok. The application should be able to handle it. You shouldn't need to depend on a file handle to be locked just to do normal work. It's removing user choice to manipulate objects that the user has the right to manipulate. I agree it's how the API is designed but I'm stating that I hate it.

[@DaveSomething]

I'm sorry, Dave... I'm afraid I can't do that.

[@tylerg6241]

Thanks another great video. I love hearing your thorough explanations. Please keep up the good work you have a beautiful mind

[@shannon208]

Wishing you an yours Happy Holidays and a prosperous 2022. Thank you for this years content!

[@thepenguin9]

"this might seem trivial at first" I had the same thought when presented with the halting problem

[@lucidmoses]

This ought to be good. I always thought it was task based and not time based as so many people seem to think. Guess we'll find out in about 7 hours.

[@alvallac2171]

*ought

[@longboardfella5306]

No wait…1 hour…hang on…16min….done

[@Merescat]

Great show! I kinda knew how this goes even back in the old DOS days. The "overhead" is the (approx) same with big or small files, but so many small files you lose out on the speed for the total transfer (copy) time. Thanks for showing the examples!

[@K-Kil]

I have wanted to hear about this for years. I always knew that it was trying to make an accurate up to date prediction from something, but wanted to know more. These videos are awesome, please keep em coming.

[@OmniscientWarrior]

This reminds me of the fury I feel when watching a Windows progress bar. "Almost done and... wtf is it doing back at one?!" I so hated how Microsoft put that in. They saw how so many people liked a progress bar letting them know how far along an installation was but then completely ignored that it was showing for the entire process and decided that they will use it for each singular part of the installation. The only good that came from that, which would have happened in time, we got 2 progress bars, one for the totality and another for the parts (and that helps quite a bit when troubleshooting when something couldn't complete).

[@harmarize]

I liked the companies better that have dual progress bar, an overall progress bar, and another one for the current process

[@WMDeception]

I have spent a measurable amount of my life staring at this progress bar. Now I feel even more connected to it.

[@jordanbarnsley2438]

Thanks for the video Dave... Always really enjoy your videos!

[@marksmith6259]

Basically answered 2 questions here, love it. it also can be fairly accurate with many files of the same size. Thanks for the breakdown.

[@ccricers]

This is how I found out it’s sometimes quicker to zip or Rar a folder and move that compressed file, instead of moving the folder. It’s not about reducing space, but about tracking and moving one big file faster than lots of little ones.

[@lekhakaananta5864]

How does this work? I mean that if you count the total time spent compressing and decompressing, shouldn't it still be longer? My understanding is that without compression, you're reading from disk A and writing to disk B. With your method, you read from disk A, writing to a compressed file on A, then reading the compressed file on A to write to B, then reading from the compressed file on B and writing decompressed version to B. If you only look at "copying a compressed file from A to B" then yeah for sure it's faster. But what about the total time with compression and decompression on either end? I don't see how it's faster. In fact if you read from and write to the same disk it could really decimate your throughput.

[@lekhakaananta5864]

But I'm asking because I haven't actually tested this or done any research so if I missed something please enlighten me.

[@christopherfortney2544]

@@lekhakaananta5864 @7:28 dave explains why. Once the file is in a target location decompressing it there the basic structure is already there it has an exact drive to copy to in the directory/filesytem etc most decompression etc comes down to your processor as well. Most good modern CPU's fly through these tasks since they have a ton of cache.

[@lekhakaananta5864]

​@@christopherfortney2544 I don't think that quite answers my question. I already understand why one large file is faster to copy, but that doesn't logically make it so that (compressing + copying + decompressing) is faster than only (copying). It would be faster if decompressing is somehow faster than simply writing files. How does decompression skip the filesystem overhead? Regardless of the decompression algorithm dependent on CPU speed, in the end it still results in writing files on a filesystem, so I'm curious how this would be the case.

[@BakrAli10]

@@lekhakaananta5864 Did you find out anything new about it? I'm interested to see if it's actually faster to zip then copy than to just copy.

[@AusSkiller]

I'm looking forward to this one. I've always thought you could probably approximate a file overhead cost and data transfer speed and using a combination of both to get a reasonable approximation of time remaining by looking at the meta-data for the remaining files, the time to copy anything under 100kb could be used for approximating the file overhead which is multiplied by the number of files, and anything over 100kb is used for approximating transfer speed, then it is just ((totalSize - (100kb * fileCount)) / transferSpeed) + (fileOverheadTime * fileCount). Another alternative is to order the remaining files from largest to smallest and alternating between copying the largest file and the smallest file so the average speed is somewhat more consistent and will get more accurate as it gets closer to mean file-sizes.

[@seb_gibbs]

when I do a large copy, it says it calculating before the copy starts, and I assume its getting a file list. I would have thought at the point it can decide precisely what order, and would know almost exactly the time it would take by previewing the file sizes first.

[@AusSkiller]

​@@seb_gibbs That's what I would have thought too, but then you watch it proceed to copy files in a way that it can't better predict the time it would take and seemingly only use the data rate from the last few seconds to calculate the time remaining. However credit where credit is due at least has never been as wrong as Steam, one time Steam thought I was downloading a game at 8 exabytes per second... although one time I was copying a 155MB file at 4.34MB per second and windows gave me an estimate of "About 44627 Days and 10 hours" so maybe credit isn't due 🤔

[@TravisFabel]

How much time does this "improving estimation" scheme cost in transfer time?

[@AusSkiller]

@@TravisFabel compared to what windows does now maybe a few milliseconds on slow hardware. Though if you order the files intelligently you might actually be able to speed it up instead.

[@brianpearce6859]

It really does seem like the prediction could be done better than it currently is, and without even a very complex algorithm. 1. Collect some data points of (file size) -> (transfer rate). 2. Use the collected information to predict the transfer rate of any arbitrary file size. 3. PERSISTENTLY STORE the collected information for later use. Keep a separate dataset for each pair of source volume and destination volume. Of course I'm sure every developer who ever tried to tackle this problem thought it would be easy at first! Like, what if you start up a second simultaneous stream of file copying? Or a third? How do you make sure the predictions get updated in a timely manner when the achievable transfer rates change unpredictably?

[@Jiyoon02]

It is so fascinating to hear piece of thought from the og dev(one of the og devs) of the progress dialog himself. Amazing video and amazing channel. I instantly subbed!!

[@hanswichmann5047]

Wow!! I actually understood most of what you were telling me & I appreciate it. To us non-coder types, just to get a feel of the challenges involved is illuminating. As always, Thankx Dave......

[@MrCichocki]

For years when needing to transfer a large amount of files (In the thousands), I've found archiving them into one large file then copying this over to the destination is so much faster than a straight forward copy operation. Even with the additional step of archiving, it's faster.

[@mydogzty]

I have seen archiving in the Winrar program, but what does it do exactly? Im into computers but this term Im a little clueless

[@mydogzty]

@@StrawberryKitten alrighty then cool thnak you.

[@narobii9815]

So its like putting it all in a zip file, moving that and than unzipping? Sounds a lot like a common shipping problem of parcel vs cargo container.

[@MrCichocki]

Yes, put everything in a single zip file then copy this over to your USB drive. It's not as bad these days with ssd external USB drives but try it with a USB drive from about 8 years ago.

[@charleshines2506]

Very true. Even for files that don't take up a lot of space despite having a lot of them.

[@x7heDeviLx]

As a 42 year old who has used windows since 3.0 I really enjoy Dave's content and the peeks into the creation and history of windows and all its features, for better or worse lol. Thanks Dave

[@lazartomic5800]

Me to

[@Psythik]

Nice; I'm 34 and have been a daily Windows user since 3.1!

[@koller]

Absolutely fascinating. Love your videos!

[@hulkprime]

very interesting! I really like the way you explain things, much appreciated!

[@stephensmith6660]

I just discovered this gentleman, and I love his stories, technical stuff, etc. But I would love to hear some of the more humorous stories that must have happened during the old and early days of Microsoft.

[@rahil320]

Dear Dave. Your videos are awesome! Being able to get an insight about how Windows was developed during its early days is a rewarding experience. Being a programmer myself it still fascinates me that how you guys at Redmond developed such a huge and complex software which runs on so many computers throughout the world. Can you pls make up a regarding what kind of development tools you used for the development of Windows. Did you guys used Borland softwares at Microsoft?

[@DanWorksTV]

every video gets even better. I'm hooked. please keep on doing things.

[@stoef]

Although I already knew most of the things you tought today, I still really enjoy your storytelling. Gread video!

[@j777]

This is also what makes it so hard to estimate the effort of many engineering/software projects : it can be difficult to forsee all the difficulties of solving what might seem at first a straightforward problem.

[@charliecharliewhiskey9403]

This guy seems to have had a hand in all of the most memorable parts of Windows to me.

[@just_jimmy]

I loved the explanation and analogy on this one. As always, great content.

[@VoidloniXaarii]

Spent a surprisingly long amount of time wondering about this over the years. Thank you

[@furzkram]

You could have explained the overhead involved in a single file copy in relation to the file size. Like updating directories and such (as in case of a power off or disconnect drive incident), you don't want to be left with corrupt files or file systems. If the time that needs to spent on this is half or more of the actual data transfer, like it is with very small files, and you have a huge pile of these, the absolute transfer speed will go down the drain, whereas it won't matter much when the file to be transferred is huge, because the time spent on the actions other than the mere transfer of file content will not increase with the size of the file.

[@only1gameguru]

From a full time Linux user: Dave is the best part of Windows

[@jw200]

Yeah he is good. I wish there was more ex MS coders to talk about this all.

[@mattbreef]

Yes, agreed, Dave is the best part of Windows. I think most of us Linux users just want to know why (insert silly/dumb/ridiculous/weird thing here) was done. We appreciate a good hack when we see one and like to see behind the scenes when we want to.

[@johnshaw6702]

@@mattbreef I found out some 20 years ago that the best book on the Widows system memory management, that I ever read, was written by someone who didn't even work for MS, but that it was required reading at MS. Many people today who write software don't even know how it works under the cover, so they are often lost when something unexpected happens. Not that knowing always helps much, but at least you know were to look.

[@AusSkiller]

@@mattbreef As a Windows user I also want to know why MS does those dumb things, those dumb things are why I'm really looking forward to SteamOS pushing some Linux distros towards a much more user friendly experience and finally being able to dump Windows without having to spend half my time figuring out how to fix the latest thing Linux devs broke. Though from watching the LTT Linux challenge it looks like that is already a much less common occurrence than when I last tried Linux.

[@eadweard.]

@@AusSkiller Yes, there's a user friendly distro coming out any day now - and there has been since about 1996.

[@lukeemery7066]

I love when I get the notification that a new Dave's Garage video is online! Always fascinating!

[@JoePlett]

Thanks Dave ....and Randall. 😁 Highlight of my day (- so far).

[@paulstubbs7678]

The problem with the progress dialogue has never really bothered me as I can appreciate the complexities, however the bit that bugs me is when everything all but grinds to a halt, no disk access etc, then after a while gets going again, what on earth just happened! - why no feedback on why things took a nap for a moment.

[@TishSerg]

Made 99% in a couple of seconds and wait multiple time of that for the 100%?

[@null1023]

I always wished the system would identify how many files in a given size range there are during a copy, and base its prediction on that. It's already going through and adding up sizes, so it (ostensibly) is already using this information.

[@rcronk]

Thanks for the video! I wrote the progress and estimate dialog for a disk imaging application back around 1999 and I can confirm that this is not a simple problem to solve. Some of the things I learned were that when we got timing wrong, it was often because we were measuring the wrong work being done (bytes transferred instead of "work" being done, like heads moving, overhead, etc.). I also learned that we missed measuring some of the work at the beginning and ending of the job - like setup, warming up caches, and flushing data to disk. It's really annoying to watch a progress bar rush to 99% and then sit there for 30 seconds while it flushes all that data to disk or some some other "work" that's not being tracked, so I added estimates of setup and teardown work that scaled with the overall job time. I also smoothed the data unless the data deviated outside a certain range so that fluctuations in the speed (we would often muticast the data across the network to hundreds or thousands of destinations, which made things fluctuate even more) would make the estimates jump around so much. If it went outside that range, I'd reset the smoothing, assuming that something actually happened to the environment that made it faster or slower so we could recalculate the new speeds instead of keeping the timings smoothed and wrong forever. I could have spent even more time on it to improve it more, but it's one of those priority things - we can spend time making the data transfer itself more robust and properly functioning, or we can spend time giving the customer a few more percentage points of accuracy on the estimate of when the job will be done? Companies usually prioritize the former, which is the correct priority, but the latter is nice too.

[@savoieadam]

Great insights. Thanks for sharing!

[@jordancobb509]

Why in some cases does it take longer to calculate how long it will take to transfer the files than it would take to just transfer the files. It would be great if there were an option to bypass the dialogue altogether and immediately begin the copy process. In DOS when you copy a directory recursively it begins the copy immediately. There is no "calculating".

[@EwanMarshall]

or start copy immediately while totalling up behind the scenes and fill in the details.

[@racvets1]

At 3:03, he hints at the answer... The dialog walks the file tree first to figure out how many of each operation it will do, xcopy just goes head first in. (I was curious too)

[@EwanMarshall]

@@racvets1 sure, but one can be copying while walking, it'll slow down the copy a little potentially, but I've had times when the walk takes a more time than the actual copy.

[@drozcompany4132]

@@EwanMarshall This was probably more of an issue with mechanical drives where reading the directories/mft would cause a seek which slowed down the file copy. With SSD this has minimal impact.

[@DrHarryT]

...And that is your solution.

[@truckerallikatuk]

Since the OS walks the file system, it knows how many files there are of what sizes. Surely you could work out the top 10% biggest, 10% smallest, and a few average files from the middle. A selection of those files, and their transfer rates would allow a pretty good estimate absenting other software getting in the way. For a single file, bytes/sec would be a reasonable way to work it out?

[@grumpyoldman5368]

Bin them into size ranges and keep historical data for how long it takes to transfer each size range, at least for local copies and maybe for network copies to see if the network is consistent or not. or just guess based on IOPs.

[@bknesheim]

Then you have to now the cache size and how many files of a given size there is. Also that there is no windows updated starting because the system "is not in use" when nobody is tapping the keyboard or moving the mouse.

[@seanvogel8067]

You would also have to take into account, when copying to and from different disks, their transfer rates, seek times etc.

[@bingokemski4473]

Kinda similar to what I thought. I would think it would be easily possible to calculate how fast or slow a file type takes a drive, than calculate based on history or server data (because Microsoft spies on you for everything else anyway) how long that file would take, than finally show a detailed graph or data sheet based on colors per file type, depending on if they wish for it to be as accurate as possible, or as fast as possible. Thus, for those who need to know the precise time something takes and don't mind waiting; it can choose to show and do each file type based on order of smallest to largest or largest to smallest resulting in a time estimate when you add all the numbers up based on history and/or server data. I believe the reason windows doesn't organize based on file type for progress bars in file explorer is because the algorithm I presume wants to take the fastest route to transfer files, resulting in a time estimate that is less accurate than your kitchen oven. Oh, also I'm not a programmer or developer, I'm just completely guessing on all of this 😂

[@bknesheim]

@@bingokemski4473 The reason is that there really is no algorithm . It just go by the next file function that give back the next file in the selected file structure. That why old directories can really kill performance on a hard drive because over time files get spread all over the drive.

[@dunxy]

Very interesting video, even if i did have my theory (files sizes, disk saturation and read/write fluctuations) on why it was near impossible to accurately predict transfer times its nice to have somebody (who's vastly more qualified than myself) confirm what i had concluded after many years of file copying and essentially ignoring the estimate! I sure im not alone in looking at the first estimate and looking at the clock and then coming back to finding it not even close to finished or if im lucky long done before the time is up! Subbed up.

[@voca-chan7953]

These videos are oddly relaxing to listen to. I want more Windows stories to hear about!

[@Elite7555]

Would be interesting to hear how different filesystems deal with IO operations like copying, and how that would affect estimations.

[@vertigo1055]

It's a Real Treat to have someone like yourself, who's worked at Microsoft, to be able to explain some of these engineering challenges that happen in the programming and programs created world from Microsoft. It brings highlights not only to Microsoft, they're 1 company, but many other companies that make software that we use everyday. These challenges are taken for granted by myself and others; of course not maliciously but just because it's not in the front of our minds when we're doing each and every operation within Windows. Your insights are very much appreciated and I'd just like to say, Thank You! Cheers! Stay Healthy and Stay Sane!

[@Tawnos_]

The Raymond Chen cutout popping up on the right brought a big smile to my face. The Old New Thing and Hard Code were two blogs I read prior to working at Microsoft that made me think maybe it wouldn't be a bad place to apply, and I still take a day every few months to catch up on TONT :).

[@EvenTheDogAgrees]

The one in Gnome (and by extension Cinnamon, which is what I use) is even funnier. Estimates are pretty stable, probably because it uses something like an exponential weighed average or something along those lines. But it has this funny feature where if you copy stuff, and then while it's underway decide to copy some more (e.g. into a different folder), it will queue it up as a separate copy job in a paused state. Makes sense so far. The funny part is when it gets around to the second copy job and starts estimating. That job has now been sitting in a paused state in the queue for say 20 minutes. The estimate starts out really high because those 20 minutes with zero throughput are also used to calculate the averaged transfer speed on which the estimate is based. That bug has been in there for years, and it's driving me nuts! 😂

[@jonathansturm4163]

🙂I was most amused by the typo: “accrurate”… I rarely saw the file copy time to complete estimate preferring to us xcopy with relevant arguments for most of the last 25 years. These days I use Beyond Compare on Linux and Win. Since there’s a chance I’ll use it on OS-X I purchased a licence that includes that OS too.

[@madmattman5675]

It's always felt exactly like this, trying to explain why copying thousands of small files takes longer than 2 large files to other folks has always been an issue. Thank you for putting it so clearly! ☺️

[@eDoc2020]

Here's one analogy: moving small a stack of papers one at a time takes longer than moving a single large book.

[@AdonikamVirgo]

I recently implemented a similar progress bar prediction engine for our app with TDD, and it is tough to get your head around how to aggregate and blend historical data with current data for a tree of operations. I'm planning to add a "This could take a while" fallback! Kudos to Dave.

[@davidclarke7154]

Enjoying your videos very much!

[@x260houtori]

I've always been very curious about how this was calculated and why it was so bad sometimes and accurate others.

[@StrafeWasTaken]

Dave: *working and making literally everything* Everyone else at Microsoft: ... Dave: Blame me I'm responsible for it all

[@techosarusrex]

was very fascinating keep up the great content

[@Don-kf6sk]

As a system admin I spend many evenings moving huge numbers of files around and watching those crazy time estimates. Did make me smile sometimes. Thanks for your detailed explication!

[@teslainvestah5003]

file copy dialogue ideas for windows 12: 0. GPU-accelerated recurrent neural network predictor, informed with all current drive performance info, voltage fluctuations, and using the microphone to listen for the sounds of human frustration. 1. take the best available estimate, quadruple it, quadruple it, quadruple it, quadruple it, quadruple it, triple it, and then wait exactly that long while displaying a 100% accurate countdown. 2. LNAS-WDp (Let's not and say we did protocol) wherein all moves and copies become soft or hard link edits, and blocks of data are never actually copied until the user tries to eject the drive. 3. Replace all data drives with Microsoft OneDrive access keys, and let all file transfers instantly edit links in OneDrive. Coincidentally, Windows 12 shuts down whenever internet connection is lost. 4: friendly language dialogues only. "This may take a few minutes." 5. eliminate user-generated content and all file operations.

[@EwanMarshall]

2) why not just make a reference and copy when data is actually edited (not useful across filesystems, but this is a legitimate thing we do on zfs and btrfs).

[@T3sl4]

Not bad, but you forgot to put it on the blockchain!!

[@TedHartDavis]

As requested: to me, it seems clear that the files copied to generate the 6:30 estimate just happened to be highly representative of the overall operation's size and nesting distribution. Or, perhaps, it was slightly worse for performance than truly representative but write saturation took effect.

[@maxbarbul]

Thank you for the back story, it was entertaining to watch. I believe, progress bar is mostly about perception and ux than accurate estimate. It still feels like it is possible to make more or less stable prediction after copying has been prepared and analyzed. You should know by that time the spread of file sizes and disks being involved. So, you can adjust estimate based on known overhead. If some heavy disk load comes at the time of copy than it’s a unexpected circumstance (and user might be aware of it or made aware of). Comparing to driving, despite roads you’re going by have limit like 55, 35 miles/h, etc, there is an average speed which is around 25 miles/h so, directions algorithm should assume you’re going this average speed. Anyway, thank you for explaining all the details involved, it is in deed a hard task.

[@EinChris75]

The analogy with that trip was just awesome. So good explained.

[@linuxretrogamer]

Solution: implement a random number generator as a plecebo and just copy at your own pace. Wonder if anyone would notice the difference between the shells guestimates and just throwing some random figures to make it look you're working?

[@petrovskikh]

Dave, is there anything you can come up about when and how Windows 3, 3.1 started using hardware 2D graphics acceleration which boosted sales of specialty graphics cards on business and consumer computers and maybe paved road for the rise of 3D gaming on PC. Maybe history of DirectX if you were involved?

[@itsjustboarsley]

You’ve solidified yourself as my favorite tech KZbinr. Thanks for everything mate!

[@RenderingUser]

Honestly, the concept and design of the windows file dialogue is still far better than any other simple bar Provides info in the most concise graphical way Just need it to be more accurate with the calculations

[@JaspervanStijn]

A reasonable explanation for a mild annoying thing in the Windows OS. Thanks for that! This does prompt me to suggest another topic. Why does Windows search always sucked so much? I mean, it's so dysfunctional. Searching a simple file name within the folder/directory you are currently navigating yield slow results if any! Using a tool like Total Commander with it's built-in search option has been far better for ages. I can't understand why this still is the case. Could you shed some light on that particular subject, Dave? Thanks in advance

[@jeffreyparker9396]

I get that this is a really hard problem. The thing is that when using other utilities or doing similar things in Linux seems to give far more accurate and less chaotic estimates than windows. Free and open source utilities seem better at predicting transfer times than the tools related by a company worth over $400B.

[@phat-kid]

true but apt-get just gives you a percentage with no estimate of time. but yeah this seems like mansplaining from dave.

[@phat-kid]

also linux file systems are better than microsoft filesystems so that was probably one of his obstacles.

[@jeffreyparker9396]

@@phat-kid well, in my experience even on ntfs you can get better time estimates in just about every file copy utility that provides them than the windows file copy. Also in my experience the older ones tend to be better than the new ones. Only thing that I can think of is the transfers have gotten less consistent, or the newer algorithms to estimate are just less accurate, at least on the copies that I have done. Granted I have not done any file copies in windows for quite a while, since I got rid of windows and only run Linux now. That means that my information could be old or for narrow use cases. I just remember the file copy estimates in windows 95 and 98 not jumping around as much as the XP ones and being more accurate and the windows 10 ones jumping around even more than XP and being even less accurate.

[@AJenbo]

@@jeffreyparker9396 From what I have heard a lot of it is down to how Windows fs-filters work. It prevents proper caching, make file operation (not raw I/O) slow and the data is often streamed threw an anti-virus application which can vary the results greatly. This wasn't a thing back in 95 and do things where simpler and more predictable.

[@yug5874]

I was lucky enough to start my career with someone who wanted to share their knowledge. It was in flight simulators. There was the latest stuff. That was not where I learnt any thing however. It was in the old stuff that was hidden away. It was a Reinfusion R2000. And I had a mentor. It was a 24 bit octal machine with paper tape. Built out of NAND gates. Thirty one years later I still have the circuit diagram burnt into my head. We had Core access boxes and an oscilloscope. Memory 19 kilo bytes. It was all you needed. XOR was cool because it saved you an instruction to clear ACC. I would just like to say thank you for sharing your knowledge.

[@MisterSixty]

Thank you! The history is always good to know thus help understand.

[@ioi935]

Dave celebrates cristmas everyday. 😂 I wish I had dad like Dave.

[@invertexyz]

Maybe someone else has suggested it as well, but I feel like the solution could have been improved by turning the time estimate into a progress-based average of each previous estimate. This way the fluctuations aren't as wild and the closer it gets to the end the more accurate the timing becomes.

[@russellcannon9194]

I discovered long ago that copying 1 large file of a given size was much quicker than copying a large number of small files totalling to the same size. Your explanation is the intuitive answer. It is all in the overhead. I was wrong, however, in my having thought that the progress meter was being computed from file counts. I had thought that the percentage complete and time to complete were computed solely from the files done/total files to do. Thank you for clearing that up for me. Cheers, Russ

[@superlogicalman]

thanks for explaining why copying/moving a bunch of small files takes longer than one big file. I use to deal with that a lot and never understood why.

[@matts7327]

That's fascinating. It would be interest if you would discuss what does break the progress dialog. I know if files get big enough it can be faster to transfer them using 3rd party software. Any thoughts?

[@charleshines2506]

I would give Robocopy a try. It comes with Windows 10 and maybe older versions too. It is a command line utility but there may be GUIs for it by now from various people.

[@pjshots]

When I upgrade my media drive, like from 4TB to 8TB, I've seen Windows take hours to calculate the data, before starting the move. I wish a button could be added to start the copy without the total size, etc. I tend to use Rsync instead, which can start straight away.

[@Graham_Wideman]

Are you using rsync on Windows? If so, which rsync program do you have?

[@the_bw_trekkie]

This is why I use teracopy for big boi moves

[@IANSYT]

why would you windows copy when you could just clone the entire drive over and then just expand the partition to fill the disk

[@stanisawwrobel933]

I actually appreciate the loading bar. I salute You Sir!

[@kingtrav]

I'll take an inaccurate progress bar over an infinite spinning circle any day

[@theosib]

If the only thing you're measuring is bytes per second, you're going to get bad results. A slightly better solution would be to predict copy time using one coefficient times total number of disk blocks plus another coefficient times the number of files. These two factors account for copy speed and file system overhead. Then as files move, perform an online regression, perhaps gradient descent, to estimate the coefficients. You can then predict the remaining time from the remaining blocks and remaining file count.

[@DavesGarage]

The video explains why bytes per second is a bad metric... did you post before watching? :-)

[@chronos1157]

@@DavesGarage Someone commenting before they've watched the video/read an article? That's unheard of! lol

[@theosib]

@@DavesGarage Oh I watched it. I guess I just missed something. Anyhow my point is that you can get better predictions when you incorporate more factors into your model. And in this case you can do online regression.

[@diablo.the.cheater]

@@theosib but why do such a complicated model that slows down the operation when you can do a less expensive naivee operation that while not very accurrate, does not slow the coping operation

[@theosib]

@@diablo.the.cheater it's not complicated at all. It's two coefficients for two linear terms plus a constant, and updating them is trivial too. This is millions of times faster than the i/o it's modeling

[@PlanetJeroen]

Not sure man, my progress dialogs on Linux seem to be quite accurate when transferring files, regardless of target.

[@DavesGarage]

Yeah, but I read in comp.sys.amiga.advocacuy that the Amiga is even better than Linux. So why are you using Linux?

[@PlanetJeroen]

@@DavesGarage wait, how is that any excuse for the increasing/random transfer times in windows dialogs?

[@elazarsirota]

I want to know who the he-- decided to write on the Office installation screen "it would only be a minute" So many times I've had a customer breathing on my neck, questioning why is it taking so long? Here! It specifically says "would only be a minute" Thanks for the great channel! Awsome!!!! Another question (maybe you've answered it on a different video I didn't watch yet): Why Windows updates taking so long to integrate to the system, while in linux it is a matter of minute or two max? Thanks again! ❤️

[@trumanhw]

WELCOME ABOARD. I see why Bill or Balmer liked you. Great job explaining that -- and thanks for the XKCD intro. Love that dude. Care to mess around with TrueNAS !? :) I'd LOVE to hear you analyze that some!!!

[@hedonisticzen]

I think the new solve should be on an AI algorithm so it's making estimates based on previous copy jobs.

[@Bobbias]

While that would definitely result in better predictions, it comes with the added requirement of recording all previous copy jobs, which will require hard drive space, IO to look up the data, and processor time to calculate the new prediction. I'd bet that would end up being problematic at best, or downright unworkable at worst.

[@hedonisticzen]

@@Bobbias I could see that especially for smaller jobs. I imagine a good breakpoint could be established. Maybe a toggle in settings that people can turn on if they find it valuable. And while they're at it an option to turn off progress all together for those not wanting to wait for the file crawl process when copying.

[@Bobbias]

@@hedonisticzen I think that's probably just overcomplicating things. I'd appreciate more customization for how windows handles things like that, but I can't see Microsoft adding that much to a system that is both a core part of windows explorer, and "just works" as is.

[@hedonisticzen]

@@Bobbias yeah, probably better to leave to a 3rd party app.

[@beardyface8492]

Along with new minimum system requirements involving 4 cores 8 threads & 64GB of RAM.. err.. I'll pass & take the current best guess ty.

[@ZeoniaSD]

I've never understood why this problem is so hard to solve. Surely, knowing large files will take longer and studying the OS overheads for small files gives relatively accurate measurements - then apply that analysis to files that are coming down the pipe in future. Sure, it has to be a continuous evaluation, but I find it hard to believe that (on an otherwise idle system with static throughput) a System32 copy could not be made more accurate than you demonstrated in this video.

[@darrennew8211]

In part it's because of the kinds of crash-proofing guarantees a system like NTFS provides. Some things have to wait to go to disk before other things can be started. You have to finish flushing out the used-space bitmap that you changed before you can write the directory or MFT entry of the file that lives in that space, etc. So lots of small files need *way* more writes, and writes that await other writes completing, compared to copying one large file. Even the order you copy files in can make a difference.

[@karlchilders5420]

Dave, I was there at Microsoft in bldg 8 for a while when you were there. It's great to see another "old timer" here on KZbin.. Good times.. :D

[@digantakoner]

Great to see such secrets from a Microsoft employee who actually built the product. Your every video is full of knowledge