Bash vs Powershell vs CMD: The Software Drag Racing Series E03

Dave tests the speed of Bash vs Powershell vs CMD, then tours the code for each to show how they differ. Seeing a problem you already understand how to solve in a language you do not yet know how to use is a great way to learn!

[@Sxxov]

i wrote that cmd sieve!! was a super fun side project, honoured to see it being picked apart

[@joshua43214]

We are not worthy :)

[@OverKillPlusOne]

All the worlds a nail when you’ve only got a hammer! :) I’m totally guilty of some really arcane batch scripts back in the day. Thankfully I haven’t needed that in decades, nowadays I abuse sh and bash :)

[@ashlandwithouttheshd]

As someone who has had to write some ugly batch scripts, your script is gorgeous

[@surferdude4487]

I once wrote a prime-number generator in dBase III back in '89. I just left it running when I went home from work for the day. It was as optomised as I could make it, but it was anything but fast.

[@AiOinc1]

@@ashlandwithouttheshd It's actually pretty hideous in the world of batch optimizations!

[@jeremyroberts2782]

It is always a thing of joy to see stuff being done just to prove that it could be done, by something whose creator never even remotely had that task in mind when it was created and carried out by people that truly know how something works. No matter how slow or messy the result appears, it is always a thing of beauty because it exists.

[@JuanFranciscoGarridoRite]

The importance of this series is not the fairness in time comparison. The best thing I learned here is that sometimes you only have one tool near you, it could be fast or slow, but even that cmd simple tool can achieve what you need. Thanks and congratulations to all the contributors.

[@flyingmongoose0]

I had been looking forward to the continuation of this series for so long. I just finished running the github files on my 2020 M1 Mac Mini (like while this premiere happened).

[@logaandm]

CMD calculating primes! Would Turing be surprised? One thing to be possible in theory, quite another to do it it practice. I'm impressed.

[@AiOinc1]

Anything is possible, it's very versatile. Wrote the first version of an emulator for my Homebrew CPU in it!

[@robsku1]

I'm impressed as well, but it was totally not unexpected - in fact, when this thing started, I already expected that there will most definitely be someone who will write an implementation in BrainFuck - I just checked, just in case, and there in fact is a BrainFuck implementation. I mean... Not to belittle the crazy men who write these things in CMD, Bash, etc. etc., but BF kinda always takes the lead on how insane it is to program any "thing X" on language not designed for it. I think Turing wouldn't be surprised - if such thing as microcomputer running something like CMD was introduced to him back in his time, I think he would have marveled how powerful this high-level language of CMD was :DDDD

[@mph8er]

Was really happy to see PowerShell come out on top of this. In a different life, I used to be a hardcore Bash guy. Over a decade ago, I got tasked with taking over a system that was used to automate deployment of training environments that are used to train new employees as well as existing employees and customers on new products and technology. These training environments were to the complexity and scale of production environments, but each one would function as a sandbox for students taking that particular class. The infrastructure as well as the student stations were all built on top of baremetal Windows servers and this project and its scope would completely redefine my expectations of what a scripting language should be capable of. I will likely get some flack for this, but not in Bash's wildest dreams could it accomplish what I was able to accomplish in PowerShell. Some background of what the PowerShell script did... First, it would prompt the user to either select a date/time to deploy or deploy now. Afterwards, it would hit a SQL server and find a list of classes in a table and present that to the person running the script to select from. Once you selected a class to deploy, it would pull the list of available baremetal servers and if the deployment job was for 'now', it would run a join against 'in use' servers to get a list of systems that were currently free. The user would then select which server to deploy the class to, some of which required 3 baremetal servers per student. After selecting the servers, PowerShell would create a background task-scheduler job in Windows Task scheduler for each machine so that it could multitask. It would add the machines via their MAC address to a System Center server and make selections on the System Center server about what OS and featureset each system required. From there, it would connect to the server via an out-of-band (iLO, iDRAC, CIMC, etc) powershell module, feed it a configuration file that would wipe the system, then initiate a PXE boot which would cause the server to get a DHCP address, then connect to System Center to pull the OS image, drivers and config. At each phase, it would monitor the status of the baremetal server and wait until it reached the OS before installing Hyper-V, System Center, and whatever other roles and features it needed via a dynamically generated answer file. After Hyper-V was installed (as most were virtual machine deployments), it would automatically install and configure iSCSI initiator, connect to a third party storage array via its own PowerShell module to clone a template LUN for that class' storage and set Access Control (via IQN and IP) for the desired server to access the cloned LUN. There were also physical switch configs and firewalls that were modified as part of all of this as well. Afterwards iSCSI storage was accessible from the baremetal server, PowerShell would launch a class-specific PowerShell script from the iSCSI disk. This would import virtual machines, create virtual switches, setup NIC teaming, Storage Spaces Direct, Switch-Embedded Teaming or whatever else the class environment needed for that station. There were a number of possible failures you could run into as well; system off, ip not responding, etc that the students might cause from previous classes, so I had to accommodate those via the same script as well as log everything that was happening to a database. Finally, it would create randomized user accounts in Active Directory and grant them access to their respective server, then email the person who deployed the class the credentials for the user stations. In the end, this allowed people who had no technical background to start the process and have a production-scale lab environment for as many students as they needed within 10-15 minutes for any of hundreds of class. It was a metric ton of work and took nearly 3 months before I had the first class deployed and another 3 months to incorporate all the other classes and work out the quirks. This project required interfacing with everything from system center, exchange, active directory, the IPMI device in the server via a third party powershell module, the third party storage array's powershell module, among many, many others to get all of these dominoes to fall just right and man was it an endorphin rush watching everything happen without me so much as touching the system. All of this was 20k+ lines of native PowerShell 5 and it proved to me that Bash is inferior to PowerShell in nearly every way as I don't think I could've accomplished it at all in Bash. Today, we've largely moved onto templated nested VM environments, Ansible, and Python for class deployments. In addition, the vast majority of the equipment and software we have now uses Restful APIs, so much of this could now be accomplished with curl in Bash, but for years, this was the mainstay of our lab deployment infrastructure and it was awesome! Anyway, love your channel and been waiting for this. I'm hoping to see a Python vs PowerShell one in the future as I've become quite comfortable with both of them.

[@gerardmarquinarubio9492]

I just got anxiety thinking about this

[@engelsteinberg593]

What you think about replacing Bash by a Haskell like language.

[@mph8er]

I don't know that much about Haskell, but replacing Bash as a Shell with Python would be the best move in my opinion.

[@mph8er]

@@gerardmarquinarubio9492 I got anxiety scripting it. Ended up pseudo-coding the entire thing initially though, broke down each piece of the puzzle and it went surprisingly smoothly. Honestly the most problems I had was with System Center, the PXE Server and the answer files and our DHCP server needing to support both UEFI and BIOS installs. It wasn't (and still isn't) very well-documented getting UEFI and BIOS working in harmony in a PXE environment and it requires configuration in DHCP to identify the host as BIOS or UEFI, separate images and separate answer files.

[@engelsteinberg593]

@@mph8er But I truthly hate Python.

[@thisnthat3530]

One of the best features of Powershell is being able to reference an arbitrary dll and directly utilise its functions from within the script.

[@artstrutzenberg7197]

...AND having the entire .net framework bolted onto the thing....and being able to pass output as an object instead of raw text--makes input/output chaining a helluva lot simpler.....

[@romanpul]

@@artstrutzenberg7197 and at least partial support for bash commands, which makes it a lot easier to operate when jumping back and forth between windows and linux machines frequently

[@artstrutzenberg7197]

@@romanpul the fun one? Did some .net 6 service installs on a Linux machine. the install scripts ended up being a combination of powershell calling various Linux commands :D

[@gregmark1688]

I remember back in the 80s somebody made a set of batch files for CMD that could do basic math -- it consisted of hundreds of files named 1.bat, 2.bat, 3.bat, etc, in subdirs named plus, minus, etc with sub-subdirs named 1.bat, 2.bat, etc etc. At least, I think that's how it worked. It was one of the kludgiest things I've ever seen, but it did actually work (so to speak).

[@maxmuster7003]

COLOR.BAT is easier to understand on how this is made for as the 1.bat ...and we can use the choice command to press only the number 1 (checking errorlevel in revers order) to jump forward to the call COLOR.BAT command, so we do not need to write the full name of the batch file to call.

[@robsku1]

Oh dear god :D Btw, there's an example of doing object oriented programming with DOS batch files (which automatically also means that it's usable with CMD as well) - it involves some really kludgy stuff that involves creating sub-directorys and batch files at run time and then executing them when required... It actually doesn't need manually creating a set of files like that, the implementation is much smarter, but also extremely kludgy :D

[@wictimovgovonca320]

The kill command is a built-in command in bash so that is probably why they are using it as no process needs to be forked. I believe the Korn shell, or at least some variants of it, has built-in math functions as that would be the primary reason why bash was relatively slow. Kudo's to the guy that tried this in CMD, I wouldn't have even considered that. After I started programming on big iron back in the late 70's, I left programable calculators but that would be an interesting comparison. My last one was the old HP-29C, as long as your program fit within something like 99 instructions you could do some neat things. Wish I still had one, and that it worked (I expect battery and continuous memory would age away).

[@EwanMarshall]

Even if it isn't, OS caching should take care of it if being run multiple times, the date command has to run the formatting code after getting the unix timestamp from the kernel, the kill command is just asking the kernel to lookup that process ID.

[@ovalteen4404]

BASH also has built-in math. $((expr)). It should be possible to calculate a square root by Newton's method, at least.

[@robsku1]

A quick check reveals that ksh has at least following mathematical functions: abs acos acosh asin asinh atan atan2 atanh cbrt ceil copysign cos cosh erf erfc exp exp2 expm1 fabs fpclassify fdim finite floor fma fmax fmin fmod hypot ilogb int isfinite sinf isnan isnormal issubnormal issubordered iszero j0 j1 jn lgamma log log10 log2 logb nearbyint nextafter nexttoward pow remainder rint round scanb signbit sin sinh sqrt tan tanh tgamma trunc y0 y1 yn Also, it supports floating point numbers, but that's not useful for this purpose... But proper sqrt returns floating point number, so you would want to truncate it to integer for purpose of checking if a number is a prime or not. So, hmm, ksh implementation should definitely beat bash - someone has to do one, if someone hasn't already ;p

[@robsku1]

@@ovalteen4404 Yes, but ksh has floating point math support and a whole load of arithmetic functions, including sqrt, so I would think that ksh implementation should be able to leave the bash implementation eating dust :)

[@robsku1]

When I originally read this, I thought it was smart - not because "no process needs to be forked", but because the rules are that you aren't supposed to use calls to external commands, but rely on only what's part of the language - which is why calling `bc` for faster math for example would be a no-no. The use of sleep and kill for timing is a really smart idea, as both are bash-commands. The full ingeniousness really dawned on me right now as I realized that not only calling built-in 'kill' is faster and doesn't violate the rules, but it dawned on me that if bash didn't implement it's own kill (and here's also the reason for _why_ it does implement it - I mean, it doesn't implement commands that already exist by default on every Unix/Linux installation): ```Kill is a shell builtin for two reasons: it allows job IDs to be used instead of process IDs, and allows processes to be killed if the limit on processes that you can create is reached.``` *Calling /usr/bin/kill wouldn't even work!* Because `read` is also a bash built-in, it doesn't have a PID! The value it uses is the job ID of `read` and /usr/bin/kill would have no clue about it, just as it won't be listed by `ps` either. ...however... The implementation in the video is older though - it uses `sleep`, which breaks the rules despite of using `kill` to check if it's running. I checked the current implementation, and it uses this instead: `read -rt "$RUNTIME_SEC"

[@surferdude4487]

It blows my mind that a prime siv in CMD can be written, never mind that it actually works. They could have saved some time with that square root calculation by saving the previous value and checking if it needed to be incrimented from there.

[@AiOinc1]

You can write anything in CMD, I assure you.

[@Sxxov]

great catch. my impl was basically a direct port from bash, so if this applies to the cmd one it could be an optimisation for the bash impl too

[@TAP7a]

@@AiOinc1 I'm sorry that you had to live through whatever situation taught you that. No-one deserves that level of suffering

[@douggale5962]

Yeah, half the fun of implementing really fancy cmd stuff is the surprise and disbelief when people realize you did it in cmd.

[@toby9999]

@@AiOinc1 I can't write anything in CMD. I hate it.

[@AiOinc1]

I can't wait to find out that CMD is the slowest and still write another emulator in it anyways Also, it's immediately clear to me that this guy isn't an experienced batch programmer but someone who happens to know a bit about OOPs that found ss64 and tried to transliterate some existing code. Awfully warped implementation there, I'll have to take a stab at it myself. There IS a FOR command that operates over a range - FOR /L IN (0,1,16) for example. EXIT is also a very slow command to process, so "GOTO :EOF" is usually used in place of EXIT/B Oh yeah, and whitespace is a big deal in batch script performance. Removing that can sometimes increase speed huge amounts and I'm sure in such a calculations heavy situation it would matter a lot.

[@Sxxov]

bang on with the batch & oop experience. i would've used the for loop, but from my previous experience, it was much slower (didn't test it though in this instance, premature optimisation maybe xd). never heard of the `exit /b` Vs `goto :eof` perf difference, i just preferred `exit /b` due to how u can fit it inside macros (set variables that are expanded on the top level) interesting tip as well on whitespace being a perf point, coming from whitespace agnostic languages i'd never thought that could be a problem. a minifier for batch perhaps? finally, do reel me in if you ever decide to reimplement, i'd love to catch the ride (::

[@bristolsdjcubed]

What an amazing channel. Thank you Dave. You are an inspiration!

[@DavesGarage]

Wow, thank you!

[@willd4686]

I think this project is going to last decades. Potentially centuries. It’s a gem!

[@nitepone]

Awesome video! Made me really respect the effort that went into that batch solution.

[@ryancraig2795]

PowerShell is surprisingly powerful with a lot of useful extensions. For handling JSON, for example. Not so popular as UNIX shells and tools, though. Working in plain CMD shell for anything complicated seems like an exercise in masochism to me.

[@benjaminkeebler4218]

I was so happy to see this posted - I thought for sure this series had been dropped. Now I feel like I need to make a ksh version, since that's what I happen to work in on the daily...

[@jimlawson629]

I've been loving these videos. Can't wait till you get around to the lisp implementation!

[@nosenseofhumor1]

this is very entertaining and informative, thanks for doing this dave

[@DavesGarage]

Glad you enjoyed it!

[@Fz3r0_OPs]

I love your videos, thank you for sharing all that knowledge. Greetings from Riviera Maya my friend.

[@alanmusicman3385]

Over a period of years I did oceans of C shell and quite a lot of Bash. However, latterely I needed to do some in-depth script for Windows automation and had to take the plunge into Powershell. Sooooo glad I did, cos it not only does everything the others can do, but (not being an agglomeration of stuff from different people at different times like the *nix commands set is) it is far more intuitive to use and far, far better documented. I've not done anything this mathematical, but for the things I have done, I have found it's performance to be very impressive.

[@just-a-waffle]

As a Microsoft Admin, I use powershell almost daily, and like it a lot (even if I’m not doing anything that advanced with it) There’s so many things in m365, exchange, and windows that cannot be done in the GUI and require powershell

[@technerd9655]

Now that I am thinking about it, it seems a lot of people who went into programming got their starts on C64s and other 8bit systems coding in BASIC. We had a PC in 1984/1985 when I was 2 or 3 years old, it's no wonder I never learned to code as a kid (when I may have had the patience and energy to do so). Sure I found HTML (the one thing I sorta self-taught myself in high school) and Turing (an educational language originally developed at the University of Toronto and was part of the computer science curriculum when I was in high school) somewhat cool, especially the air hockey game I made, but I never enjoyed the process. I remember going to my friends place and seeing their C64 and watching them type in a program, or some convoluted (compared to MS-DOS) command to load and execute a program from tape or disk. To me as a kid that was primitive. Today, as an adult, I wish my friend still had his C64 and would trust me to mess around with the hardware as I'd rather be soldering than coding lol.

[@stevekarvo6299]

Why aren't these things just in the GUI to begin with?

[@technerd9655]

@@stevekarvo6299 agreed!

[@just-a-waffle]

@@stevekarvo6299 the GUI is just using the same APIs and cmdlets that are available in powershell, but have to be programmed by Microsoft, and they just knock out the basics in the gui. A couple good examples: in m365, if you want a new user to be in all the same distribution lists as an existing user, you need to do them one at a time in the GUI, but in powershell you can pull all the groups of the existing user and use a ForEach loop to add the new user to all of them in about 2 seconds. Another example is properties of an object, there’s hundreds of attributes stored about any object in m365, AD, etc. In AD you can use ADSIedit to access all the attributes directly, but if they put everything into m365, the gui would be huge and clunky. With powershell you can easily access and manipulate those hidden attributes.

[@MK-of7qw]

Being able to create new users in AD and even assign them a license in 365 is a nice ability. And terminating users and unassigning licenses.

[@JaseTheAussie]

This is great! I’m a Windows sysadmin, and use PowerShell all the time, and write all my automations in it - I’m not this level though, aspirations :)

[@stevehill5637]

Thank you, Dave. Great info!

[@DavesGarage]

My pleasure!

[@eyadaldwaik3880]

The "new" operator in powershell means its referencing a .Net framework function, so powershell basically can do whatever you do inside a C# project, as long as you stick to .Net framework

[@1992jamo]

Not sure I'd use that description. It's not an operator, It's accessing the constructor which is a static method of the .NET class.

[@MsDuketown]

Whether you phrase it a reference or an operator, you'll end up using .NET. So the question is if you see PowerShell as integrated part of .NET.

[@just-phillip7577]

Hey Dave! As you know I tinkered around with the prime Sieve recently and wrote a program from what I understood. Thanks for mentioning the Bit Array in this video, I picked up on the way the index of the array is used to represent the value of its position, Thanks! That will probably become very useful in many ways. It is very similar to the way I have used an array of characters to convert between binary, octal, decimal, and hexadecimal: [ '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'A', 'B', 'C', 'D', 'E', 'F' ]. Now rather than an array of characters I understand the concept of using a Bit Array in the same way. THANK YOU!!

[@joegee2815]

I started with just the Bourne Shell years ago, toyed with CSH (hated it), then found bash and was enamored. I think I tinkered with zsh, don't remember what I thought but was very happy with bash for many years now since it's default on most Linux distros.

[@colday74]

Nice Ghostbusters reference in the middle there Dave. Good work!

[@mobslicer1529]

batch files sure are weird, but they do actually have tons of cool features

[@maxmuster7003]

I put assembler mnemonics or mashine code in form of hexadecimal values(0-FF) inside of batch files to create executable with a little help from Debug. But it is also possible to use visible ASCII character as binary mashinecode to redirect the bytes into a new executable only with a pipe operator(ECHO some hyroglyph>file.exe), but not all values are visible character. I saw a bash script with an elf file header and a routine for linux too.

[@GamingHelp]

I smacked the like button (AND subbed) again before even watching JUST for your work on NT. But, I commented to say this: As a fellow content creator, I love that you added some of the bad takes at the end. :P I keep intending to do that too, but I'm always so burnt out by the time I'm done a video, I just want to export it and go to bed.

[@mattbox87]

Dave I love how you acknowledge MS has made less-great stuff precisely because they have always been zealously backwards-compatible. I should own up here; I've been an MS hater for the longest time, but I've relaxed on it because 1. you and 2. I really ought to be professionally compatible with what's used in practice. Thanks Dave; thanks for being the reasonable middle man that brought me over. Also: zsh ftw!

[@Rich3rd]

I’m so excited for this one! I’m rooting for powershell

[@lostcarpark]

Fun! I was not surprised that CMD was slowest, and impressed that it could do it at all.

[@chadpomerleau8731]

Great video. Loved the intro!

[@DavesGarage]

Thanks so much!

[@tarantula_live]

Glad to see you that you use ZSH also. It's always my first install on Linux

[@tagKnife]

Wooo the drag race is back!

[@judgewest2000]

i've been doing powershell for i guess 3 years or so... it's almost just c# with different syntax with all those libraries available so it will have the edge in those respects. for you bash folk it's definitely worth a look at with PSCore available on the major OS's

[@jarrettmattson]

The commentary in these videos is gold.

[@the8bitbarn836]

Well that was quite a race! I tried to picture what each of those languages would look like if they were vehicles on the road, and well I have never seen a unicycler being chased by scooter following a golf cart but there you go! BTW I bought your book! and I'm through the 2nd chapter! Great words of thought there! I'm starting to wonder if I'm Autistic as well! That would explain a lot in my life.

[@only1gameguru]

I love the disclaimer at the end. Zsh is the best shell I've used to date. All others I stagger through (even bash now)

[@engelsteinberg593]

Also power shell?

[@only1gameguru]

@@engelsteinberg593 power shell still confuses me. I do very little scripting in windows though

[@DavesGarage]

First! Again!

[@hubster6381]

It's almost like you have an inside track on when these things are going to drop...

[@HaroldKuilman]

Good job, go tell mommy!

[@gryff8400]

😂😂

[@GHHodges]

A rough day for you Dave, but another great episode for us.

[@DavesGarage]

Thanks!

[@mkelly0x20]

For Bash, date would require exec()ing an external command, while kill is a shell built-in, so there should be a measurable difference in runtime between the two.

[@ovalteen4404]

BASH does have the ability to find programs in the current directory. It checks the PATH just like CMD does. If the PATH has "." as an item, that's where it will look. UNIX has long considered that to be a way to shoot yourself in the foot though, so it is not added by default.

[@AlexZanderMuro]

i remember i wrote a "classroom control program" for a high school compsci class that was just a batch file that presented the teacher with a menu asking for their classroom number (the domain had a naming scheme where each pc in a classroom had a matching start to the domain name, eg. room 227 was something like HS227Cxx) to load the next portion, which would call their list into the next stage, which would ask for a computer number in the room, and ask if they wanted to shut them down, send a custom alert (shutdown window with custom message), view all the processes running, or kill a preset list of common games kids were running at the time. it would then go back to the "computer select" screen. i really love taking the longest way to do something

[@EinChris75]

PowerShell is one of the biggest achievements to practical computing. It's clear concept, huge API, and broad availability makes it an extremely useful tool. It's a bit hard to learn, but it enables scripting where no script has run before.

[@JohnnyWednesday]

(laughs in WSH)

[@KvapuJanjalia]

I found PowerShell to be perfect for writing DSLs (Domain-specific languages); also it is very easy to integrate PowerShell SDK into a .NET app - it will be able to execute those DSL scripts without PowerShell runtime installed system-wide.

[@CristiNeagu]

Last script I wrote in CMD was a script that would take in a file name, pass it to a program as an argument, and save the result as the same file name with an extension added. Took me close to an hour to figure out...

[@guilherme5094]

Very cool👍!

[@anon_y_mousse]

Interesting comparison and results, though I'd like to see a comparison with them all using external programs as well. For most general calculations, if I need them specifically at the terminal or in a shell script of some kind I use 'bc', but given the way it processes, I definitely wouldn't speed test with it if you value your time.

[@psyience3213]

When I took a Linux course in college I got into Bach and tcsh scripting for a little. The Linux shells are quite powerful

[@YannMetalhead]

Good video!

[@bobuccman1424]

you could also make it in fish shell, its a great alternative to bash for unix

[@rodneysmith1750]

Thanks, Dave for today's video and remember that "a little ugly isn't all bad..."

[@WillKew]

I am impressed that CMD was possible, but 10 hours? Holy smokes

[@Dyllon2012]

I’ve written a ton of bash and somehow never knew about the $! variable. Very handy.

[@DavesGarage]

Gotta read more Dan Brown Linux guides :-)

[@MsDuketown]

Another one: $0, for native prompt @path.

[@redringofdeathgamer]

You should have shown the output screens.

[@owlstead]

Wow, that sieve in bash took almost as much time as compiling the Java JDK on Windows back in the day (30-45 minutes on Linux on the same machine, vs 10 hours on Windows and one and a half day to prepare the libraries).

[@itmeurdad]

That Dan brown dig earned you my upvote. I've been a member of the arcanum my entire career and made an industry transition and ended up in PowerShell word. I miss my silly little symbols and the Gregorian chants.

[@Fetrovsky]

My friend did something similar to benchmark languages a few years ago, but with a raytracer.

[@nuxar8778]

Hi I’ve recently read a blog introducing paging tables and found it very fascinating. If you know how they work I’d love to know how they’re implemented in windows or a little backstory on them!

[@kelton5020]

I enjoyed the video. It almost feels a little cheap that powershell won since it's using the .net framework for it's math. A few suggestions: It would be nice if there were links to the other drag race videos, as well as links to the related github repos in the description.

[@randomblogger2835]

kill is a bult-in so it'll probably be faster than date which is external, also the test is simpler.

[@Blzut3]

I thought the comment above the sleep stuff would have explained the use of kill adequately, but I suppose it's not entirely obvious why it says subshell instead of subprocess. In order to compare with the date function the output needs to be captured so in effect using kill, which is a bash builtin, avoids the need to fork two processes each iteration. Forking processes is very very slow compared to calling a builtin. Of course Bash only makes it through one iteration in the test conditions, but for smaller sizes it will make a difference. As mentioned at the end calling out to awk, bc, or whatever to do sqrt would be cheating. Especially since awk itself is a language in the competition. I did try using a faster algorithm but it kind of became a wash since accessing variables in bash is quite slow. Given that sqrt isn't called often (I think only once per run?) it wasn't even worth trying to justify optimizing it. I call shenanigans for not using the inlined getBit version of the script though! :P

[@xrafter]

Do you know if declaring a variable with the 'integer' flag will speed this up? declare -i number.

[@Blzut3]

@@xrafter It doesn't. Profiling bash pointed to the actual variable name look up being where most of the time was being spent. It's been awhile so I could be misremembering, but I also don't think Bash does anything different storage wise when the -i flag is used, it's still a string, just a conversion is done when writing to the variable (thus normalizing the value). Also fun related fact associative arrays in bash are faster than indexed ones since associative arrays don't perform expression evaluation of the array index.

[@DavesGarage]

Your comment might have adequately conveyed it to your intellectual peers, but mortals like me sure didn't get it! What's the difference between a subshell and subprocess and why doesn't a subshell have a process? Is it a thread context in the same powershell.exe process, or a fibre, or just a context? You say it all like it's obvious because you already know it, but slow down and explain yourself to us noobs. I was kind enough to go out of my way to give you credit in the episode, so less snark and more explain would be great. If sleep is a builtin vs the /bin/sleep, no one has said that, so I don't get why any one of sleep, kill or date is cheaper than the other. Which one is the builtin?

[@Blzut3]

​@@DavesGarage Outside of the joke that was the very last sentence of the post no snark was intended, but rather was just admitting that I could have been clearer and trying to explain it. My bad if you took it that way. Let me start with the builtin question: sleep and date are not builtins, but kill is. The "help" command in bash will give you a complete list of builtins for your particular version of bash. Builtins avoid the need to fork/exec and are thus extremely cheap to call compared to a normal program. An easy exercise to see how much of a difference there is, time the execution of 1000 calls to /bin/echo vs the builtin echo. It's about 3 orders of magnitude on my system. A subshell is a particular type of subprocess. Specifically it's a term for when bash forks itself and executes two code paths in parallel, so they do indeed have a full process and independent copy of all variables. (To be clear bash does not do multithreading, just multiprocessing.) The most obvious case of this happening is when you capture a command's output with $(...cmd...) (or backticks). The code inside of the parenthesis are executed in a "subshell." There are also less obvious times a subshell is created is if you pipe the output of one command into another. The right hand side is technically executed in a subshell and you can even use braces to process that output over multiple commands. Now if one uses $(date) in a while loop we see that nominally a subshell gets created which in turn would create a process for /usr/bin/date. (In practice, bash will see that date is the last command for that shell and do tail call optimization to skip calling fork.) Ideally we want to measure only the performance of the code being benchmarked (i.e. runSieve), so we'd like to use a builtin command to avoid the expensive fork/exec. There are a few ways this could be accomplished, but using a background timing process (sleep in this case) and checking if it completed via the builtin kill command is probably the fewest lines of code way to do it. This pattern does three things: 1) sleep only needs to be spawned once for the entire execution vs $(date) which would need to be spawned every iteration. 2) We can move the process creation outside of the timed section of code saving a fraction of a second on even the first iteration. 3) Feels good to know that no external programs are used during the actual benchmark, it's pure bash between tStart and tRun. As an aside, it is technically possible to replace sleep with the read builtin. I didn't do this since doing it right would make it even more cryptic, still require spawn a process (a subshell), and is outside of the benchmarked code. If one wanted to get rid of all non-builtins date could probably be replaced by reading something in /proc like /proc/uptime, but then it wouldn't be portable. Hopefully that helps!

[@akin242002]

I started off as a bash/python advocate, but at my current job they use powershell for scripting. Powershell is very underrated and powerful scripting tool. It has won me over. I'm glad this video confirmed that.

[@engelsteinberg593]

But Power shell cannot be used in windows.

[@akin242002]

@@engelsteinberg593 For windows home, no. For Windows Pro or higher tier, yes.

[@engelsteinberg593]

@@akin242002 I mean Linux.

[@addas4]

Hey Dave, I have a question regarding Command Prompt... How modern command prompt accept environment variable although the original DOS doesn't? Thanks!

[@Warrentvoid]

I’ve been coding in PS for many years and can rip out some very functional code quick smart. I wrote a complete electricity generation and retail system in PS. Everything the CLR can do, you can invoke in PS. Integration with SQL Server is ridiculously simple. I am a great fan. I’ve coded in ksh and CMD batch, but these environments can’t hold a candle to PS.

[@johnterpack3940]

I really like this series, even though I haven't caught all the episodes. One thought I had was to see what changes when given a different task. The prime sieve is a great test of pure compute power, but it isn't what I would call an "every day" kind of task. I frequently have to search for duplicate files. I use a PS script that groups files of a certain extension by size (since only files of identical size can be duplicates) then hashes the files in each group to see if they truly are duplicates. There's a ton of directory traversal, file sorting, hashing... but it generally burns through some 16,000 files in just a couple seconds. I suspect this isn't an uncommon task. Would it be faster in a compiled language? Or is the bottleneck in the OS having to grab the files? Not sure how something like that cold be fairly tested though. Not to ramble, just wanted to toss out an idea for a "rematch" of sorts that might play to a different set of strengths.

[@RealYethal]

Could you also include nushell in future comparison videos?

[@asdf8asdf8asdf8asdf]

Would’ve been a really really nice lagniappe to compile the bash and present the results of the compile run. Also I bet a Canadian dollar that even the fastest PS w/o the built-in sqr function would be slower than bash.

[@itachi2011100]

This video drives home the point that while shell scripting languages are powerful enough they're geared towards certain types of tasks and if you aren't doing those tasks you need to re-evaluate your decisions.

[@dotslashsatan]

Trying to fast forward to see bash win but it’s a premier!! Dammit haha

[@xrafter]

powershell math methods helped.

[@dotslashsatan]

@@xrafter I’ve no doubt bash can and will be optimised to win!

[@DanielLiljeberg]

I remember when ported a bunch of subversion commit hooks from BASH to CMD and I had to truncate whitespaces from a string in CMD... OMG was that less intuitive than I excepted it to be :P

[@BrianOrange]

Interested in running the prime sieve on TempleOS and HolyC? That would be an interesting video.

[@rfvtgbzhn]

11:57 It's partially true, however DOS batch files that use special tricks often don't work in cmd.exe.

[@madbradfreeman]

I love my zsh!

[@jamesarber904]

(commenting before watching) I'm an MVP and most of my tools are in PowerShell. I know PowerShell is pretty slow. So I expect it to be last

[@xrafter]

It won.

[@xrafter]

Kill is a shell built-in in bash. Built-ins doesn't fork a new process making it less expensive than date

[@DavesGarage]

OK thanks! I had no idea of that, and people keep saying it like it's obvious. I appreciate the heads up, now I know...

[@lawrencedoliveiro9104]

ldo@theon:~> type kill kill is a shell builtin

[@lohphat]

How are results passed between PS cmdlets? In normal *nix or cmd there's the stdin and stdout char streams, but PS deals with objects and lists, what is the data encapsulation format and how is it passed?

[@j0seant]

What I don't like about Powershell is that it depends on .NET, the system tools in my opinion should have as few dependencies as possible and be native..

[@xrafter]

It also slower for me than both cmd and bash.

[@guspaz]

Since Windows has shipped with some version of .NET out of the box for decades now, can it really be considered a separate dependency or not native? Bash has dependencies too, for example. libc being the big one.

[@xrafter]

@@guspaz And libc is needed by most C programs, and if they want they can just link it statically with bash. But that will increase the size of the executable. Readline library is also something bash needs, and as explained in the wiki : GNU Readline is a software library that provides line-editing and history capabilities for interactive programs with a command-line interface, such as Bash. It is currently maintained by Chet Ramey as part of the GNU Project.

[@guspaz]

@@xrafter You can also build .NET Core executables to be statically linked with the required components of the .NET Framework included in the EXE such that the user doesn't need it to be installed. But, as I said, Windows has had some version of .NET built-in as part of the base install since Windows XP, and PowerShell has been included in the Windows base install since Windows 7, so dependencies are meaningless when it comes as part of the OS.

[@MsDuketown]

yeah, the .NET dependency also drags in the C language relation, clib, as well as sdl. And for legacy reasons I would add VBA on top of the list, in both functional programing and with interpreted languages.

[@AbhinavKulshreshtha]

I didn't knew there are compiler available for bash. Can anyone point one to me? Also for cmd, just a curiousity.

[@cloerenjackson3699]

@Dave's Garage: I was looking for the video when you tell us which language is faster than assembler and does 4000 passes... when?

[@arnabbiswasalsodeep]

Can you add the github link to the description? 6:38 Line 17 has an error! YOu can verify by looking at 7:26 I wonder if the "doom fast inverse square root" type trick could be used to calculate the square roots, or using the basic idea of it & how the math works to implement the function in the scripts for a normal square root (not inverse) whole also not using the inbuilt math function to see if they do parse the loop quicker or slower. This will give an idea about the io head lag cuz of api, pipes, etc and wont be for comparison of languages by comparison of the parsing performance.

[@UncleKennysPlace]

Ten hours! I nearly swallowed my tongue; hilarious! We used to write stupidly huge batch files to gather data from several different machines; I can hardly read them today.

[@RNMSC]

One of the contract jobs I had was to support a collection of batch programs that handled data being provided by vendors for a company. Most of it ran on a fairly early blade server that I don't think would have passed Y2K compliance. (different issue.) I ended up being hied into the company for my networking skills, and left after demonstrating that I didn't have the project management skills to handle a load of 80-100 network projects, but with the batch processes I ended up entirely re-writing the monitoring code in a few hours after accidentally deleting the earlier code, and learned a lot about re-entering batch scripts. and safely exiting. The best part was hearing through the grapevine that by my being contracted in, the support call volume for the collection of programs went from between 5 and 20 calls a day to 1-2 calls a week, and going down. In the first month. There was a reason they wanted to bring me on full time. I was also readily willing to admit that _at the time_ I was not ready to re-write the entire package in any of the compiled languages I knew about. It's possible I could now, but I don't need to, and since it was moving files about, I would probably re-write the entire system, including monitoring, in bash. Well, I might do it in python if I had to write something that would run on both windows and Linux or bsd. I could learn to write it in powershell, and it would probably be faster, but it would be part of a learning experience.

[@Sxxov]

10 hours is on native cmd! all other impls ran inside a linux docker container, meaning you'd have to run docker, run wine, then run cmd *shudder*. it does work, but i never made it to the end to get a time with that config xd

[@wwusirius]

Amazing that they were able to more than double the powershell version! It's trying guys. XD

[@endrawes0]

Dave, is that your dog?! So cute!

[@neiszners]

What is the overall leaderboard? If that is possible quantify...

[@TheSulross]

there's the rationale that when one needs to do task that are more involved, then move out of shell scripting languages and code it in perl - well now we must code in python. but the thing depend on runtimes that are not nearly as tightly tied to the underlying OS as the shell is. There might be version issues to deal with - especially Python's 2.x vs 3.x. And both perl and python have all these library modules that might require a download and install. Sometimes the host is too locked down for that to be possible. For all these reasons I've written bash shell scripts that have gotten a bit complex - much more so than I'd normally be inclined towars. But bash is like this universal given in the land of Linux, MacOS, and pretty much so for other Unixy platforms. Have bash script - will travel

[@MsDuketown]

Python 2 versus Phyton 3, but also Bash 3 versus Bash 4 versus Bash 5, as well as Powershell 5.0 and 6.0. It's all interpreted slightly different, and that's why it's versioned. But yeah, perl involves the web scenario's and should be the ultimate async test. Like a ping. In practice, it's more the workflow art of task management and pipe orchestration.

[@SwervingLemon]

Now I'm curious to know how quickly cmd could be used to simply create a file with the contents of a c++ binary and invoke it.

[@fnjesusfreak]

Pretty much anything bash can do can probably be done the same way or very similar and much fasterr on ksh (especially Korn's own implementation).

[@ch2laughlin]

Its awesome to see this all in popular scripting languages. While I don't expect CMD to ever be speedy I wonder if it could be just 1 or 2 days if we optimized the SQRT function.

[@robsku1]

I didn't expect it to be speedy, not even able to come close to bash, but I didn't expect it to be *that* slow either! Considering what you could run with 4DOS and 4NT already back in the late 90's, while not very demanding, I suspect they would fare much better at running this (although 4DOS wouldn't really - it's a 16-bit DOS application running in real-mode). I'm currently wondering how fast it actually is on real Windows CMD.EXE - the implementation benchmark is designed to be run with Linux and uses Wine to run the CMD implementation. Wine has it's own version of CMD, which is possibly implemented very differently. I don't understand what in the CMD's scripting syntax could possibly make it so hard to interpret faster than the results say, I don't understand - and that makes me wonder if Windows CMD.EXE is as slow? The benchmarks are done under Linux using Wine, and if I'm right and the syntax itself doesn't cause this to expected performance, it would be completely reasonable to assume that the slow speed is because of inferior implementation from Wine's developers - it's not known from being slow, quite the opposite: Wine is reported to run some games better than Windows, though IIRC it's been reported specifically with Direct3D games (Direct3D being Windows only Wine has to actually translate D3D calls to OpenGL - but perhaps because Wine is coded by actual wizards, or more likely because D3D is actually inferior to OpenGL and shouldn't be used, it still seems to run just as good, or even better, than Windows), but overall the consensus is that it's not any worse - but then, CMD.EXE has no role with performance of actual Windows executables, the developers likely focused only on making it compatible and didn't want to waste more time than necessary on it. But I don't have a copy of Windows newer than Win2000 (and it's CMD isn't up to the job, it lacks features that were added much later) - and I wouldn't install it even on a VM just to satisfy my curiosity. I don't have friends who could simply send me a copy of CMD.EXE from their Windows install because they don't use Windows either, and I'm not going to ask for someone on IRC to send me a copy so I could test the real CMD.EXE from Windows with Wine. So I can only hope that maybe someone will see this, or my other post about this, and try the implementation on actual Windows CMD. Even the documentation of the implementation instructs to use Wine, and instructions for Windows guide to installing Ubuntu using WSL and installing Wine on it, but there's nothing preventing anyone from running the .bat file on Windows. Meanwhile I want to check if Take Command (formerly 4NT) can run it - JPSoft says it's compatible with CMD.EXE, but I don't know if they have been able to add all functionality of current CMD.EXE _and_ not have them clash with their own extensions to CMD batch syntax. But Take Command and 4NT also advertice performance increase with regular CMD scripts, so I want to see it's performance - unfortunately that still leaves me with no knowledge whether Windows CMD performs as badly as the Wine's CMD. I'd like to also rewrite the implementation for Take Command (and to lesser extent to 4DOS, even if I had to limit it to smaller sieve for memory limitations sake) that takes advantage of it's extended capabilities, and to see how well that compares. This idea is constantly nagging at me in my head, because I'm so often seeing retro videos/articles on DOS, and videos/articles on using CMD to do crazy things _not ever mentioning 4DOS/4NT/Take Command,_ almost as if I'm the only person on earth who ever even knew about them. It's my chance to remind people that "hey, this was/is a thing." :D

[@ch2laughlin]

@@robsku1 The windows version runs much faster than the reported numbers. I get about 24hours to do the standard set. I'm using Win10 build 19045 on a i7-6820 @ 2.7GH. Not great speed but much better than what the benchmark on WINE is reporting. Truth in advertising I am using a version of the Batch scripts with several fixes made which I've not PR'd back yet.

[@robsku1]

@@ch2laughlin thanks you - have you tried to run it both ways (windows vs. Wine) to see how much it differs on same hardware?

[@SimGunther]

Still waiting for the top 5 languages that were promised in the last episode!

[@KaziiTheAvali_inactive]

i would love to see a drag race between diffrent linux distros. stuff like pop os, manjaro, arch, ubuntu, and more. shure some of these are based off of one enother but i doubt this makes them the same

[@u263a3]

It’s Dave! I click

[@GoodHomeVideos]

I feel like Powershell cheats a bit since it sits on top of .NET. Those math routines are likely the same ones used by C# but I’m not certain.

[@noVicda]

They 100% are, but disallowing it that means you are outlawing the standard library for the language. Personally I think using standard Unix tools should be fair game, but you don't want someone just writing the whole thing in an awk script and negating the whole point of the drag race.

[@KenPiper]

@@noVicda Would be interesting to see the results of a bash version that makes full use of the standard "library"! Nothing totally cheating like writing out a version in C then compiling/running it to get the results, but doing things like awk for help with math functions. It makes me wonder if the overhead for calling all those processes would end up making it not as beneficial as it might first seem to be.

[@Codeaholic1]

Export doesn't export to the outer scope. It makes a bash variable accessible as an environment variable for child processes. Bash conflates variables and environment variables. It's a bit confusing. Rather than calling kill every loop of the sieve I'd probably create a parent process to monitor my child sieve implementation. There's also the "timeout" command that does similar to this.

[@Blzut3]

Timeout could indeed be used, but since that's not a builtin and can't directly call a function one would need to rig the script up to have multiple entry points. Thus using sleep/kill felt more in the spirit of the competition. (One could probably also wrap the sleep/kill pattern in a function and use a trap to emulate the timeout command over a function but that would be a bit of a mess and can't imagine it being significantly more performant.)

[@Codeaholic1]

@@Blzut3 you could still do it with builtin. Make a parent process start a child process to calculate the sieve unbounded by time. Then in the parent sleep blocking without any loop. Then kill the child. In the child add an EXIT trap to dump the metrics you need to see how far it progressed

[@Blzut3]

@@Codeaholic1 That is more or less what I was trying to say in the last sentence, but you explained it way better. :)

[@romanpul]

Did someone already implement a prolog version of the sieve?

[@brianbrwa]

bash is useful for scripting. I do have a fair amount of unix familiarity. Getting sound to work on a bootstrapped pizza box was fun

[@siggimund]

-Only watch to around --3:24-- 'til now but- something bugs me about the fairness of the comparison. Powershell might be fast but you have to load the .net-framework (I assume,- please correct me if I'm wrong) every time you run a script or one-liner, unless you are working from the powershell prompt. I've noticed this, writing cmd-scripts and looking for advice for som advanced functionality, hard to implement in cmd. The advices almost always seems to be a powershell one-liner, but incorporating that into a cmd-script often slows it down considerably because of the aforementioned loading of the .net framework. Neither cmd nor bash has this disadvantage. Well, you could just always work from the powershell prompt, which admittedly gives other advantages like persistent command history and context sensitive auto-completion like bash, but what about scripts started from the gui? -Anyway on to watching the rest.- PS! Really enjoy your videos. PPS! My solution to achieve the aforementioned _advanced functionality_ from cmd often is to use calls directly to "bash" binaries (awk, perl, sed and friends) from an msys installation if the script only needs to be run on my system, otherwise powershell is the "portable" solution (between win systems that is). PPPS! Though I must admit, trying to write _advanced functionality_ in cmd most of the times is the programmers equivalence of self-flagellation (which curiously enough doesn't seem to stop me🤔😬😁).

[@Luix]

Still waiting for Go!!!

[@alfblack2]

oh wow. nize. didnt expect. Oh compute processing. Would be more interested in file management benchmarks.