3 Gotchas with WSL 2 around Disk Space, Memory Usage and Performance

[@NickJanetakis]

Recently I made a 2nd video that covers how you can compact your Docker Desktop WSL 2 VM file which is necessary to do if you want to regain disk space related to files that Docker creates. That video is here: kzbin.info/www/bejne/rYSrfnmOYpmcack. IMO you should watch both videos (this one and the 2nd video in that order) because they work together.

[@DT-dc4br]

Thanks Nick, saved some head scratching. Attempted to d/l a big dataset for some GPT fun and found myself with 10Mb free drive space...despite the failed download. Time to check your tutorials for some containerization options & put it all on an external ssd :)

[@NickJanetakis]

@@DT-dc4br No problem, btw I have like 70 blog posts on Docker that haven't made their way to KZbin yet too at nickjanetakis.com/blog/tag/docker-tips-tricks-and-tutorials. Some have but there's a lot that haven't (and probably won't because they're short tips).

[@rileyabel5387]

you prolly dont care but does someone know a way to log back into an instagram account..? I was dumb lost the account password. I would appreciate any help you can give me!

[@dawsonkevin5666]

@Riley Abel Instablaster :)

[@rileyabel5387]

@Dawson Kevin i really appreciate your reply. I got to the site thru google and Im trying it out now. Takes quite some time so I will get back to you later when my account password hopefully is recovered.

[@NickJanetakis]

Here's the timestamps: 0:30 Combating poor file performance in WSL 2 when accessing mounted drives 2:38 Ensuring WSL 2 doesn't use up to 80% of your system's memory 6:17 Clearing out WSL 2's memory usage on demand with a Bash alias 8:21 Reclaiming disk space that WSL 2's VM won't free up by default 10:12 Be careful because WSL 2 won't auto-mount your external drives 12:56 How to shrink your WSL 2 VM to reclaim disk space in PowerShell 18:10 An alternative to the Optimize-VHD command on Windows 10 Home 19:10 Recap and hopefully Microsoft fixes these issues in future WSL 2 patches

[@listajmf]

The pain to go through so we can have a proper Linux development environment... I would rather run Linux as my main OS and put Windows in a VM. "Looking Glass" is an open-source application that helps achieve unbelievable low latency for Windows VM.

[@NickJanetakis]

@@listajmf Hi, Running Windows in a VM isn't an option for me because when I record videos, it's recording the Linux side of things. Having to open those recorded videos to edit in a Windows VM would be too disruptive to my work flow.

[@iamworstgamer]

I have been using wsl since version 1 but man you have done lot of research and your knowledge is astonishing.

[@secondculs]

THANK YOU! I just found out about the WSL2 memory usage that crippling my laptop. I didn't even know that the WSL2 has some storage reclaiming problem. Thank you, once again.

[@NickJanetakis]

Sure, no problem.

[@bensmith4252]

Excellent work with this video Nick. Really useful, especially how to reduce the amount of memory WSL2 uses!

[@NickJanetakis]

No problem, thanks a lot for watching.

[@nikos.andreou]

Thanks for the video Nick. In my case, the vhdx are located in C:\Users\Me\AppData\Local\Docker\wsl\data C:\Users\Me\AppData\Local\Docker\wsl\distro Debian is located in distro, data are located in...data :)

[@deomartinez77]

Dude Thank you for posting these paths. I seriously had no idea where to look.

[@theobgshow]

Superb work. Thank you.

[@RobertJohnTaylor]

This helped a lot, my system memory was being pegged.

[@pxlbltz]

I ran across this virtual disk bloating issue myself and although I use Windows 10 Pro I don't have the optimize-vhd program. I read somewhere it's because my Windows doesn't use hyper visor or whatever.. anyways.. I found the alternative way that you are showing at the end but I ran into another problem: even though I detached the vdisk after compacting I couldn't access the disk anymore. Windows said it's in use. Luckily this was not a big deal for me because there was no important data on the virtual disk. I uninstalled and reinstalled the Linux distro. I found that a slightly different method works fine. First you check if the distro in question is running with "wsl.exe --list --verbose" If it is running terminate it like this "wsl.exe --terminate Ubuntu" (name of your distro of course) then start diskpart and enter the select command like in your example but do not attach the disk. Just enter "compact vdisk" and that's it. For me this works great, just in case someone else has the same problem, that I had.

[@NickJanetakis]

It's interesting that optimize-vhd isn't found. Which build of Windows 10 Pro are you on? Thanks for the tips. I'm sure it'll be useful to some folks out there.

[@pxlbltz]

@@NickJanetakis Hi, it's Windows 10 Pro Version 20H2 Build 19042.867

[@OtRatsaphong]

Thanks for the heads up, Nick. For some reason I haven’t been able to upgrade Windows 10 to 2004 here in Australia, even though I bought a brand new copy.

[@NickJanetakis]

You might need to manually update it from Microsoft here www.microsoft.com/en-us/software-download/windows10. Either the top or bottom link depending on what it says is available. The bottom link is how I got the update on the day it was released.

[@Ycros]

I'm in Australia and I got the 2004 upgrade option finally about 3 weeks ago.

[@OtRatsaphong]

@@Ycros Great!! How did you get it? I will look out for it. Would be good to try out WSL2, but in the mean time, I'm playing with setting up a Linux desktop using elementary OS or Kudentu.

[@OtRatsaphong]

Looks like Windows 10 2004 have finally arrived.

[@Ycros]

@@OtRatsaphong I just kept checking and eventually I got the option in windows update. From what I understand they are rolling it out to people in phases, so not everyone is getting it right away. They are also scanning for driver problems/hardware incompatibilities and blocking the upgrade for certain people based on that until those issues are fixed.

[@MalkizChannel]

Hey! The video was informative, thank you very much. I want to say that this video does not have the ability to turn on subtitles. English can be difficult for me and I often turn on automatic subtitles. Other videos on the channel do not have this problem. Not sure what could be the matter? Maybe something is disabled in the video settings?

[@NickJanetakis]

Hi, to be honest I'm not sure why they're not included but I don't see the CC icon here either. I didn't disable them. Maybe it got uploaded during a time where KZbin's auto-generated CC feature was disabled. As a video creator there is no option to auto-generate new captions. The best I can do is upload subtitles that aren't auto-generated but that would require manually creating them.

[@MalkizChannel]

@@NickJanetakis Hi again! I tried to solve the problem with subtitles - I downloaded your video and uploaded it privately uploaded to my channel. Subtitles were generated overnight (at least a couple of hours they were gone). You can repeat this yourself with this video. i.imgur.com/pbP4TbI.png Or you can download the ready-made text with timestamps that I generated for you: pastebin.com/dr8r9SRg Put it in a text file (.txt or .sbv format for KZbin), go to the subtitle settings for this video and indicate that you want to add subtitles from a file with timestamps.

[@NickJanetakis]

@@MalkizChannel Thanks. When I upload your generated subtitles (with timestamps) it comes up as 1 massive subtitle at the start of the video instead of breaking it out for each section of time. It's definitely all formatted correctly as well.

[@MalkizChannel]

@@NickJanetakis Yes, you are right, now I tried to insert such subtitles myself and it didn’t work very well. Now the method is exactly working, I made it on video for you. It is difficult to describe everything in clear language in the text. New subtitles (without the timestamp): pastebin.com/a23ZTTza Video instruction: kzbin.info/www/bejne/bKqoeotojMmlb9U

[@NickJanetakis]

@@MalkizChannel Thanks for going the extra mile, I really appreciate it. I've followed those steps and after I click the "edit timings" link after pasting in the text it brings up a dialog box saying "Syncing captions to the timeline is taking longer than expected. You can safely save and close this dialog and check back later.". It seems like it's going to work, it might just take a bit of time to correctly convert a massive blob of text into sync'd timestamps.

[@nervous711]

FIrst of all, thank you for you information, it's what I've been looking for! been bugged with this space issue for a whole day! cuz I'm new to ubuntu system. Some quick question: Does the method reclaim all of the space before the packages installation in miniconda virtual environment? Because I noticed that before the packages installation in the venv, the disk was around 28 GB. However after I removeed that venv with "conda remove -n PACKAGE_NAME --all", "conda clean --all", “sudo apt clean”, "sudo apt autoremove", and then finally your method, the disk was back only to 24 GB. I wonder if it's the method to not reclaim all of the space in the first place or might just be some other problem?

[@NickJanetakis]

No problem, this will only reclaim what's been cleaned up on your file system first. It would be expected you run any clean up commands within WSL before running the reclaim commands.

[@nervous711]

@@NickJanetakis Got it, thanks!

[@scuzzyhoulihan9853]

you can download the wsl 2.2.1 prerelease finally! Not sure if the disk space fix is in. So far the memory usage has been fine. The tunneling and mem usage is a great fix/improvement. fixes this stuff: Enable DNS tunneling by default Integrate DNS tunneling with Linux native Docker Change the default reclamation mode to drop cache Fix hang when the guest crashes during distro initialization

[@NickJanetakis]

Thanks. Based on github.com/microsoft/WSL/issues/4699 being open I'm guessing disk space is still an issue. Version 2.2.X is considered a pre-release on GitHub btw, at least at the time of this comment.

[@scuzzyhoulihan9853]

Yes disk space still an issue sadly been using it this week. But the ram issue is resolved finally.@@NickJanetakis

[@LDS-Hellgineer]

First of all, thanks for the video. Unfortunately both tasks: 1. optimize-vhd -Path .\ext4.vhdx -Mode full and 2. diskpart select vdisk file="C:\...\ext4.vhdx" compact vdisk Did not work for me. Altough the progress bar is visible and in both approaches, it finishes at ~30% with sucess / no error The blocked memory on my C drive is not gained back. Maybe someone had the same issues and could help. I have w11 pro and use wsl2 my ssd is 2tb so: wsl2 is taking virtual: 1024 GB physical: 960 GB even tough its actual size is just 29GB

[@iamworstgamer]

awesome video. I have one question. Can I use windows resources on demand with wsl2? If I am running a wsl2 for development with 4 gb of ram but some scripts require more ram. But if i set it in wslconfig it will restrict the ram. I can define config inside the wsl2 distro but it has to be on demand. I don't want to define it again and again.

[@NickJanetakis]

Hi, thanks. The memory config option is on demand in the sense that if you set 6 GB, that acts as a hard limit but it won't always use 6 GB by default if you're not using it. If you want to raise that limit to something higher then you may want to set a higher value by default for that memory config option. At the end of the day if you have scripts that need more than the limit then the limit needs to be raised. Since it's on demand'ish by default you can set a fairly high limit without it overtaking your system by default. I don't know of any config option that says "Use 6 GB as a limit but if a script needs 8 GB in the end, just let it use 8 this time". It might exist, but I didn't find it in the docs. The hard part would be for MS to know when you want the limit to be allowed to go higher or not which makes me think there likely isn't an option for that. If I were you I'd set a higher limit and roll with that because it's not actively using that much memory all the time. It's only an enforced hard limit.

[@hirisraharjo]

So if I understand correctly, wsl2 takes up space pointing to where the ext4.vhdx file is located. Is that correct? anyway, Excellent video!

[@NickJanetakis]

Thanks. Yeah that ext4.vhdx file is your WSL 2 instance's filesystem wrapped into a vhdx file.

[@nikhilranka9660]

Thanks for the detailed tutorial. Any ideas on how to get WSL2 connect to the internet with VPN turned on, on Windows?

[@NickJanetakis]

No problem, I don't use a VPN personally. There looks to be a lot of responses of it working on Google tho, but it depends on what you use. I would start by Googling "WSL 2 vpn".

[@martinrj30]

Can WSL2 mount unmapped drives? I'm trying to 'fsck' and can't get 'parted' to see all my disks.

[@yubikhadka]

Hi Nick, Thank you for this wonderful tutorial. Lately, I came across similar problems and tried to reclaim space using the command you provided. I am running windows home machine. However, when i try to run 'attach vdisk readonly'. i get the following errror that says Diskpart has encountered an error. The process cannot access the file because it is being used by another process. and Then I checked by system event log for more info, and the error was related to hyper-v-storage which says Failed to map guest I/O buffer for write access with status 0xC000012D. Device name = C:\Users\yubi\AppData\Local\Docker\wsl\data\ext4.vhdx . Any help would be appreciated. Thanks

[@NickJanetakis]

Hi, personally I'm not running Windows 10 Home so it's hard for me to say exactly went wrong but did you first shutdown wsl? That step is important in any version of Windows. It's mentioned in the first command to run.

[@yubikhadka]

@@NickJanetakis Hi Nick, Thank you for the quick reply. Yes, I did. but still the problem persists. I tried restarting, shutting down. nothing helps.

[@NickJanetakis]

@@yubikhadka Do you have something else open that's referencing WSL?

[@yubikhadka]

@@NickJanetakis I am not sure. well, docker is shut down. i am not sure about other services thats referencing WSL. are there any other specific services that i need to check?

[@yubikhadka]

There was something called Hyper-v-host compute service running. Once i stopped it, its working now. But i don't know how. Thank you for your advice Nick

[@alecclews]

I had trouble finding the vhdx file (I installed using Chocolaty). Turns out it was located at "C:\Users\All Users\chocolatey\lib\wsl-debiangnulinux\tools\unzipped\ext4.vhdx"

[@NickJanetakis]

Thanks for the heads up. I only installed WSL distros through the MS store.

[@while-loop]

I give wsl all my ram, but I have a Task Scheduler script that drops cache every 5 mins. I did have the alias, but I chose to automated it.

[@firstlast313]

Will share that script with me?

[@kayzweller_dj]

Will you share how to setup that? Thank you

[@Archil1997]

You should probably reclaim disk space from your docker and docker-data distros as well.

[@NickJanetakis]

Yeah, running a docker system prune on a regular basis is a good idea. I've been doing that for a while now.

[@Archil1997]

@@NickJanetakis I had the same problem with docker's vhdx files not returning memory, pruning didn't help and I had to reinstall docker. Now I think optimize-vhdx shown in the video could've helped.

[@NickJanetakis]

Oh I see. You mean the VMs created by Docker. That's a good point. Let us know if you end up shrinking it. The prune command would only remove things like dangling images (useful to run on any system, but isn't related to fixing low level VM disk usage issues).

[@zihaosong1514]

GOAT

[@jaisonvj8203]

i have docker installed in my wsl that is ubuntu 22.04. when i build image and delete images it using more disk space on my c drive. finally i used docker system prune -all command in wsl. i showned me 70 gb reclaimed. but in windows the drive is still full .can you give me a solution please. i dont have docker desktop. i am using docker in wsl

[@NickJanetakis]

12:56 in the video covers what to do. If you have Docker running in WSL it'll work. It's unrelated to Docker Desktop.

[@vicentepallamare2608]

Also COM ports are not being passed through as expected ...

[@jmpt19]

Sometimes I need to use hyper-V and sometimes WSL, can I change on command between these two?

[@NickJanetakis]

Yeah, I still run a few native Hyper-V VMs outside of WSL. It works no problem.

[@Jamesalec63]

Can you use cloud access to do this thinks especially with Microsoft cloud storage or a USB stick with a good storage space? Just a thought?

[@NickJanetakis]

To do what? You can mount a USB stick into WSL 2 and use it as an external drive. Or do you mean to use it as a basis for the VM's vhd file? You can customize where the vhd file gets created by following this github.com/MicrosoftDocs/WSL/issues/412.

[@Jamesalec63]

@@NickJanetakis either of these? No way to use Microsoft cloud this way as well?

[@NickJanetakis]

You might be able to mount a network location. I haven't tried personally.

[@Jamesalec63]

@@NickJanetakis well not sure but I imagine anything that can be done to make a sfe connection that is exactly what you've shown is completely correct but if there are other ways to apply this connections please let me know? I am new to this but find interesting with virtual tools and other things I guess there both a fun side and a important side and to know where to start or not enter something you not sure of but I guess it is a learning curve for the master to?

[@NickJanetakis]

@@Jamesalec63 I'm not really sure what you're trying to do. If you're working in WSL 2 and Microsoft has Linux tools to connect to your cloud storage, you should be able to use those tools.

[@DevThought]

I am running wsl2 for web development, and when I delete some files from repo, I don't know where they went? I imagined finding them in my windows recycle bin, but they are not there? how do I make sure I am deleting something permanently?

[@NickJanetakis]

They are permanently deleted from the OS's point of view that's running inside of WSL2. It's just the WSL2 VM's disk space isn't going to be reclaimed until you manually shrink its disk using the workflow in this video.

[@DevThought]

@@NickJanetakis thank you. You should make video on developer beginners guide with WSL. An other question. I try to run a React project that repo is in my windows environment from WSL, it run very very slow?

[@NickJanetakis]

@@DevThought This video covers a bunch of guides on the tools I use, including how to get situated with WSL kzbin.info/www/bejne/n5W6XpSkbp6Hd68. Beyond that, the installation instructions for most things inside of WSL is the same as native Linux. Yes it will be slow then. Your source code should be in WSL2's file system to get the best result.

[@divyamsharma5198]

I am not able to find alias file in your repo

[@NickJanetakis]

Hi, it's at github.com/nickjj/dotfiles/blob/42a3c5bae24575d9324b10b2f83587a995840289/.config/zsh/.aliases#L164, since this video I moved to zsh from bash. I added a direct permalink to it in the description, thanks for the heads up.

[@divyamsharma5198]

@@NickJanetakis thank you so much nick

[@divyamsharma5198]

@@NickJanetakis damn that drop_cache command freed 800 mb from my memory usage . Thanks alot buddy

[@NickJanetakis]

@@divyamsharma5198 No problem, that's a lot!

[@aravindch9123]

how to create wslconfig file? when i tried to create new file it is showing text file how to convert to config file? your video given me understanding on why laptop is hangging after i installed docker and started using custom ide provided by my mentor Great video Thank you

[@NickJanetakis]

No problem. From within WSL 2 you can make the file by running: touch .wslconfig from your user's directory in Windows.

[@aravindch9123]

@@NickJanetakis thank you Nick 👍🏼

[@nipenmajumder]

Hey my laptop have 8 gb ram .i assainged 3 gb ram in docker but when i start docker every thing starts to lag any work arround for me?

[@NickJanetakis]

I would bring up task manager and check your total system memory's usage. If it's near maxed out, maybe reduce Docker down to 2gb?

[@SIDDK24]

is it possible to change main storage on other drive also storing on other drives & runing .sh files it says move to case sensitive path

[@NickJanetakis]

Hi, main storage for what exactly? If you use a mounted drive (outside of WSL) that's how you can access files on a different drive. You should be able to run shell scripts from those drives. Just make sure the script is executable.

[@SIDDK24]

Wsl2 install on c drive if i sync anything storage gets full & some shell scripts are case sensitive so i can't run them also i find one guide on github which set case=force but doesn't work on my device

[@NickJanetakis]

@@SIDDK24 Why not run the shell scripts with the correct case sensitivity? I run plenty of shell scripts within WSL 2 and on a mounted drive too.

[@SIDDK24]

@@NickJanetakis actually i trying to build custom rom & recovery on wsl2 so that's why I want case sensitive

[@ecptavares]

I've installed linux with wsl2 and I uninstalled but my harddrive space did not go back to what it was before I install linux.What do I do to recover the space I used to install linux with wsl2? thank you

[@NickJanetakis]

Did you remove the app from add / remove programs?

[@ecptavares]

@@NickJanetakis Yes!

[@NickJanetakis]

@@ecptavares Did you follow the video at 12:56?

[@ecptavares]

@@NickJanetakis Yes but there was no CanonicalGroupLimited .... inside the folder packages .

[@kayzweller_dj]

Where to put that .aliases file? Seem not work for me

[@NickJanetakis]

Your WSL 2 home directory, but then your bashrc (or equiv.) would need to read it. An example of that is here: github.com/nickjj/dotfiles/blob/0f801d67cfb73705045784ebf5f2ee8650528a44/.bashrc#L27

[@Gold_fi]

Pls can you make a video on how to configure Vim for java and c++. I'm into competitive prog. and I wish I could use vim

[@NickJanetakis]

I don't have any experience with Java and C++. But if you require some type of intellisense, you might want to try using github.com/neoclide/coc.nvim. If you Google around that topic of using that plugin with Java or C++ you should find some video or blog tutorials on setting it up. Personally I don't use that plugin tho. For the languages I work with, regular autocomplete based on buffer content works for me. If that works for you, I made a video on my set up here: kzbin.info/www/bejne/aJebmWdrjrdnb5Y

[@hereb4theend]

Just get a Language Server Provider set up for Vim and you're good to go. 😜

[@cihad735]

Hi, does WSL 1 also have such problems?

[@NickJanetakis]

Hi, nope, WSL 1 is all good on this front.

[@cihad735]

@@NickJanetakis Thanks!

[@nuniezjorge]

this video makes me think wsl is still too new to be practical, but fun and promising (on a second thought)

[@NickJanetakis]

I've been using it for a ton of projects without issues since it was released in May. It's very practical. These gotchas are able to be worked around without much effort. I run the disk space one about once a month.

[@nuniezjorge]

@@NickJanetakis it needs more time to reach madurity and popularity, i know it has future, and for that reason i might test it for a while

[@HaiVu-lm1mf]

Hi why my drive path is different than yours? Mine is /mnt/d yours is /d.

[@NickJanetakis]

That is due to the wsl.conf file being configured to use / as the root path. It's in my dotfiles at github.com/nickjj/dotfiles/blob/master/etc/wsl.conf.

[@HaiVu-lm1mf]

@@NickJanetakis Wow TIL, that much more typing-efficient. Any drawback config the path like that?

[@NickJanetakis]

@@HaiVu-lm1mf None. With WSL 1 it was necessary to do that to even get Docker Desktop volumes to work. That's mainly why I added it, but also as you say it's easier to type so I kept it around for WSL 2 too.

[@HaiVu-lm1mf]

@@NickJanetakis Thanks for your help.

[@abduksam]

Does this also happen with wsl1

[@NickJanetakis]

Nope, WSL 1 is quite a bit different. WSL 2's file system access is a bit faster tho which IMO makes it worth using.

[@Isaac-eh6uu]

Are these still problems?

[@NickJanetakis]

Yep and the methods to address them in this video still work the same today.

[@porterneon]

Backup on USB disk? Time to start using NAS :)

[@NickJanetakis]

One day!

[@Hardcore_Remixer]

My fix for WSL eating your RAM and storage: Linux Virtual Machine. Why not dual boot? Because it will make your host machine work slower as well, just like WSL.

[@NickJanetakis]

Dual booting requires shutting down your whole environment which is disruptive in that you need set everything back up again. Plus in my case I record and edit tech videos using Linux, but I depend on specific Windows tools for editing videos. It would kill my workflow if I had to record in Linux but edit in Windows.

[@clarencegui4402]

I’ve tried dual boot on my working laptop and honestly it has more potential problems than wsl and consumed me much more time than I expected.

[@TwskiTV]

what? dual boot will make your host machine slower? makes no sense. Virtual machine will, however.

[@AslamD]

Very interesting video. Any workaround for accessing ntfs partitions being very slow? There is a huge thread on Github issues regarding this and its been locked 6 days ago by Microsoft, still the issue is open :( github.com/microsoft/WSL/issues/4197 Its so painful when you end-up doing all the server configuration and you have very limited storage on WSL partition but more storage on other partitions and still can't use them.

[@NickJanetakis]

Thanks. I'm not sure, I think all forms of mounts are slow until they decide to fix it at the WSL 2 level. Fortunately the end of the issue you linked does leave us with hope that it will be addressed.

[@listajmf]

The pain to go through so we can have a proper Linux development environment... I would rather run Linux as my main OS and put Windows in a VM. "Looking Glass" is an open-source application that helps achieve unbelievable low latency for Windows VM

[@voltmanip]

WSL 2 NEVER WORKS

[@NagarajCruze]

for simple task, you're taking 20 mins, That's bad.

[@NickJanetakis]

You could always take advantage of the timestamps that are listed with the video to skip around, or goto the description and visit the reference links and copy / paste things without understanding what the problem is or why you need to run these commands.

[@NagarajCruze]

@@NickJanetakis every second of your steps is important, can't skip anything just like that.... but 20 mins is way longer. Thanks for the video