The Ultimate Coding Setup for Data Science

In this video I go though how I setup my operating system to code for datascience. We talk about operating system, terminal, IDEs and more.
Tmux settings repo: github.com/gpakosz/.tmux
My tmux and vim settings: github.com/RobMulla/vim_settings
Timeline:
00:00 Intro
00:22 Operating System
03:53 Terminal Stuff
09:19 Virtual Environments
12:40 IDEs
Follow me on twitch for live coding streams: / medallionstallion_
My other videos:
Speed Up Your Pandas Code: • Make Your Pandas Code ...
Speed up Pandas Code: • Make Your Pandas Code ...
Intro to Pandas video: • A Gentle Introduction ...
Exploratory Data Analysis Video: • Exploratory Data Analy...
Working with Audio data in Python: • Audio Data Processing ...
Efficient Pandas Dataframes: • Speed Up Your Pandas D...
* KZbin: youtube.com/@robmulla?sub_con...
* Discord: / discord
* Twitch: / medallionstallion_
* Twitter: / rob_mulla
* Kaggle: www.kaggle.com/robikscube
#datascience #python #coding

[@saxegothaea_conspicua]

Really cool video. Is always hard at the beggining to know which is the beast set up, and chossing from so many options. Thanks a lot and keep the good work.

[@robmulla]

Glad it was helpful! Thanks for the comment.

[@luis9802]

Really cool! This is exactly what is missing. I am just used to work on jupyter notebooks and this helps transition into a more terminal-scripts workflow. Would be interesting to follow you on a project setup. Something like the tensorflow developer exam.

[@robmulla]

Glad it was helpful! Check out my live streams to see it in action.

[@luis9802]

@@robmulla they are always at 3 am, i would love to interact live but I need to sleep haha

[@nevelis]

Two tools that I discovered while interviewing for my last job is fzf (fuzzy find, super fast replacement for find + grep on file names) and ag (silver searcher, like grep -R but lightning fast).

[@robmulla]

Whoa! These seems like awesome tools and I'm going to try using them. Find in linux always seems more annoying than it needs to be. Often I use locate which can work well.

[@juan.o.p.]

Thanks for the recommendations!! I'll definitely give Tmux a try.

[@robmulla]

If there is one thing to take away from this video - is that tmux is great!

[@beethovennine]

GREAT! I'm in the process of learning Ubuntu, so this vid comes just in time...thanks!

[@robmulla]

Glad I could help!

[@liamboyd4676]

This guy’s content is always on point!

[@robmulla]

Glad you think so.

[@jstello]

So much value here! Every time

[@robmulla]

Apprecaite that.

[@bazoo513]

Excellent overview. Thanks.

[@robmulla]

Glad you liked it.

[@matematicasyprogramacioncl1251]

I was wandering what os to use in my Mac for dual boot (data science projects and some develop). I think this video illuminate me.

[@5MassaM5]

As usual. Amazing content! Maybe you could also specify the hardware setup. Screen / Desk / CPU Components / Mouse etc.

[@robmulla]

Thanks for the idea! Maybe in a future video. My PC was built many years back but it has served me well.

[@FilippoGronchi]

Agree with your point...I must buy a new machine but i don't really know which... I am thinking about a MacBook AIr M2 with ethe possibility to install asahi linux... not sure though! Iw ill be a very nice follow up to this video Rob

[@CarlosDelcristo]

​@@robmulla that would be a good video. Looking forward to it...

[@jti107]

this was awesome! I actually use a chrome book and use google colab for everything. really interesting to see different data science setups

[@robmulla]

Whoa, that's great! I'm suprised you are able to do everyhing just using colab but that is cool that you have such a light weight setup.

[@method341]

Do you find any limiting factors with the chrome book or does it do everything you want it to? Obviously not including GPU intensive tasks.

[@ArunYadav-qq1cj]

Yes you can do almost everything on colab but it sometimes gets stuck for object detection tasks and some opencv libraries

[@ashaykatrojwar880]

Was looking for this video!!

[@robmulla]

Glad you found it helpful.

[@MrDotManPeriod]

this video was very helpful for me (new to linux and data science)

[@deepak39754]

Great Video Rob.Thanks for sharing the setup .Could you pls do a video on how to ssh , using vim ? And also few tips and tricks 😃

[@robmulla]

Thanks for the idea! ssh and vim are both really powerful - need to think about how I could make it into a video.

[@deepak39754]

@@robmulla Thank you Rob

[@cedric1731]

You talk about helping us on a daily basis with a bass guitar in the background... I see what you did there :D

[@robmulla]

On a daily BASSis. 😉

[@juliocardenas4485]

I would recommend adding Docker to the set up ; it makes it easier to deploy your work

[@robmulla]

You're right. For a lot of work I do is quick and dirty so the overhead of docker isn't worth it. But I do use docker from time to time.

[@mohamedtekouk8215]

Hamdulilh i have found your youtube channel as i switch from R to python and i found your channel really really amazing

[@robmulla]

Glad to hear that! Thanks for watching.

[@lamhintai]

Would like to see your advice on best practices when convert or bridge between the exploratory / experimental code in Jupiter notebook and production code as modules. 😊

[@robmulla]

Great suggestion! I still feel like I lean too hard on jupyter but there should definately be more of a balance. It really depends on your end goal.

[@solom465]

@@robmulla Quarto in VS Code has replaced Jupyter completely for me.

[@robmulla]

@@solom465 This is the first I'm hearing about it. Reading about Quarto now and it looks like it might deserve some futher investigation! Thanks for the tip.

[@cubic3d1547]

I really haven't ever liked jupyter notebooks. 100% agree on the comments made by Rob about running in VS Code. Biggest challenge for me was exploring in jupyter but then converting to run on a different machine that didn't have jupyter installed or wanted it to run directly from a terminal. I found interactive python in VS Code to be extremely useful. Can run cell by cell just as in a notebook but then can run directly from a terminal without any modification. Not sure it's the "right" thing to do but works for me and what I'm doing.

[@Jackson_rw]

Hey man love your setup! I am pursuing a data science major this year and want to do Linux as my main OS. What colors are you using for your terminal? It's so clean

[@ibowman_UCLA_BRAIN]

Curious, and I can't figure this out based on comments, do you have a dual boot Mac machine (that boots between Mac OS and Linux) or do you instead have a dedicated Mac OS machine and dual boot between Windows and Linux in a different computer? Extremely helpful video! I'm not going quite as old school as top w/o colors but I'm about to do some C++ Qt coding for the first time in a decade, and was mostly interested in your thoughts on IDEs. I did not expect the VS Code recommend. Will go with that, thank you!

[@bazoo513]

Re: vi - Yes, it is unintuitive for novices, but once you learn _all_ of it, it is the most efficient programming editor by far. What it does _not_ provide (without extensions and/or external utilities) are niceties like keyword and variable completion, function/method synopsys help etc. IDE it is not, but as a pure editor, it is unmatched.

[@robmulla]

vim is really powerful. But I actually find most of the benefit is when editing existing code and not writing completely new code.

[@oliver_ai]

I finally learned how to load scripts in Jupyter lab. Thanks

[@robmulla]

That’s a good habit to get into. Using sys to update the path of your scripts makes it easy.

[@thanoslyras3906]

I watched the video and it is very helpful. I would like to ask what is the theme you use because it is very clean.

[@laminkhant8875]

This is such a cool content for me, now I stop dreaming about to reinstall windows :D

[@robmulla]

Glad you liked it! Thanks for the comment.

[@imanshaf8429]

Hi Rob! What do you think of Mint? Do you think it is still better to use Ubuntu over Mint?

[@ChrisHalden007]

Great video. Thanks

[@robmulla]

Glad you liked it!

[@arda3420]

Casual people: python, jupyter notebook and modules are the most important things for data science. Rob: tmux and linux or go cry

[@arda3420]

Omg and Vim too xD. You are gigachad of data science

[@robmulla]

😆 I like to start with the basics and then move my way up!

[@robmulla]

vim keybindings are so worth mastering!

[@mirandaenergypark5271]

Great video Rob - thank you. I hope you don't mind me mentioning, but you have a spelling error in your website title banner. I know, a minor thing, but being a programmer and focused on detail, you may appreciate!

[@robmulla]

Oh! Great catch. You were the first to notice so you win a prize! Just kidding, but I changed it :D

[@peblopadro]

Thanks very much for all the knowledge you are sharing! Cool bass btw 🤟 Quick Q: after installing tensorflow, when I'm importing it in a Jupyter notebook ( [ import tensorflow ] ) the kernel dies. Have tried all potential solutions in stackoverflow and nothing has worked. Any guess is highly appreciated 🙏

[@robmulla]

Hey Pedro, thanks for watching! Sorry to hear your notebook keeps getting shutdown. I would recommend trying to run the same code in a kaggle notebook just to make sure that's not the issue. If it persists maybe attempt to create a new conda environment from scratch?

[@davidmora2241]

Great video! Do you know if it's possible to use the vim keybindings inside kaggle notebooks?

[@robmulla]

Yes it is! The keybindings to switch cells use Vim-like shortcuts with j/k. I believe you can also add vim bindings within cells but I don’t use that.

[@OazadOMER]

Hello Rob can you suggest an alternative of PowerBI for Linux , Thank you

[@danieltheshark]

Thanks for sharing.. I really like the idea to work with linux. Just clean and go. but... what about the issue with the mouse scroll speed? IMWheel... I have a mx master and the first scroll doesn't work. What is this? What about that difficult switch for audio source. I guess the only way is to do that in settings? Do you have problems with your gpu? NVIDI? Are you using vs code + juypter notebook? I can't help --- but this sucks. Kernel broke so many times Thanks

[@gauravmalik3911]

What is the difference between conda env and Python env, which one to chose when? Is python env is used when we are deploying the model?

[@sirsquirrel0]

“Well that’s just me” - me too! 👌👍

[@robmulla]

haha. Glad I'm not alone!

[@philwebb59]

It's so funny hearing you praise bash. It's like tripping back to the early-80s when I used to praise the original Bourne shell and argued how GUI would never be popular. Of course, that was before 'ol Bill stole the mouse from Xerox. Boy, was that a bad business decision for Xerox.

[@arielspalter7425]

That’s awesome tutorial, learned quite a few things. I have a question though- how does Excel fit into here?

[@robmulla]

Good question! Ubuntu has LibreOffice which is a quick alternative to excel, and you can also use office 365 to access excel in the browser. But for some people I know not having excel locally is a deal breaker.

[@arielspalter7425]

@@robmulla thanks! One more question- could you please expand a bit how you run the Linux environment? Is it a dual boot or VM, or a dedicated machine altogether? Thanks!

[@robmulla]

@@arielspalter7425 I run linux as my main OS.

[@satyamARAS]

I used pop and parrot is on vms for learning Linux command and they were good pop And parrot felt like windows lite , ubantu or whatever distribution you use highly depend on project/team/company/domain , i guess?.

[@robmulla]

Not familiar with either of those, but I started using Linux Mint and when switched over to ubuntu way back.

[@Gruuvin1]

When you ran htop, I got jealous. 32 cores with 64 threads? What processor are you running?

[@robmulla]

Ryzen Threadripper! 🔥🔥

[@jcbritobr]

The problem i see in vscode for python is the notebook. Seems like the things not fit exactly in interface, but pylance is awesome

[@robmulla]

I agree. If the notebooks in vscode felt like jupyter lab I would only use vscode.

[@pomiro]

Thanks!

[@robmulla]

You bet!

[@alpsgrin4501]

How to create a cluster and set multi GPU in one computer hpc claster which Linux do you recommend and what settings?

[@robmulla]

First you need a lot of money.... 😀 then you buy the hardware. After that is just setting up ubuntu and installing nvidia drivers. Good luck!

[@bazoo513]

Re: "why Linux" - I have to do most of my work on Windows because my clients do. But I make sure to install CygWin on every machine I work on (including clients' servers running SW I maintain) - command line is immeasurably more efficient than GUI in 95% of cases. On my Windows I now have LTS versions of Ubuntu (22.04 currently) under WSL2 and, again, do almost everything from bash.

[@robmulla]

I’ve hear WSL2 is good but haven’t used windows in a while so I haven’t needed to use it.

[@demohub]

👍 thanks for sharing

[@robmulla]

Thanks for watching!

[@MeuRefrigerador]

Not that I'm advocating for Windows, but shouldn't you be able to do terminal stuff from linux using PowerShell?

[@robmulla]

Yes, or even better WSL- which allows you to run linux on windows. I've never used it but I've heard others like it.

[@aoa1015]

You read minds don't you !!! :) I wanted to ask about your software and hardware setup and recommendations.

[@robmulla]

haha! That's awesome. I get asked a lot when streaming about my setup so I finally made a video about it.

[@oskwon72]

Great! 99% overlapped tools for me except for Org-Mode in Emacs

[@KabeloMoiloa]

Consider using emacs-vterm or the newer emacs-eat instead of tmux.

[@MrEo89]

I’ve heard of the “eat” stuff, but dk how it’s better? Like, eat-jupyter is one in particular I’d love to check out for my older rMBP

[@6872elpado]

I'm amazed that you keep broken environments 😂

[@robmulla]

I'm amazed you don't! 😉

[@6872elpado]

I will try then :D

[@kongson14]

it helps!

[@robmulla]

Thanks! I'm glad it does.

[@prashlovessamosa]

I want you make everything in detail not in 1 hour i want to learn everything because your tutorials are awesome

[@robmulla]

Thanks for the comment! Are you saying you wished the tutorial was longer? You can always watch my live stream or past live stream recordings where I go at a slower pace. Good luck!

[@thatsunpossible312]

Anaconda is no longer free for commercial use, so be careful. You can instead use conda-forge, which is free. The “conda” command line tool is written in python and can be slow. Instill mamba to get the same functionality written in c++. All of this is more important in windows than on Linux. Like you, I just use pip on Linux.

[@sawekb8102]

I worked for some time on Linux but one thing that make me switch again to Windows was that there was no way to install Excel on Ubuntu

[@robmulla]

That's true, but you can use office 365 in the browser I believe. I also try to use pandas instead of excel whenever possible.

[@geodancer7281]

Have you considered LibreOffice?

[@sawekb8102]

@@geodancer7281 Libre office is limited but it is an option

[@geodancer7281]

@Ralph Reilly - That's harsh. LibreOffice is, well, libre. That's not bad for an Excel-workalike. I appreciate your black/white attitude, but when you are searching for a tool, something close might be better than nothing.

[@johnarroyo6376]

I would beleave if u were talking about MS Word, that's a nightmare... but Excel? U could easily work with some other alike excel app, and use it as is u where on excel

[@serdartohma]

In vs code in import pandas as pd and import numpy as np , pd and np is faded. I wonder why? and code is not outputting. can you help me?

[@robmulla]

I'm not sure, but vscode will grey out the lines of imports if they aren't being used in the script. So if you never use `pd` or `np` they will be grey. But that shouldn't impact your code.

[@kesleyraimundo3527]

Why use pip instead of conda to install packages?

[@gustavojuantorena]

I don't know why, but suddenly I want to try tmux

[@robmulla]

You need to! It’s great.

[@shreyaskulkarni5823]

Can you make a video on dual booting?As a student I feel dual booting is a powerful tool

[@robmulla]

It can be kind of tricky, and hard to show in a video tutorial because it requires shutting down your PC a few times. I'll try to think of a way that I could make it work.

[@bazoo513]

Thanks for Tmux tip. BTW, 64 cores, 64 GB RAM? I am green with envy 😀

[@robmulla]

I built my machine myself! It’s pretty old but powerful.

[@sougaaat]

Threadripper? MY GOD!

[@michaelomosebi6374]

This is alot.... Impressive or overwhelming...do not know which to say. Do you have a video that describes your journey, from start? I would really love to see it

[@robmulla]

Thanks, hopefully it’s not too overwhelming. I don’t have a video that describes my journey but have plans maybe in the future. Appreciate you watching.

[@aggbits]

windows + WSL for the win!

[@robmulla]

Windows is overrated. Change my mind. 😁

[@aggbits]

I don t want to change anyone's mind .. Just saying that using windows with WSL2 integration gives you the best of both worlds.

[@robmulla]

@@aggbits I've been hearing a lot of good things about WSL2 so I don't want to bash it (pun intendend).

[@DylanOkyere]

You didn’t add the tmax repo in the description

[@robmulla]

Just added it! Thanks for the reminder.

[@siddharthyadav2120]

Hey Rob! Just curious... do you play chess?

[@robmulla]

haha. What made you guess?

[@nissum5179]

What laptop is goof for data science?

[@robmulla]

Good question. I think it really depends on how much you plan to do locally vs remotely. What really matters to me is that the laptop is comfortable and easy to use.

[@DD-uq8qg]

Kind of the same things that I use too

[@robmulla]

Love it. What do you use that is different?

[@Mutual_Information]

Very interesting. I use a Mac at work and Windows at home, and I love bash via macos.. PowerShell on windows is garbage. But I learned that Windows has, as of fairly recently, a very good recreation of Linux. It's called WSL2. It's not bad. But maybe I should just become an OG linux user like you lol

[@robmulla]

I've heard good things about WSL2 but still prefer avoiding windows unless I need to. You should try setting up dual boot and checking out ubuntu.

[@arminhejazian5306]

i've been using WSL for couple of months and it's great.

[@MaxMustermann-on2gd]

WSL and while in windows Git Bash. + use windows Terminal

[@sayednab]

I tried VSC but I didn't like it at all. it's so confusing and busy. whereas Jupyter notebook is intuitive and easy to understand.. also there is not many good tutorials on VSC contra Jupyter.

[@incremental_failure]

Ubuntu is ok but Arch based distros are a bit better. The package manager is definitely much better. I use paru as a pm and it's great. That said, my VPS still runs Ubuntu.

[@robmulla]

Interesting. Maybe some day I'll give it a try. What is VPS?

[@incremental_failure]

@@robmulla Virtual Private Server. Your cloud box.

[@TheCodeGuy12]

OO-boon-too

[@robmulla]

ewwwwww

[@AbdulREHMAN-co2kd]

i work with jupyter notebook the layout is lame but simple e to use

[@hatwong3637]

Couldn't agree more that Linux is a better platform over Windows as a coding machine. However, I always got stuck when installing GPU drivers on my Linux machine...😟

[@robmulla]

Ah, I’ve been there before. But it is a lot easier than it was before.

[@Pi_Maaster]

debian can be better than ubuntu uses exactly the same pachages and with an extension can look similar to ubuntu

[@mirandaenergypark5271]

Sorry, I meant your KZbin banner...

[@robmulla]

Thanks for the heads up!

[@tainicon4639]

Ubuntu is not the most popular or most stable linux distro. Though It is very beginner friendly.

[@robmulla]

My bad. What's the most popular distro? Debian?

[@tainicon4639]

@@robmulla I believe based on Reddit poles (which may be biased due to the type of people who would take those poles…) Arch based distros beat out Ubuntu based distros. But Ubuntu is a close second. My stability comment was about the issues I have had with apt randomly breaking my system. As a general philosophical statement against Ubuntu, it’s preloaded with Spyware that sends information back to the canonical. But it’s is easy to install and is often the first choice for people migrating over from windows or Mac. So it’s a solid recommendation. The arch based distros tend to attract a certain type of elitist (even though it’s not hard to use…) which is probably best to avoid when you are first starting out.

[@tainicon4639]

@@robmulla great video BTW!

[@soren-1184]

I would use miniconda instead of anaconda and mamba instead of conda.

[@adrianmuresan7764]

I am using anaconda in windows and it started running out of SSD space. After watching your video, I ordered a 2Tb SSD with the intention of installing Ubuntu on it and using your setup. Trouble is, this Ubuntu bastard has a habit of installing on only part of the drive. I made a partition for swap, another for tmp, another for root and that was it, with the expectation that the home partition will take the remaining space. That´s not what happened, I got a 30Gb home partition and 1700Gb unused hahaha. Back to windows for me, I installed windows on my spanking new drive, until I figure out how to install ubuntu and still have space for anaconda. When I think that i used to be only an ubuntu user until aroud 2013.

[@robmulla]

Oh man! That’s annoying. I’ve spent many weekends just trying to fix issues like this. But trust me once you get Ubuntu setup you won’t need to go back and you’ll be happy you switched. Good luck!

[@adrianmuresan7764]

@@robmulla thanks for the reply, Rob. It helps knowing that also the masters have trouble with 'yak shaving'´. In any case, I have now the 250gb SSD where the windows used to reside, when I feel like it I will give it a shot and install Ubuntu there. I do believe you are right about the superiority of Ubuntu.

[@PerteTotale]

I don't see why you condone windows: I had geospatial pkgs installed with pip/whl, with much effort, but it let me to know and understand python much better. So I'm now on windows and with geospatial wizz and QGIS witches, and with conda, and with multi envs. I must have broken your unix egg shell

[@robmulla]

Windows is fine! I’m just saying for me I’m not a big fan. But I also haven’t used it in years so things may have changed a lot since then!

[@linuxmill]

you don't like emacs?

[@robmulla]

I’m sure it’s great. I just started with vim.

[@KarmaKula]

ctrl+b and shift+% ??? I just can't....

[@robmulla]

I map caps lock to be another control.

[@KarmaKula]

@@robmulla that's interesting

[@hansdietrich1496]

As you talked a lot about your terminal ... why bash? There are way better shells out there!

[@CrazyFanaticMan]

The best IDE is notepad

[@robmulla]

The best IDE is ms word!

[@freiTzer]

I use visual studio and pycharm

[@CrazyFanaticMan]

​@@freiTzer Rookie mistake

[@rosetwinz]

​@@CrazyFanaticMan I am relearning to program after a long break. Might I ask why you say so? Is it just because of your work flow (distractions in typical IDEs i.e. autocompletes) or something else?

[@CrazyFanaticMan]

@rosetwinz It really depends on what im doing. If I'm doing Python scripting, which I do a lot at my work and I did a lot in the past for school and other things, then something barebones works best for me. The default IDLE editor with some custom styling or a plain Jupiter notebook like Google Colab. But, if I'm doing something like web dev which I don't often do, using VS Code with useful extensions like Live Server and Thunder Client (for API testing) I actually found incredibly useful. So, ignore my previous comments, it was just me trying to be funny haha In reality, whatever helps you get the job done, that's what you should use :)

[@linuxmill]

"old school users"... guilty😅

[@kslanov]

Moving files with bash commands 🤦‍♂️

[@robmulla]

What’s a better alternative?

[@rishavpapaji5349]

you are soo wrong about ubuntu

[@robmulla]

Ok. Change my mind.

[@rishavpapaji5349]

Sorry don't take it personally but it something is popular doesn't mean its most up to date or latest, most latest packages are in arch linux and it doesn't mean it most stable its a tradeoff between two, but you might have to say that ubuntu has largest community and support not the most updated or latest kind....

[@rishavpapaji5349]

Ubuntu 22.04 use python 3.10 which you know is technically outdated

[@robmulla]

@@rishavpapaji5349 I'm not taking it personally. I apprecaite your feedback and want to know what you think is a better option and why. What OS do you think is best? I'm still using Ubuntu 20.04 because I haven' gotten around to upgrading yet 😅

[@rishavpapaji5349]

@@robmulla I use Arch By the way

[@adrienpacifico5979]

I felt ashamed when I saw your htop compared to mine! 64 cores! I only have 16!

[@robmulla]

What you lack in cores I'm sure you make up for in other areas 😂

[@johnlucas7333]

How many people are doing "data science" projects? Welcome AI

[@robmulla]

Not sure I understand your point. Are you saying that AI has replaced the need for science?

[@petermoore8811]

tip .. I got board and stopped watching. because I thought it was going to be more focused on data science instead you devoted a big portion of the start on working environment. People (video creators) need to assume we know how to operate in linux vs windows vs mac. My tip is provide a link to basics and get on with the title of the video. Having said that. I commend your effort and keep it up.

[@robmulla]

Thanks for the feedback Peter. Sorry you got bored for the video, hopefully my future videos- or some of my other existing ones will be more interesting! I’ll keep your suggestions in mind in the future.

[@bash-shell]

This is a Linux porn rather than any help video.