The vedio i was looking for.Thank you so much .

it seems needlessly complicated. All the rescaling is unecessary for tree models. I am also not sure that because there is 0.3 correlation between two one hot market variables that we should remove market_id altogether. I didn't even follow what was happening with the PCA. Were the PCA variables even considered important to the model? You threw out some engineered features that might have been useful, and then put them back in the end? You might have thought of demand / supply ratio in terms of (total dashers - busy dashers)/(total open orders). And just because two variables are quite correlated doesnt mean their difference or their quotient wont also be useful. If i were doing this problem I would choose one robust baseline model like random forest, much less likely to overfit than boosted models. Time permitting i may try something more exotic, to compare with the baseline. And you don't need to worry about scaling, as i said, with trees. Also - when you have high cardinality categorical features, i would just choose the top 10 in terms of frequency and bucket the others - then do OHE. Or switch them to target encoding. I have to say all the stuff with VIF i have never seen before, I wonder did it actually help? How bad are your results if you just choose the top 10 most important features as decided by random forest and run the model again with just those features. Forget PCA and VIF. So what is your ultimate RMSE? something like 1000 seconds? I got <900 but i was working with a dataset where i junked most of the Nans and dupes. and extreme values, e.g. times over 2 hours. So i was left with ~90% of the original data in my training. Perhaps that is how I got a lower rmse. In the end i used only the top 7 features. (No PCA or VIF). You should talk more about the result in terms of how feasible it is. 18mins on average is not bad i guess, given the huge variability in execution time this three way marketplace can produce. Its a lot of moving parts. Anyway, i am not sure what new folks will think of this exposition. I think it will scare them, and that. is a shame. Being a DS isn't just about showcasing how many techniques you know, sometimes you take a breath and close your eyes before just diving in.

Great video, learnt a lot but when i run the code the files are correctly moved to datasets but they also still exist in the original directory. Wouldn't we have to update the directory that csv_files() looks in as they are now stored in datasets instead?

kzbin.info/www/bejne/qHWWcqx4bqesgZYsi=dkp6s-fp1cWplikb

Dude, are you shadow ban??

This is so True.

Hello, sharing the links to the articles cited in the video, will help to get the most of this .

great video! thank you so much! my results are showing only 50 rows of the df but the channel has over 200 videos. how can I get all of the others past the first page? i feel like it has to do with the pageToken maybe?

Very useful! Thank you for sharing the process

Toooo many GIFs..please keep the video simple. Have some mercy on my eyes 🙂

Too many GIFs in this video....was very tough on my eyes!

So essentially learn ai and how to organize data for it?

you explain everything so well!! grreat job!!!

Lol. Commercial products turn the whole academic field into trash. You gotta have the commercial mind, then.

Game changer

I am missing Nate's explanations. Please send me playlist of vidoes only by Nate on SQL.

excellent excellent tutorial!

Guys leave data science for us buisness people. You computer science grads make software. You have studied to make tools not use tools.

You are very talented at teaching and explaining everything with no bloat while also not assuming anything about your audience. It was an excellent series, and I have learned a lot. Thank you very much!

For someone who lives in Continental Europe, I am experimenting a different reality where everything that counts to recruiters is the credential and the level of experience of the candidate. I feel they should test the ability to write codes more instead of basing their choice on their gut.

Skill issue

the doom posting is crazy

This is something I relate to so much! Spent the last year interviewing with 10 different companies and every single one is a completely different experience where people don't know what they're looking for. The worst, and the most frustrating experience I had was with this company that's into electronic equipment manufacturing that was setting up a new ML team. The title of the role had everything in it (Data Scientist, Software Engineer, R&D Engineer). I was really excited for the role and pretty much aced all the screening and ML rounds with ML researchers and PHDs. Everything was set in place and I was 100% sure of getting the offer (the panel was also confident about me), until this "Software Engineering" final round spawned out of nowhere. I have never done much leetcode in my life as a Data Scientist, and absolutely bombed the interview as I was asked a backtracking question that I had no idea of how to solve. Till this day, this has been the most frustrating experience ever.

bro think he fireship

But everything is now just AI

hahaha YES. Bloody hard but yeah...

Whats the solution then?

I had to go through 5 rounds of interviews. In the final interview, they asked me to prove Poincare conjecture, and I just happened to know how to do it, and that's why I got the job. Ever since I got the job I have been asked to clean Excel sheets and provide summary reports to the head of sales

had an interview for a software engineer job the other day, bluntly asked the guy about the really dodgy Glassdoor reviews.... told me the place is a hellscape and that 10 people have quit in the last 2 months... then asked me if I was ready to continue.... think we are done here lol ( though I really appreciated his honesty)

Requirements of DSA, Devops, System Design is also an issue for other specializations. The whole Computer Science interview process is f*cked up

thanks for addressing this issue, was struggling for years

REAL man i wish i saw this before I got sold on the hype for real

Pronounce “Kubernetes” wrong? Rejection.

When everyone thought future of AI is AGI, its actually API

so what should i study ...... currently doing ml and i know web dev and devops

75k😂😂 3x it and i may consider it

Ai generated voice

I manage to land an amazing job that I actually like, but my last interview was a fucking topology problem… You know, the math field used in relativity. If I wasn’t a physicist that just happened to re read a topology book just because, I wouldn’t pass that interview. Why topology? Why not?

even the fucking script is chatgpt generated, this shit is generated entirely by ai

question, as this is what im currently doing-- im working a normal basic programming job to pay my bills while also working on solving my own problems using my own ML(and otherwise) solutions, in hopes of starting my own business, specifically to avoid all of the problems youve described

i was asked to explain the working of a transformer model for an internship position in india they were expecting to pay me 250 dollars per month😭

Any unicorn with all these skills doesn't apply to employers, they apply to *investors*

As a Sr ML Engineer and AI engineer I can confirm this interview process is exhausting to the core. Once you hit 30, you’re already over the grind and you just want to enjoy work life balance - so studying for weeks on end for a stupid interview is really exhausting

For fresher ml engineer they need SQL, ML ops, Pipelines, spark, pytorch, tensorflow, keras powerbi or tableau, Azure,gcp or aws or everything, Excel , statics and maths, NLP , python or R or both . Now LLM with some magic also needed

ML is the new fullstack

I was able to land a junior data science position this year, after working in business intelligence for 2 years before. I was promised building data pipelines, forecasting, python etc. just to end up doing power bi off sql queries for non tech execs and sales people who tbh don’t even bother using them

I mean, the field was started by researchers and mathematicians. The knowledge built up is really huge here. No wonder half of the positions are for phd’s.

This is the most relatable video ever. I wish i would've have chosen another field. Interviewers expect everything man, i have been rejected so many times because i didnt know something which is no where related to the Job description. I recently gave an interview for NLP engineer and they asked something related to computer vision which i used to work on 3 years ago, i obviously have forgot many things so couldn't answer it, and they rejected me. They somehow thinks that a ML Engineer should somehow know everything related to AI + software engineering + DSA. Man i m tired of trying to crack this field.

You guys are getting interviews? Best I can get is application rejection e-mail. Man I gave 4 years for application rejections.

Crazy