Math for Data Science and the Most Commonly Missed Areas

A subscriber asked what math they should focus on for data science. The short answer is, the same as a quant.
However this answer depends on what you think a data scientist does as everyone seems to have a different definition. I define a data scientist as someone who builds models. This is the same definition I use for quants. Math is a core language for logic and is required to build sound models. Those utilizing packages without a deep understanding of the modeling process and the models' math are a danger to the firms they work for.
My Recommendation for Math Basics:
Calc 1 and 2
ODE and PDE
Linear Algebra
Proofs
Measure Theory
Probability Theory
Math Undergrad for Quants:
• Undergrad Courses and ...
Gilbert Strang's Linear Algebra Class:
• An Interview with Gilb...
Introduction to Stochastic Calculus with Applications (Book Review):
• Introduction to Stocha...
Website:
www.FancyQuantNation.com
Support:
ko-fi.com/fancyquant
Quant t-shirts, mugs, and hoodies:
www.teespring.com/stores/fanc...
Connect with me:
/ dimitri-bianco
/ dimitribianco

[@-es2bf]

I have a master's degree in mathematics. Currently working as a data scientist and the level of mathematics used in data science is very very very low. People throw around "AI MACHINE LEARNING DEEP LEARNING BLABLABLA buzzwords" when in reality what they are talking about is simple regression models. Besides regression, most of my time is spent writing SQL commands in the terminal.

[@gcloptonAPSU]

From my experience, the level of difficulty largely depends on what you’re working on

[@-es2bf]

@@gcloptonAPSU please let me know of any industry besides academia that lets you have fun with actual mathematics while working with data.

[@mikeh5050]

@@-es2bfUnderstanding the math behind it is important for building and interpreting models no?

[@paul5324]

Biostatistics?

[@jamestucker4800]

I was initially studying "data science + computer science" as a undergrad, but made the switch to (statistics + computer science) as the emphasis on maths and mathematical statistics covered in the statistics course was far beyond what was covered in the "data science" program, and I now have a much more solid understanding of the fundamentals to build off.

[@jgo8305]

When I first found this channel I was working as a mortgage underwriter after finishing my undergrad degree in mathematics. I was good at it but hated it. You inspired me to go deeper with statistics. Currently I am interning at an Ivy League writing SAS code everyday and more than halfway through my MS in applied stats. I want to say thank you Dimitri, you're up there with Lex Fridman in terms of people I look up to and strive to emulate. Thank you for the passion and advice you put into these videos.

[@DimitriBianco]

Thanks for the feedback! It is great to hear the videos have been helpful in finding your career. Statistics was also my gateway to meaning for applying the math.

[@shehzaibirfan]

Thanks a lot. Being in grad school and trying to balance between how much theory I require for maths to be a good practitioner, is a tough question. Great video. My takeaway; there is no shortcut in life so more solid is theoretical foundation, better practitioner you would be.

[@byronwilliams7977]

Thats a really great question and response. I've got an undergraduate in Applied Mathematics from a reputable institution in the US, however I don't feel I have enough Maths. But then I see a bunch of folks online with next to nothing under their belt raking in that Data Science money. How do you strike the balance, and what should that balance be ?

[@alan713812]

Love videos like this, book recommendations. I truly believe that’s the way to study a subject the right way. Wish more people understood that. Thanks for sending this message !

[@edwardgrigoryan3982]

Incredibly informative. Thank you.

[@Simba365]

Thank you for answering my question I appreciate the knowledge

[@unajoh6472]

Thank you so much for all this recommendation! Your videos always give me great inspiration and movitation.

[@DimitriBianco]

Thanks for being a part of the channel!

[@adrianfletcher8963]

I'd like to see the videos on constraints and boundaries for evaluating models. That seems like something along the lines of what I don't know I don't know. I also appreciate that you get straight to the point and add more details about what you can learn in the book. I don't think anyone watching your videos would be bored if went over details.

[@adisurani9092]

Thanks Dimitri for all the content / advice / suggestions! Hope I'll get to buy you a coffee / have a chat in real life in the near future! I just got my first job on the quant side (Quant Dev) and I wanted to say a small thanks for all the help along the way, ranging from advice on Masters degrees / to courses / topics and further on to career paths / long-term direction!

[@DimitriBianco]

Thank you! I appreciate the feedback and the Super Thanks!

[@EvanZamir]

I don't know many data scientists who know measure theory. I certainly don't, not for lack of trying. I have like 10 books on measure theory, and have never been able to get through any of them lol. But I'm glad someone out there appreciates it! Just don't see how for most of us, it's really relevant in most business settings. It would be great to see a video showing a real application and how I can benefit from it.

[@-es2bf]

If you want to see where measure theory is used,I suggest you to study probability theory. However, mathematical probability theory as developed by Kolmogorov is far away from the buzzword lingo of clueless mba's who just need regression models.

[@tybaltmercutio]

@@-es2bfThis doesn‘t really answer the question asked. „Measure theory is used within another branch of mathematics called probability theory“. This we already know. I guess, the person asking the question was asking for a concrete example of a practical use case.

[@AL_383]

Hi Dimitri, your channel is an absolute gem and I was wondering if you have any recommendations (or anyone reading this) for channels similar to yours but more focused on the datascience side of things.

[@shagungarg7628]

Hi Dimitri, I wanted to know how old enough data do I need to prove stationarity in a linear combination of prices of financial instruments.

[@lebronjames8507]

THANK U FOR THIS VID IVE BEEN TRYING TO FIGURE OUT WHAT MATH I NEED FOR SO LONG GOD BLESS U BROTHAAA

[@josephzhang8809]

Your videos are not well structured or organized, making people hard to find the information they need. But your content is excellent . Just little bit more improment or clearer outline with several links will boost quality of video so much

[@screweddevelopment12]

I’m a data science grad student at umich, and it feels like there is pretty limited opportunity for data science roles right now. Most of them appear and then are closed after a day. I’m in the pipeline for a couple, but I’m hoping that there will be more opportunities as I get closer to graduation. Otherwise, I can always get some random remote/hybrid IT job with decent pay and wlb, and then do Kaggle competitions / OSS contributions with my spare time. I’m really liking how practical the math is for data science though. If you aren’t doing everything properly and ensuring that you satisfy all of a model’s assumptions, it will blow up in your face at the end during testing.

[@prohero113]

Hello, out of curiosity, which data science program did you take from umich? I noticed there are couple, MSDS and MADS.

[@thedailyepochs338]

Hello Dimitri, lovely video as usual. I am in the data field and plan on creating an AI solution for finance. Do you think going through the cfa curriculum will be give me some domain knowledge to know what aspects of finance Ai can make better. Just asking for your opinion. Thanks

[@DimitriBianco]

It should give you an overview of finance and some of the main problems however I would do research afterwards on what the quant side is already doing. I have seen a lot of firms creating automated solutions for things that either aren't time consuming (automation adds little value) or there are already complex solutions and automation can't enhance it.

[@Kinghercules]

Correction sir, ppl who are getting a DS degree in undergrad or graduate level don't take or need to take proof writing. 😄 Measure Theory is the second part of Real Analysis it's not something that DS need.

[@liiilllliiilllliiiil9821]

Dmitri, I am currently studying an undergrad data science degree at the University of Bristol. I know that undergrad data science is usually quite undeveloped; was wondering if you could take a look at my course and specify it’s shortcomings (University of Bristol BSc Data Science, under the program catalogue section). Thanks, insightful video as always!

[@ian4692]

I'm also at U of Bristol but doing postgraduate. I think the BSc DS in Bristol offers a good collection of modules for UG students (covers all Dimitri mentioned). It looks almost like an applied mathematics degree. It should be able to provide you with a solid foundation for further studies.

[@leonmozambique533]

Not sure what you would use DEs for in data science. You missed out on an algorithms book. In the real world, most optimzation problems are too complex and intractable to be solved analytically with calculus and linear algebra. That is where optimization algorithms come in such as Newton's method, gradient descent, coordinate descent, mirror descent, etc and to have a strong background in computer algorithms in essential for your brain to be able to strcture problem solving in a way that it can be interpreted by a computer.

[@andresrossi9]

These are typically included in the DS program and most of the methods you mentioned are included in calculus lol

[@granteckhardt4878]

Do you think higher level math courses like real analysis are worth taking for someone doing a math and cs double major interested in data science?

[@DimitriBianco]

Yes! Real analysis really opens the doors to more abstract thinking about math. I often find issues with models developed by someone and the ability to identify those issues comes from a deeper understanding of math which is taught in courses such as real analysis and measure theory.

[@grahamsayle]

How would you possibly get to measure theory and probability theory without proofs? At my university they both require 2 real analysis courses

[@DimitriBianco]

This video isn't a complete list of all possible classes one needs. It highlights the math core which can be found in the math for undergrad quant preparation video and then highlights common topics that many data scientists are missing. Measure theory is often taught at the graduate level. This is why quants and data scientists in the finance industry require masters or PhD degrees. You simple can't fit enough valuable courses into an undergrad to build solid models.

[@datawithtess]

After my undergraduate in mathematics. I learnt about data analytics. Am looking forward to do my Masters I data science or quant

[@datawithtess]

Love you man

[@meteor8076]

measure theory, what a simple topic) the measure theory itself has a lot of pre-reqs

[@DimitriBianco]

Yes there are prerequisites such as analysis 1 and 2 at many schools. This is however a topic I think that is often missed and adds a lot of value in building robust models.

[@meteor8076]

@@DimitriBiancoyes, but to know analysis 1 and 2, you already must know calculus and how to write proofs and set theory. Overall the candidate should have a solid math maturity.

[@julianbeatty2909]

@@meteor8076Measure theory...., That is NOT an undergraduate math course. At my university, it is basically Graduate real analysis, which is a PhD math level course.

[@tobiassteindl7792]

@@julianbeatty2909 What uni? Just wondering because we do Measure Theory in 3rd semester undergrad

[@julianbeatty2909]

@@tobiassteindl7792After looking at a sample of the book he showed "measure theory and probability theory", I have determined that what people call measure theory and what 'level' they teach it at are different. The book he showed looks more like real analysis. At my university there is a course called Measure theory (or aka graduate real analysis) that uses the book: www.math.wustl.edu/~victor/classes/ma5051/rags100514.pdf. Regardless my point was that there are good reasons why certain maths like "measure theory" are missing among quants (especially new hires). Hiring someone straight out of university, you will get someone that is either (a)Very good programmer, can "do" mathematics such as Calc 3, multivariable statistics, but likely hasn't taken many or any proof-based math courses. These are usually computer science/engineers. (b) Very good at mathematics, has some experience taking proof based courses such as real analysis, but probably not nearly as good of a programmer as the average compsci/engineer graduate. These are usually pure math students. So in conclusion. It's pretty unreasonable to expect undergraduate students to have both (a) and (b). If you want both you need to hire graduate students. Even then I would say that almost none of the graduate computer science/engineers have taken anything close to our Graduate Real analysis course that covers measure theory.

[@sellbythebell]

❤ your content my friend. #aspiringquant

[@tvstation8102]

I can't imagine doing PDEs but not Calc 3!

[@DimitriBianco]

I find it odd as well. I have mentioned in the past that calc 1-3 was required however many people pointed out Calc 3 wasn't a prerequisite for many masters but ODE and PDE were. I had to self study to catch up but calculus is really just a series of foundational classes that are needed in most other math classes.

[@madhavpr]

Measure theory has been the bane of my existence for a few months. But I'm sure I'll get through the pain and torture.

[@DimitriBianco]

You're already ahead of most of you are studying it. I'm by no means an expert in it however just trying to consider the ideas while developing models goes a long ways.

[@madhavpr]

@@DimitriBianco Thanks!! I'm very interested in advanced math. The subjects that I care about require more than a passing acquaintance with measure theory and measure theoretic probability. Sometimes I go so deep into a rabbit hole that I lose sight of the big picture. Any ideas or tips on where to draw the line?

[@DimitriBianco]

@madhavpr this is really the purpose of a graduate degree. It should teach you enough of every topic too develop the tools. Then in industry you should specialize. As for topics to cover, I would try and get the basics down for regression, time series, machine learning, and optimization. Then I would slowly work on a specific area and try and change focus once a year. I've spent a career focused mostly on statistics and have slowly been trying to improve my math skills.

[@madhavpr]

​@@DimitriBianco That's great to know. I underestimated statistics in my initial days as a data scientist. Thankfully, I am wiser now. I cannot make a living by ignoring statistical inference and related fields. I've been a data analyst/scientist/ML engineer etc for 8+ years. I use different modeling techniques depending on the business problem and try to read up as much as I can. Despite being in the industry for close to a decade, there is so much I don't know. As far as advanced mathematics is concerned, it's purely for my personal (and perhaps, professional) growth. I try to do math 8-10 hours a week depending on my schedule. I'm working on understanding and solving problems in a) measure theory and b) metric spaces and basics of functional analysis. Let's see how it goes.

[@learning_with_irving4266]

I used to think Quants were wizards...and they are especially explaining it in today's programming heavy environment by producing videos like these

[@sorvex9]

Most of them cant code for shit my friend

[@colinmaharaj]

20 years ago: bayesian arithmetic Today : machine learning

[@-es2bf]

Yes or 10 years ago: Linear regression. Today: AI DEEP LEARNING MACHINE LEARNING ENGINEERING DATA SCIENCE!