A no-formulas, graphical introduction to Copulas and why they are useful, all using simple Python libraries. Join the discussion: dirtyquant.com Github link to the notebook: github.com/tin...
Пікірлер: 140
@ArnauViaM2 жыл бұрын
Hi! Thank you for your work! Let me summarize the process to see if I got it, you start from two sets of data points which you know have some dependence (but not what sort of dependence) your do not know the distribution of each data set by itself either. You look at it and try to figure out the distribution that best approximate its behavior, once you have settled on one particular distribution you use its relevant cdf function to transform both of them to a uniformly distributed observations, then you try figure out the correlation between both uniform datasets, but to give a final twist, you use a function instead of a constant value for correlation, and that function is the copula. Hope I get at least some of it right, thank you!
@dirtyquant2 жыл бұрын
That is 100% correct! Perfect summary
@TheAndiii0072 жыл бұрын
The production and editing quality is astonishing and very refreshing and in my opinion, rather uncommon for quant finance videos. I hope your channel will grow!
@dirtyquant2 жыл бұрын
That is one of the nicest things anyone has ever said to me. Thank you very much. Glad someone appreciates the hard work. :-)
@johneagle43842 жыл бұрын
Videos like yours are very useful. They provide an easy-to-follow introduction to seemingly complicated subjects. Thank you!!
@dirtyquant Жыл бұрын
Welcome mate!
@ghostwhowalks56233 жыл бұрын
WOW - been searching for this for YEARS!!!!!!!!! Thanks for sharing!
@dirtyquant3 жыл бұрын
Glad you found it useful Chet!
@ghostwhowalks56233 жыл бұрын
@@dirtyquant - def! So quick Q - can this be used to generate correlated bernoulli variables? Left more details on your main website(in the forum). Would appreciate your thoughts. thanks!
@dirtyquant3 жыл бұрын
@@ghostwhowalks5623 hi Chet, I saw your message in the forum. Let me reply there.
@ghostwhowalks56233 жыл бұрын
@@dirtyquant - no rush sir! Thanks!!
@silvera11095 ай бұрын
Great video. I loved "loading word" and "let's play a game" - hilarious! Thanks 🤣👌
@souvikchakraborty3095 Жыл бұрын
I discovered your channel today and I sincerely hope that you never stop making videos :)
@dirtyquant Жыл бұрын
Thank you. That is very flattering. I hope to make many more
@kalmanfilter12243 жыл бұрын
Thank you so much for this video ! I had some hard time trying to understand copulas, wish I had seen your video earlier
@dirtyquant3 жыл бұрын
Glad the video helped
@chinhhoang4546 ай бұрын
The channel name is quite misleading, it should be CLEAN Quant!!! Thank you so much for your works!
@JustCheckingMusic3 жыл бұрын
Really nice explanatory video of the concept of copulas! Thank you for creating this content.
@dirtyquant3 жыл бұрын
Glad you found it useful. Appreciate you commenting. Will keep making content!
@rossic77023 жыл бұрын
thanks a lot, I've been triying to understand copulas to apply the COPOD() model of ML and now, wow
@dirtyquant3 жыл бұрын
Makes me very happy to hear this! Well done
@anlsen53713 жыл бұрын
Thanks for your great effort. I hope you will achive the audicence that appreciate your work
@dirtyquant3 жыл бұрын
Thanks for your kind words! I hope so too :-)
@bjoernaagaard2 жыл бұрын
This is my new favorite channel
@dirtyquant2 жыл бұрын
Very happy to hear that John!
@PTEeasy3 жыл бұрын
Thank you! that was excellent. I wish you could also put Sklar's theorem somewhere so we can relate things to that.
@dirtyquant3 жыл бұрын
Glad you enjoyed it. I try to keep things simple and formula free. Might do a video just in Sklar. Cheers!
@KohsheenTiku Жыл бұрын
brilliant explaination, i spend longgg longg time to understand from other stuff, but this explanation was wow!
@dirtyquant Жыл бұрын
Ha! Glad it helped you out. I always struggled too but one day it clicked. Now I want to tell people about it :-)
@shubhangipassi25933 жыл бұрын
You've got a subscriber here for sure. I am in the actuarial field and your videos are very succinct and informative. Helped me quite a bit, thanks
@dirtyquant3 жыл бұрын
Thanks so much. Glad you found the video useful
@vl30.73 жыл бұрын
Definitely deserves more views!
@dirtyquant3 жыл бұрын
Appreciate it! Hopefully the views will come
@17teacmrocks3 жыл бұрын
yes, but most ppl will not understand. it's just like there have been real trader channels on youtube that went nowhere. meanwhile fake gurus have millions of subs
@dirtyquant3 жыл бұрын
Indeed. It’s a real shame.
@HrUssing3 жыл бұрын
The probability integral transform i what makes the CDF uniform right?
@dirtyquant3 жыл бұрын
Yes, you take the original data and transform it into uniform using the CDF of that distribution, which is the probability integral transform.
@HrUssing3 жыл бұрын
@@dirtyquant Alright, pretty cool method. Thank you for the video, definitely the best material i have found on this subject!
@dirtyquant3 жыл бұрын
That’s great to hear. Trying to make this subject accessible was my goal.
@c0mmment3 жыл бұрын
Question - you talk about "fitting" the copula near the end of the video. What do you mean by that? In your example there is no fitting, you just plot one CDF vs another CDF.
@dirtyquant3 жыл бұрын
Hi, so with the gaussian copula there is very little to "fit", the correlation value/correlation matrix is all you get out, but it's not fitted in the traditional sense. With Gaussian all you have is the CDF in the uniform space, and from that extract a correlation. So yes, you are indeed correct, but the method is the same for more parametric copulas, the initial steps are the same. Thanks for watching and commenting. Cheers!
@giovannistephens31073 жыл бұрын
Thanks, man! Please, create the next (more advanced) copula video!
@dirtyquant3 жыл бұрын
Noted. Will do!
@vector27893 жыл бұрын
@@dirtyquant Yes please!
@dirtyquant3 жыл бұрын
Let me thing of a good next step. Maybe some copula simulations. I love simulations :-)
@giovannistephens31073 жыл бұрын
@@dirtyquant Maybe something like a DCC-GARCH vs. a Copula-GARCH. I would love to see both applied to a portfolio optimisation and backtest.
@dirtyquant3 жыл бұрын
Now you talking my language. Let me see what I can do
@mehdiagunaou44243 жыл бұрын
Good video !! you helped me a lot. You deserve more views.
@dirtyquant3 жыл бұрын
Thanks for watching. Glad you are enjoying the videos.
@lade_edal Жыл бұрын
Such brilliant production quality!
@dirtyquant Жыл бұрын
Thank you. I am flattered
@munozariasjm3 жыл бұрын
It was a really good introduction, keep the excellent work on!
@dirtyquant3 жыл бұрын
Thank you. Will do!
@shirishsam084104473 жыл бұрын
You got a subscriber, nicely explained :)
@dirtyquant3 жыл бұрын
Thank you! Glad you found it useful
@RohitKaran0073 жыл бұрын
Very nice and simple explanation
@dirtyquant3 жыл бұрын
Glad you enjoyed the videos
@kanejiang29383 жыл бұрын
Thank you Dirty Quant,You are nice man.Could u make a vidio about how to select copula model?
@dirtyquant3 жыл бұрын
Thank you! Good idea, let me look into that
@corradoforza3 жыл бұрын
Thank you so much, this was very clear and useful! Actually this video save me a lot of time! Keep going 💪🏻💪🏻💪🏻
@dirtyquant3 жыл бұрын
Very happy you found it useful. Just trying to make copulas more widely used as they aren’t complex once you see them under an easy light.
@kubetail1210 ай бұрын
From what I am reading, I am getting an idea of what copulas do, but I am trying to figure how to apply copulas to my problems is the tough part. Plus, I am thinking of publshing my work in a journal that is more measurement science oriented, so I really want to get it right. I am good at math until it comes to proofs and I notice statistics tend use more a mathematician type presentation that say engineering and physics math.
@404username2 жыл бұрын
Hi, could you possibly elaborate why and how you transformed the beta distribution into UDD?
@dirtyquant2 жыл бұрын
Hi, We transformed to Beta distribution to uniform as that becomes the common language that we can use to use copulas. Whatever distribution your data is in, there should "hopefully" be a way to translate it to uniform, that way we have more flexibility in using multivariate distribution. Most Multivariate distributions assume that each of the marginals (the individual pieces of data) follow the same distribution. If Data A and Data B are both beta distribution, then you can go ahead and use the Dirichlet distribution, which is the multivariate Beta. But that if Data A is Beta and Data B is normal? Copulas are the answer. In order to make it happen, we need to translate the data into a format which is common to both of them. That's why uniform comes in. To transform it we just use the Cumulative Distribution Function (CDF) of that distribution. Simple as that. The skill, is to know WHAT distribution our data is in. Hope that helps
@Megal0lbanana10 ай бұрын
Excuse me, but what is the presenter referring to by marginals?
@dirtyquant10 ай бұрын
Hi mate, marginals is just a fancy word for the distribution of the 2 separate datasets. So time can have a certain distribution and money spent a totally different one. These 2 datasets are the marginals.
@devaughntaylor9121 Жыл бұрын
Really helpful!
@dirtyquant Жыл бұрын
Glad it helped
@lukeweidner9110 Жыл бұрын
Wow that plot showing the CDF uniform distribution totally clicked for me. I had read about that before but it didn't make any sense until now
@dirtyquant Жыл бұрын
Very happy to hear you found some value Luke
@tiga765911 ай бұрын
Amazing video! Thank you so much
@dirtyquant11 ай бұрын
You are super welcome
@luzianlechner1884 Жыл бұрын
amazing thanks!
@dirtyquant Жыл бұрын
Welcome!
@yingfeng94043 жыл бұрын
Thank you for your video :)
@dirtyquant3 жыл бұрын
You are more than welcome!
@sylvesterrokuman75202 жыл бұрын
Thank you for your video! It was very informative. I'm in the civil engineering field and doing hydrology studies for my masters thesis, and like you said in the beginning of your video, most textbooks I came across jump straight into the deep mathematical concepts without giving an overview and intuitive understanding of the subject as you have did here. I am currently using R to do this and I haven't learned python very well so do you have the R version of your example and could you share please. Thank you!
@dirtyquant2 жыл бұрын
Hi mate. Glad you found the video useful. Plenty of places to get dizzy with maths, few places to explain it in plain English. Sorry but I don’t use R. It’s well used in the stats community so I think you should be able to take me example in Python and translate it to R. If you use Jupyter you can run Python and R so that would be handy! Best of luck in your studies mate
@ekolytih3 жыл бұрын
good stuff - but can't believe you are running this without a github portal for the notebooks ;)
@dirtyquant3 жыл бұрын
I do! It’s in the video description. Link to Github for all projects used in the channel. Thanks for watching!
@ekolytih3 жыл бұрын
@@dirtyquant who on earth would click 'show more' and find the link lol. That been said, keep good video coming :D
@dirtyquant3 жыл бұрын
@@ekolytih haha! Indeed. I will mention it on my next video so people are aware. Thanks for that
@Anushdf13 жыл бұрын
Your videos are super helpful. Thank you very much for your time! I have been looking for a good book of ML in finance. Do you recomend the one is on your desk?
@dirtyquant3 жыл бұрын
Hi, I would recommend sebastianraschka.com/books/ It's not Finance specific, but it's really well written and easy to understand each of the models. Advances in Financial Machine Learning by Lopez Del Prado is very hard, and in my view not that practical for most people, but you can take a look if you like.
@HeyCurlyBoy3 жыл бұрын
Thank you so much ! this was so well explained!
@dirtyquant3 жыл бұрын
Glad you found it useful
@humarani513 жыл бұрын
Thankyou so much Sir, its really really helpful... They you explained it superb. Well, can you please tell me the about the data you generated? I mean is this R? Or anyother software... Plz help me . I wanna replicate it in the same way as you did . Thankyou
@dirtyquant3 жыл бұрын
Hi. This is all done in Python, using Jupiter notebooks. You can get my code on my GitHub page, link in the video description. Glad you enjoyed the video :-)
@spearchew Жыл бұрын
enjoy the channel but personal preference would be to reduce the background music as you get deeper in to the video, it can be a little distracting.
@pinakibhattacharyya78532 жыл бұрын
That's Halperin and Dixon's recent book on your desk.
@dirtyquant2 жыл бұрын
Sure is!
@taotaotan56712 жыл бұрын
Really thank your excellent video! One small question here: the correlation between gamma and beta distribution changed after the transformation. We can only indirectly control for the correlation by specifying the covariance matrix of the multi-Gaussian distribution. I am wondering if there is a way to directly control this correlation. Thanks again for providing great resources to the internet!
@dirtyquant2 жыл бұрын
Hi Taotao, I am happy you found the content useful. What I am trying to show is that by using correlation, we are assuming a linear relationship between the 2 datasets. This is where we get the corr of 0.72, while actually it's 0.8. If you are happy with this, and understand that the data has a unique structure, then you can stop there. What copulas allow us to do is to use a universal language, the transformation from whatever distribution to the uniform, so we can apply our special copula (which might have strong tail dependence etc). It's just a tool to make our life easier. If you are happy with just corr between beta and gamma, then happy days!
@taotaotan56712 жыл бұрын
@@dirtyquant Thanks for replying! I see what you mean, basically Pearson correlation, which assumes a linear relationship, isn’t a good measurement for distributions like gamma and beta. Instead, other measurements, like spearsman correlation is more appropriate to use here. Thanks again! Love this channel!
@minecraftscienceINC2 жыл бұрын
Intresting video. You assign the scatter plot in the video to be of the Gaussian type - but what about the clustering around (0,0) and (1,1)? Shouldn't the Gaussian copula have a larger grouping at (0.5,0.5)? I am a little confused about that at least.
@dirtyquant2 жыл бұрын
Hi, can you point to where you are seeing this? I did this video a year ago and I can’t remember it all. Thanks for watching
@minecraftscienceINC2 жыл бұрын
@@dirtyquant It is about 13:36 where you show the plot, and at 14:00 you tell that it essentially was a gaussian coupla-like behaviour
@dirtyquant2 жыл бұрын
@@minecraftscienceINC I think our eyes are deceiving us then.
@qlofficial5502 жыл бұрын
Thank you! This is so helpful!
@dirtyquant2 жыл бұрын
Glad it helped
@barsraday9531 Жыл бұрын
👏👏👏
@dirtyquant Жыл бұрын
Thank you!
@tareqahmed1812 жыл бұрын
Thank you for the very nice introduction to marginals and copula! Is there a way to get in touch with you regarding a problem that I have at hand?
@dirtyquant2 жыл бұрын
Sure. Wrote a post on dirtyquant.com Glad you liked the video
@coco-il4gr2 жыл бұрын
Thank you so much !
@dirtyquant2 жыл бұрын
You're welcome!
@user-or7ji5hv8y3 жыл бұрын
Cool
@dirtyquant3 жыл бұрын
Cool indeed!
@Blaze0988903 жыл бұрын
Not sure what we are trying to model here. My understanding is that we are looking to better capture relationship between the variables time spent and money spent. What we are doing here seems to me is trying to model the Normals that were used to generate the uniform seeds. Why do we care about the correlation returning back to 0.8 when the correlation of 0.72 captures the data (albeit, doesn't catch non-linearity) more accurately? I just don't see what the transformation to uniform gives us. It is simply a long-winded way for calculating the correlations between the CDFs of the two Gaussians we started with. I see that in the real world we observe the time spent and money spent and then we can use that to find the correlation between the CDFs of the Gaussians but I don't see how that is useful. What am I missing?
@dirtyquant3 жыл бұрын
Thanks so much for your attentive reply. The core reason to use copulas is to allow you to use different distributions for the marginals, i.e. the individual data sets, and once we have those, allow us to have a non gaussian relationship between them if we want. In my example we have a non linear correlation between the 2 variables, time and money, but by identifying the type of distributions in each, and transforming them to uniform, we now have a linear relationship. We are using a gaussian copula here, because that is all we need here. But It could be the case where the dependence might be really strong in the tails, so big spenders spend alot of time on the site, far more than your average, and now a gaussian copula isn't sufficient any more. As you say, "it's a long winded way to get the correlation of CDF", yes indeed, that is copulas. The reason why finding the correlation of the CDFs, is because you then have the true, non linear relationship between them, which allows you to simulate data and find probability distributions. so when someone spends 30 mins on the site, how much are they likely to spend. The next step after this is to have many variables, each with their own distributions, and then be able to pick the most suitable copula, or type of relationship between the transformed variables. Hope that clears it up. This is a basic example, without using formulas, as copula maths can be brutal for newcomers.
@user-wc7em8kf9d3 жыл бұрын
Woow !!!
@dirtyquant3 жыл бұрын
Glad you enjoyed it!
@vonGameTheory Жыл бұрын
Would love to understand copulas but couldn't watch this video with the music (and even talking DJ) overwhelming it -- how am I supposed to concentrate on what you're saying? I thought it must be an accident but it was apparently intentional. I guess all your videos are like that? Very strange choice -- unwatchable.
@dirtyquant Жыл бұрын
Haha. Sorry I can’t please everyone. Merry Xmas :-)
@LunakSocioAnthroLinguist Жыл бұрын
okay this not semantics in linguistics, cool video though
@drachenschlachter6946 Жыл бұрын
9:26 brain afk...
@samimocni74773 жыл бұрын
Very good explanation! Ps. The music is the background is not needed 😀
@dirtyquant3 жыл бұрын
Glad you found it useful. Gotta have beats! That’s the Dirty Quant style!
@samimocni74773 жыл бұрын
@@dirtyquant Looking forward to next clips with and without music 😀
@dirtyquant3 жыл бұрын
@@samimocni7477 Keep everyone happy!
@drachenschlachter69462 жыл бұрын
Simple introduction.... 16 min video.... There you see how complicated this shit is
@dirtyquant2 жыл бұрын
It ain’t easy!
@drachenschlachter6946 Жыл бұрын
Sorry but is explainied so so bad....really. It was easier to understand the Literatur then this video...
@dirtyquant Жыл бұрын
Lol. Sorry to disappoint.
@drachenschlachter6946 Жыл бұрын
@@dirtyquant disappointment just like my son
@drachenschlachter69462 жыл бұрын
Sorry but your video. Didn't help at all....
@dirtyquant2 жыл бұрын
Can’t make everyone happy :-(
@drachenschlachter6946 Жыл бұрын
@@dirtyquant yes but the description is really confusing...
@neoepicurean37723 жыл бұрын
Please, cut out the background music. For people that watch at a speeded up rate it is a nightmare. I'm trying to watch on x2 and just sounds like someone rattling spoons.
@dirtyquant3 жыл бұрын
Hi, I have looked at the stats and it’s such a small % of people who listen to the video at a sped up rate that I would rather keep my style, with music in the background. But thanks for watching.
@neoepicurean37723 жыл бұрын
@@dirtyquant It's no surprise that people who use sped up function aren't watching, as people that want to watch at x2 will find a video without background music. That's why I was letting you know, as otherwise your content is excellent. With all the online teaching over the past year the speed up function is very widely used now. Most people that I know that use videos as a learning resource only watch videos at sped up rates. It's a life hack that is catching on. My university has gone up to x3 now which is amazing for getting through recorded lectures.
@dirtyquant3 жыл бұрын
Totally understand, but I feel like videos without music are so damn boring, that I would rather lose the speed up crew than bore the rest to death. Thanks for the feedback. Maybe I will upload 2 versions, one with music and one without, so you can choose.
@neoepicurean37723 жыл бұрын
@@dirtyquant Check the stats for yourself, but it's something like 26% of users now watching in sped up mode in 2019. Up 10% from 2018 and expected to be around 50% about now. I study 'futurism', and this is certainly not going to be something that goes away any time soon, People are even watching dramatic content sped up now after Netflix introduced the feature due to popular demand. It's only music that ruins my sped up life! Anyway, do whatever you want. All the best.
@dirtyquant3 жыл бұрын
Cool. I will post both and see which one gets more traction. KZbin should introduce that feature, where you can choose the audio track for people that want 2X. Have a good one!