Model Validation: Detailed Process

[@yiweizhang286]

Thank you Dimitri!!! I have a model validation interview tomorrow, this video is a lifesaver!!

[@leonescobarvallarta3691]

Hi Dimitri. Amazing content and your video is super accurate. Tbh I never considered Model Validation a role/path for Quants, I used to think they were only into statistical arbitrage or HFT. I spent around 4 years in validation in two global banks and (based on my experience) in validation everything was super structured, so 99% of the time you ended up being able to follow the same steps and techniques. You had these frameworks which should be followed religiously as recipes for each type of model, data, risk, etc. Making it look very repetitive, operational. Which leads to a second point, that in all the validation teams I worked on, colleagues with more technical backgrounds (maths/stats/act sci) tended to leave their role after just one/two years and seasoned validators were mainly econ/finance grads with decent understanding of the theory but great skills to apply techniques and follow methodologies (which again resulted in lack of flexibility to use something different, let alone more sophisticated). Executives knew about this rabbit hole and I remember they decided to change the name of the whole area to something like Center of Data Science and Modeling (or some bs) in order to make it more appealing. Not sure if it worked as I moved to consulting more than 10 years ago, so things might be a bit different now.

[@alperaydyn]

Teşekkürler.

[@DimitriBianco]

@@alperaydyn thank you!!!

[@nicolasjorquera8465]

Damn, just got hired as an Associate Model Validation and this video is gold! I’ve commented before but thanks! (Also if u have any resources that u think would pertain specifically for time series credit risk validation it would be much appreciated. Currently rewatching your time series videos). Keep up the great work sir, thanks to u I was able to join this field with a Data Science background

[@tekaligngelan1318]

Hi Dimitri. Excellent video as always. Now it has been a year since I started working in model validation. This is the best content on the subject matter. I learned a lot working with you and your channel. Thanks my friend.

[@DimitriBianco]

Thanks Tek! It's nice getting industry feedback.

[@Iamine1981]

I was most recently in a role where we were outright told by the Senior Management that replicating models is a NO/NO. Most of the validation focus was spent on cosmetic aspects such as writing nicely presented reports, and being finnicky with the words. Each validation section read more like a novel than actual quantitative findings with supporting evidence. Needless to say many in my team did not understand the basics of our Production Model, the theory behind it, nor its internal implementation structure within the firm. As a consequence, clear and well-targeted tests could not be developed/streamlined into an automated framework for re-usability. This was by far the most soul sucking role I have ever had in Model Validation.

[@DimitriBianco]

I had the same experience when we had a leadership change. I gave it a year and then left.

[@Iamine1981]

@@DimitriBianco In your experience as a model validator, would you say a lot of the work was on replicating Production models in order to have an alternative to benchmark against, or were you also forced in practice to rely on the suite of "tests" provided to you by the Model Owner/Developer? I found such tests to be unsuitable as they were cherry-picked to specifically make sure they didn't address the weaknesses of the model. Another major issue is how to raise challenges to models without having the Owner push back in a non-collaborative fashion. Appreciate the content of your channel by the way, as you address many valid points about the day-to-day life of Model Validation.

[@ninjasuper8]

Best quant mentor ever! 🥳

[@DimitriBianco]

Thanks!

[@pankajthepankster]

Thanks a lot Dimitri for the video !!!!..... This question was always in my mind for which I couldn't find any good answers .

[@DimitriBianco]

Thanks! I'll be making a video on what I do daily as well.

[@cristina5593]

Hi Dimitri! Thank you for such a thorough overview!!

[@oscarms1021]

thank you very much for making this amazing video!

[@jessefreitag5374]

Hi Dimitri-- another great video! Tossing up different Master's programs and a program that would allow me to move into a PhD from a Masters if I would like. This said program is known for going more the traditional mathematics/statistics route rather than the MFE route. One can say it is a math degree geared towards quantitative finance, and not the other way around. Would you think that the more statistics in a degree is better towards a Model Validation role rather than a MFE program? Sorry if the question is phrased poorly

[@DimitriBianco]

Yes, more statistics for model validation is better.

[@jerome8471]

Hey Dimitri. Thanks for your videos. They've been super helpful in my journey. I had a question if you don't mind. How does model validation differ on machine learning models? It would appear to be more involved, but I've seen many solutions like DataRobot provide tools to assist in validation that seems to make it easier.

[@DimitriBianco]

I'll create a video response so I can add more details.

[@jerome8471]

Awesome! Thanks!

[@velvetbreeze1464]

How can I learn model validation?

[@bee123-r1p]

Thank you very much for sharing your experience!

[@DimitriBianco]

No problem

[@spinLOL533]

Looking like a real scholar with the beard sir

[@shukailu6731]

Hey Dimitri, I have been a long-time fan of your videos since 2018. I am finishing my Masters in Finance soon with a minor in Quant Finance. I have internship experiences as a developer at a startup FinTech firm and credit risk consultancy at Murex. Do you think I stand a good chance to work as a model validator in Singapore if I brush up on my statistics?

[@DimitriBianco]

I don't know too much about Singapore's quant markets but either way I would study and apply.

[@Wheytt]

Hello Dimitri! If you had to do it again, which undergraduate would you go into? Thanks

[@DimitriBianco]

If I had to do it over again, it would probably be statistics or computer science.

[@zirokl]

Awesome content brother

[@DimitriBianco]

Thanks!

[@evans8488]

Great stuff

[@danielmusah8851]

Great Job!!

[@Channelofdk]

Interesting, I thought model validation were just the folks that make me do stuff?!