Financial Machine Learning - A Practitioner’s Perspective by Dr. Ernest Chan

[@RaydelGomez-m2j]

This video is pure gold, a gem buried in the immensity of youtube, this is a lecture I would pay for attend to, and it is here totally free!! I don't really understand how it does not have more views.

[@michaelmcfarlin832]

Because most people just want a way to turn $10 into $5k in a week. Those videos get views.

[@marcc16]

0:09: 📚 The speaker, Ernie, discusses the application of machine learning in finance and the challenges it faces. - 0:09: Ernie has a background in physics and transitioned to finance, working at firms like Morgan Stanley and Credit Suisse. - 0:22: Ernie is now a managing member of QTSCapital Management and a founder of Predict. - 0:27: Ernie has authored books on quantitative trading and made contributions to the quantum finance community. - 1:44: He discusses the difficulty of extracting value from applying machine learning to finance. - 5:01: Overfitting and the complexity of models are challenges in applying machine learning to finance. - 6:57: Advances in avoiding overfitting have made machine learning more useful in recent years. 7:26: 📊 Machine learning is becoming more popular in finance due to advancements in techniques, such as dropout and random forest, that help overcome overfitting and make the models more transparent and interpretable. - 7:26: Advancements in machine learning techniques, like dropout and random forest, have helped overcome overfitting in finance. - 10:55: Feature selection is a powerful tool in machine learning that ranks and explains the importance of variables, making the models more interpretable. - 13:07: Machine learning models should be primarily used for risk management and capital allocation, rather than as a primary signal generator. 14:17: 📊 Using machine learning to predict the financial market directly is unlikely to succeed due to competition and market evolution, but it can be successful in predicting private information and optimizing trading strategies. - 14:17: Machine learning programs competing for the same predictions in the financial market would have arbitrated away any detectable patterns. - 14:41: Applying machine learning to fields like radiology is different as the target (e.g., cancer) doesn't evolve to avoid detection. - 15:46: Machine learning programs in the financial market are unlikely to beat those used by large firms like Citadel or Google. - 16:18: Using machine learning to predict private information, such as the spread between stocks, can be successful as it is not publicly available. - 17:40: Machine learning can be useful for risk management and capital allocation in trading strategies. - 18:00: Traditional quantitative strategies focus on modeling prices and fundamentals, while machine learning is more suited for alternative data. 20:46: ✨ Machine learning models in trading can handle non-linear dependencies and provide probabilities for trade success, but assessing statistical significance is challenging. - 20:46: Traditional quantitative models cannot handle non-linear dependencies, while machine learning models can. - 21:57: Machine learning models are difficult to replicate, reducing alpha decay. - 22:04: Machine learning models are opaque and difficult to understand, making it less likely for two people to build the same model. - 23:09: Machine learning models provide probabilities for trade success, allowing for better capital allocation. - 23:46: Traditional quantitative models provide buy or sell signals without indicating the likelihood of success, while machine learning models provide probabilities. - 24:05: Assessing statistical significance is challenging in traditional quantitative models, but machine learning models allow for generating multiple backtests to assess significance. 27:43: 📊 The most difficult and time-consuming step in financial machine learning is financial data science, which takes up 80% of the time in constructing a strategy. - 27:43: Financial data science is the most difficult and time-consuming step in financial machine learning. - 28:06: Financial data often contains numerous problems, such as look-ahead bias and parameter adjustments. - 31:23: Processing raw data into useful features requires domain expertise. - 32:32: Machine learning itself is a solved problem and can be easily implemented. - 34:01: Constructing a trading strategy and backtesting it is a crucial step. - 34:20: Assessing the statistical significance of a strategy is necessary. 34:33: 📊 Using machine learning in finance requires human intelligence and expertise in converting predictions into portfolios. - 34:33: The human intelligence part of financial data science is less difficult than the first part, which is financial data analysis. - 35:02: Converting predictions into portfolios requires domain expertise and following standard routines. - 36:26: Most financial data is not stationary and needs to be transformed before using it as a feature in machine learning models. - 37:28: Linear and logistic regression models are commonly used in financial machine learning, while deep learning models are not suitable due to limited data. - 38:18: Predicting the profitability of your own strategy, rather than the market, using meta-labeling can lead to successful machine learning applications in finance. - 39:40: Machine learning models can provide valuable insights and signals for trading decisions, such as detecting market risks and suggesting strategy adjustments. 41:37: 📚 Machine learning in finance requires domain expertise and is not a replacement for human traders. - 41:37: Feature selection is important in explaining investment losses. - 41:48: Machine learning can accurately predict macro variables. - 43:14: Stationarity is necessary for time series models to make accurate predictions. - 45:07: Machine learning models can handle both numerical and categorical data. - 46:33: Machine learning in finance requires deep domain expertise. - 47:55: Worship of deep learning is a common mistake in financial machine learning. 48:25: 🎯 Deep learning may not be effective in solving problems with limited data and directly predicting returns, but it can be useful in generating simulated price years and providing a more nuanced understanding of the market. - 48:25: The blind worship of deep learning can be detrimental to financial machine learning. - 48:42: Deep learning may not succeed in directly predicting returns or competing with well-funded organizations. - 51:02: Capital allocation can be improved by using machine learning to provide a more sophisticated expected return as an input. - 52:50: Traditional quant finance methods provide static probabilities, while machine learning offers dynamic probabilities that vary with the input. - 54:01: Recurrent convolutional neural networks and other deep learning methods can be useful in reducing the need for engineered features in time series data. - 54:18: Deep learning methods may not eliminate the need for engineered features, especially for cross-sectional diversity. 55:15: 💡 Reinforcement learning can work well at high frequency for trading, but there is no convincing evidence that it works in longer time scales. - 55:15: Different kinds of inputs are necessary for successful predictions. - 55:22: Domain expertise is required for making successful predictions, not just deep learning. - 55:49: Reinforcement learning can react to people placing orders on the order book, making it useful for high-frequency trading. - 56:33: There is no convincing evidence that reinforcement learning works in longer time scales. - 56:39: Reinforcement learning has not been successful in longer time scales. - 57:06: Predict Now AI is a no-code financial machine learning tool provided by Dr. Chan's company. Recap by Tammy AI

[@entratedlikensatace]

This trader is so pure and transparent... Much love and appreciation.. From south African

[@pochl.614]

One of the most insightful videos about ML for trading I've ever watched. Definitely deserve more views.

[@bonadio60]

Just fantastic!! 57 minutes of pure great content. Thanks so much to Dr Ernest Chan and the guys who interviewed him.

[@fminc]

The MAN Dr Ernest Chan. Thanks for having him. People are really missing out.

[@valentinfontanger4962]

What I love about this presentation is that you can feel that he actually tried all of these techniques and can tell you what works and what doesn't. I can almost feel je is still pissed that you cannot win trying to generate signals using the model.

[@ToniEsteves]

Thanks so much to Dr Ernest Chan. A valuable interview.

[@pauldacus4590]

Chan has nailed what will differentiate winning ML from losers: The ability to take the input data and apply creative transformation on that input. "Dumb" linear models on raw, widespread and well-known data just won't work (moving avg's, daily % return, etc), because everyone is doing it, and the Big Boys have long since abandoned it because it has stopped working long ago.

[@colekinyon2267]

I believe that the features like moving averages, etc. should be transformed insofar as making units for the output consistent. As an example, for financial time series data (which are highly circumstantial and typically don't contain well-defined regularities, at least short-term) if you are trying to predict output of Close Price using input consisting of Open Price, you should not apply a transformation of Transformed Open Price = e^{Open Price} even if it causes more model assumptions to be satisfied because this in turn might produce predictive output that are not remotely indicative of the actual output. I therefore think that for financial time series data the features are better off unchanged

[@edrodven]

Excellent presentation Dr. Chan at approximately minute 31 you make the asseveration that you don't trust anyone, due to the fact that originators can change data. Being that DLT can assure the data integrity we posit that an oracle network running on a private federated DLT could solve this challenge.

[@patryknextdoor]

if ML is better to be used for risk management and capital allocation ,then what piece of technology is suggested as the primary signal generator?

[@kleemc]

I found one way to avoid the reflexivity problem as a retail investor is to avoid highly liquid big markets. Focus on small assets or assets in foreign countries where the big boys either can't play or isn't worth their effort to play. Liquidity of such assets will be big enough for retail traders but not for institutions, therefore avoid competing with all the PhDs in the institutional quant funds.

[@michaelbulley795]

This is an unbelievably powerful and valuable video. Really great.

[@ashxos]

The knowledge shared in this video is unimaginable. Grasping it is big task.

[@phanquochung3924]

great presentation, the great questions from the host. Thank you so much, top-notch content

[@BlackSwan-sq2iw]

Great content. Only disagreement is the outright rejection of deep learning. I agree that simply throwing raw data into DL models would not work, but the same is true if you do that with random forest. Feature engineering is crucial. I also don't think the lack of data is the show stopper for DL in trading. There are lots of high frequency market data available nowadays (yes you gotta pay for most of them). Plus data generation/augmentation techniques are widely available. So quantity of data isn't an issue imo, but instead a properly feature engineered dataset is. In addition, there are many other important pre-requisites like noise reduction, regime change detection, etc.. that would be needed to make any model work, DL or traditional ML. Another advancement in DL that is very useful in my opinion is stochastic deep learning model, which can offer the ability to not only make a prediction but also the confidence in the prediction. There are many open-source stochastic DL libraries around, eg. TensorFlow Probability to do that quite easily nowadays.

[@colekinyon2267]

I agree with everything you said except for the feature engineering. For financial time series data (which are highly circumstantial and typically don't contain well-defined regularities, at least short-term) you have to be careful in doing that, because you are essentially artificially altering the way in which the input data is fed into the model; this in turn might produce predictive output that are not remotely indicative of the actual output. I therefore think that for financial time series data the features are better off unchanged

[@anatolyalekseev101]

@@colekinyon2267 Excuse me, but wtf are you talking about? The more 'alternative ways' of looking at data you provide to the model, the merrier. And for timeseries data it's a must to create TONS of, say, numaggs over all kinds of running, expanding windows features.

[@manup9575]

Deep Learning on order book (in conjunction with several other variables such as z-scored IV, HV, etc) provides execution edge.

[@jeroenvermunt3372]

Finally, fractional differentiation. Why does nobody mention this

[@drvlasov5543]

That video was great. So many insights!

[@mikiallen7733]

Just one question I forget to pose is regarding the cpcv algorithm , do you apply this technique of back-testing for each potential strategy during the development phase in your fund Ernie ? If so what are the outcomes you see till now

[@mediarockit]

@36:55 bad sound. Price of a [spider]?

[@mickaelburguet4686]

I didn't really like Ernest Chan's books too much, but this video is just great

[@rich111296]

What was the other platform he mentioned after QuantConnect around 34:28?

[@MuhammadAlfiansyah]

Wow 0 comment on this. The knowledge is huge in this video.

[@jameschen2308]

Honestly tho!

[@spearchew]

great video. i note several of the challenges of financial data science are quite equities specific... just one incentive for considering other asset classes (e.g. futures).

[@alexeygrom1834]

tough quants have such junky internet connection what a shame ! but anyway i wanted to ask if that metalabeling mr Chan talk about is just like hidden makov chains market regimes ?

[@upstandingcitizen3877]

“Do you REALLY think that your ML algo can beat Renaissance’s ML algo??” Well not with that attitude, Ernie.

[@valentinfontanger4962]

I took it personal when he talked about the worship of deep learning

[@mikiallen7733]

Thanks Chen for the work you have done so far in educating the community , however one thing am quite surprised that you have mentioned is the part of modelling / strategy development process using ML framework which is regarding the uniqueness weighting and sequential bootstrap procedure so in your Experience Ernie do these concepts have marginal benefits in terms of more accuracy of say RF model or higher returns with and without incorporating them ? As I can see you are in favor of metalabeling and feature selection techniques than any other concepts in Marcos laprados book . Your response is highly appreciated

[@johanlarsson9805]

2006 it was a fringe activity?! I doubt it, in 2009 when I learned a lot more about it it already felt like an insurmountable wall of established practices so there was no way of getting a piece of the cake.

[@karlbooklover]

forwarded to the Reinforcement learning part and have to disagree, RL is way more powerful in my experience than other methods, and I have tried them all

[@anatolyalekseev101]

would be nice if you could share main ideas & comparison details in a medium article or similar... RL in finance seems very lucrative but I decided to try 'classic' ML +reasonable trading policy first )

[@GohOnLeeds]

Brilliant, Sir!

[@positivelifei]

Dr. Ernest 👏 well job done

[@constantinefourier_bai850]

impressive presentation

[@honghaiz]

Great video

[@marcelfsilva]

Why only ML, not Computational finance ?

[@pineapple3832]

because ML can do anything traditional computation finance can do but better

[@LSl0]

9:30(((, but the general idea is great if, to be honest in algorithm applying!

[@outofbounds9479]

This guy is epic

[@zashkata]

BIG BUCKS! :)

[@dimitrijmaslov1209]

.thanks.

[@fitybux4664]

Alpha decay as new machine learning experts enter the field is real. So the question is - what does Dr Ernest Chan even gain from any of this? 😆

[@anatolyalekseev101]

Imho "capital allocation, not primary signal generation" thing is a BS. It's just a different name for the same thing. Say, I have 3 strategies that normally get allocated $1 each. Then at some moment my ML model suggest to allocate $100 to model #2. I can, of course, call this "capital allocation", but it's in essence a signal generation as well, 'cause the rest of allocations are close to noise. But maybe I misunderstood Ernest and he meant allocations like 0.3, 0.3, 0.4? But hard to believe such high % can be given to strategies that are not expected to work well. Also predicting "whether certain trading parameters will perform well today' seems like, still, indirect prediction of market movement. It can't be other way round, 'cause every trading policy should be based on expected asset price movements. Having a 'certain parameters' filter works 2-ways: it reduces train dataset size for ML (which is bad), but it theoretically allows ML to focus more on particular search space (which is probably good, as the model transitions from jack of all trades, master of none, to master of some). Which force prevails, is a matter of experimentation and asset/TP subtleties.

[@garantprontime9172]

You guys should join the subreddit: www.reddit.com/r/mltraders/ or visit the Website: mlalgotrading.com for more sources about Machine Learning and/or Algotrading. Both highly recommended as I'm using it daily.

[@richoffks]

S/o to Chan 😂 you know it’s some correct mathematical info when a Chinese guy is saying it 😂

[@vishalnangare31]

34

[@HenriettaKerr-g1u]

Perez Cynthia Lewis Dorothy Harris Kevin

[@petivel]

so painful

[@monanica7331]

BTC for $75K by end of this year& Control of The Currency is already Decentralised And now the China disruption would simply Decentralise the Mining setup for the better

[@igor-fk3vv]

Lol, yeah no, didn't happened.

[@swmaidebbarma4702]

@@igor-fk3vv 🤣🤣🤣

[@hardy2175]

She meant 7.5k.

[@henrifritsmaarseveen6260]

people that can automate trading must be able to make a trade in 5sec.. then and only then you can automate it with ML dont predict the future price .. you need to look more to when to trade and if you buy or sell All the ML gurus that predict the price are fake , because they dont understand the problem and there by never will find the answer.

[@anatolyalekseev101]

on the contrary, imho, successful trading is not possible without forecasting (even if it's not called by that name).

[@cherub6723]

ml,ai, will never win the game of trading

[@pineapple3832]

it already has, over and over, look up renaisance technologies

[@gandaslifeadventures7708]

You will lose money unless you’re Jim Simmons. Lol.

[@Maximus18.6]

The only thing that I've got from this video is the Chinese speaker repeats many times how difficult and extremely difficult is predicting stocks price and he promotes his web page. This video is purely advertisement, not even a single line of code.