Algo Trading - Walkforward Testing

[@andy.robinson]

Thanks Kevin. I've watched many of your other videos where you mention walk forward testing and the importance of out of sample data, but didn't really understand the concepts. This is my lightbulb moment! 💡

[@AlgoTradingWithKevinDavey]

Glad it was helpful!

[@placebo64]

Thank you, sir. After watching your videos I can tell why you're number one and win all the competitions. And you're very generous too. I won't forget that.

[@AlgoTradingWithKevinDavey]

Thanks Nick, appreciate the kind words!

[@Michal_Sobczyk]

Kevin, I'm still not sure about this after reading your book. After WF Testing is complete, what parameters do you actually trade in the future? I would suspect the correct approach is to treat the future as out-of-sample and optimize the parameters on the in-sample period equivalent to the period of time used in WF Testing (eg. 5 years) and then reoptimize at the end of each period. Am I right?

[@AlgoTradingWithKevinDavey]

Thanks for the question. That is exactly what I do. I teach exact details of it in my workshop, since it is easy to do it in a way that gives weird results, but you have the basic idea down cold!

[@tpompeu1]

Hi Kevin! I am from Brazil, I have trading Forex with great results with your methods. Thank you!

[@AlgoTradingWithKevinDavey]

Thanks, your comment made my day!!! I hope your success continues!

[@roym1444]

Found this video while reading your book. Thank you for sharing your process.

[@AlgoTradingWithKevinDavey]

Glad it was helpful!

[@award7502]

After watching several videos on Walkforward testing, I don't get how you then use combined test results. Say if you re-optimize each period and find that in each period you have different settings, what settings then you should use to go live? Do you take the average of those settings and start trading live? I totally understand the need for walforward testing from your explanation, but the actual final use figure is not clear. Could you please elaborate on this final part?

[@AlgoTradingWithKevinDavey]

Let's say you optimize for 2 years, and use the new parameters for the next year. This would be 2 Years IN period, and 1 Year OU period. Jan 1, 2021 you run your optimization over 2019 and 2020 data. You get best parameters from that optimization, and use them for 2021. Similarly, you do this every Jan 1. Hope that helps!

[@award7502]

@@AlgoTradingWithKevinDavey It certainly does, thank you Kevin.

[@hansmeisner5226]

Thanks very much, Kevin, for this helpful concept. If I remember correctly, you used a slightly different approach in another clip, where the in-sample periods all started in the same base year (e. g. 2008-2012, 2008-2013 etc.), as opposed to keeping the in-sample period length constant for all x iterations (e. g. 2008-2012, 2009-2013 etc.). Which version does lead to better live results in your experience?

[@AlgoTradingWithKevinDavey]

Thanks for the comment Hans! I have created strats both ways - some with fixed start date (anchored walkforward) and some with moving start date (unanchored walkforward). I prefer the latter, but both can work.

[@hansmeisner5226]

Thanks, appreciate it!

[@GanovAlex]

WFA seems to be very good think, after reading Pardo book and seeing your video now i'm writing own tester and continuously learning myself of deeper testing understanding. Thanks for your job

[@AlgoTradingWithKevinDavey]

Good Luck! Walkforward definitely has some major pitfalls, and most people do it incorrectly. So be careful!

[@T-nt2cq]

Hi Kevin, once the walk forward testing is done, what are your thoughts on recording all the chosen parameters for all the OOS's, as you have done, then adding them up and taking a simple average of them to use going forward? Thanks

[@AlgoTradingWithKevinDavey]

That is not what I do, but if you tested that approach over enough strategies and it worked well, then that is a possibility.

[@markminnis5595]

What is the difference between walk forward testing and walk-forward optimisation? Do you always want to choose the optimal parameters or is it better to.choose the parameters which are more stable over time or the parameters wich have small sensitivity to changes to parameter change is small?

[@AlgoTradingWithKevinDavey]

Basically they are the same. There are many different ways to perform walkforward, and to define what is "best." You could pick optimal parameters - however you define them - and that could involve sensitivity or stability factors. In my class I teach an approach that has worked for me, my students, and has literally millions of backtests supporting what we do (I just ran a 50 million backtest study in early 2023 to confirm earlier findings).

[@ElvinKennedyLatayan]

Thanks for this , Kevin! \m/

[@AlgoTradingWithKevinDavey]

No problem!

[@leonjbr]

Very well explained.

[@AlgoTradingWithKevinDavey]

Glad you think so!

[@simonhuber6859]

Thanks so much for this! I almost get it ... though what do I do with all the yellow out of sample data? Do I look which metrics prove to work the best, based on the yellow years? And do I have to tweak the metrics again, only looking at them or what do I choose in the end?

[@AlgoTradingWithKevinDavey]

You combine all the yellow out of sample periods into one equity curve, and then evaluate its performance.

[@M19Capital]

so if you first optimise 2007-2011 and test out of sample, you still optimise again for 2008-2012, so what do you do with all the 5 results from different periods of optimisation, I dont think I heard it in the video and its very interesting, do you just take the average of all of them 5 tests?

[@AlgoTradingWithKevinDavey]

You put all of the out of sample periods together to generate an equity curve.

[@garrettolson4137]

Hi Kevin, great video. I purchased your Building Winning Algorithmic Trading Systems book, and have *loved* it, but hearing you explain this out loud really helped me get a better grasp of the walk-forward testing concept. I'm a Forex algo trader, and I'm attempting to build strategies that I can run against multiple pairs simultaneously. In the interest of simplicity, and not trying to overfit/over-optimize, I was hoping to use one set of parameters on all the markets for each strategy. I guess my question is something like: Do you think this is a good idea? Or would you suggest performing the walk-forward test on each market separately, and using each set of market-optimized parameters individually?

[@AlgoTradingWithKevinDavey]

Thanks! You can do it either way, and there are pros and cons to each approach. "One size fits all" will lead to less curvefitting, at the expense of lower profit. "Custom parameters for each" will give better backtest performance, but it might not translate into future performance. One thing to avoid is to cherry pick pairs - many people test on all pairs, then throw out ones that do not perform well, keeping only best ones. That is dangerous.

[@garrettolson4137]

@@AlgoTradingWithKevinDavey - Interesting! Since each pair behaves a little differently, I would have thought that picking pairs that performed well was similar to having a strategy that works well on gold, but not on crude oil, for example. So, in my case, initial limited testing revealed that this particular strategy performed dramatically better on JPY pairs (which did very well) and most of the Majors (which did pretty well), vs. on CHF pairs (which performed absolutely terribly). If you were developing this strategy, would this cause you to throw the whole thing out? Or would you move forward to walk-forward testing on just the JPY/Major pairs?

[@AlgoTradingWithKevinDavey]

@@garrettolson4137 - Hard to say what I'd do, since I do not have all info in front of me. Too many factors to give a solid "thumbs up" or "throw away."

[@T-nt2cq]

Hey Kevin, When testing 1Lot in currencies, on a pair which doesn’t have the USD in it, eg CHFJPY - would you test with the position size as100,000 yen or 100,000 USD? Thanks

[@AlgoTradingWithKevinDavey]

You could do either, it depends. Maybe the JPY /USD conversion is important to you, for example. Then you should include it.

[@SailorPilot]

Great content! Thanks. 🙏👍

[@AlgoTradingWithKevinDavey]

Glad you liked it!

[@QuantumSwift]

Loved it really appreciate it thanks

[@AlgoTradingWithKevinDavey]

Glad you enjoyed it

[@PrasanthChandraLingala]

How is it different from just optimizing on trailing few years/months and use them for a year/month and repeat?

[@AlgoTradingWithKevinDavey]

That is the general concept.

[@mindaugasb6962]

Good methods, but i invited my own. I call it 'real life' or `system survive' method. Another good but more difficult is optimization on synthetic data method. But its another story. Thanks for interesting videos!

[@AlgoTradingWithKevinDavey]

Thanks for the comment. I'm curious, what is the "system survive" method?

[@mindaugasb6962]

1. Develop system any method you prefer. WF, Brute force, etc. 2. Put system result to your database.. 3. Go back to that system after one year.. If system survives different market events like market crash this March, etc. your system passes "survive stress test". ;)

[@petersmith730]

Is there a huge difference between using 5 years instead of 4 years of in sample data per window?

[@AlgoTradingWithKevinDavey]

There could potentially be.

[@leonjbr]

Hi teacher. Do you have an estimate (even if its a rough one) of what percentage of plain-vanilla strategies "survive" the WF analysis?

[@AlgoTradingWithKevinDavey]

Somewhere between 0-5%

[@leonjbr]

@@AlgoTradingWithKevinDavey Here's the translation of your text into English: "Thank you, professor, that response relieves me, because recently I took the time to write a program in Python that performs Walk Forward analysis, and I was surprised that the inmense vast majority of the strategies I tested did not pass the test at all. If the initial strategy was practically a straight line, when trying to pass the Walk Forward, what you get is practically noise and in many cases even a negative capital curve. Only one of the stretegies yields a positive result after WF analysis, but by very little, with a lot of noise and with a shape much less attractive than the original curve. I thought the program was poorly made, but after your response, I see that it's not, that probably the code is fine, and what's happening is that Walk Forward is a very difficult test to pass. Which, in reality, is positive because it eliminates a lot of strategies that appear to be good but wouldn't work with real money. Thank you for your response and for all your videos."

[@leonjbr]

@@AlgoTradingWithKevinDavey Thank you, teacher, that response relieves me because recently I took the time to write a program in Python that performs Walk Forward analysis, and I was surprised that the vast majority of the strategies I tested did not pass the test at all. If the initial strategy was allmost a straight line, when trying to pass the Walk Forward, what you get is practically noise and in many cases even a negative capital curve. Only one yields a positive result, but by very little, with a lot of noise and with a shape much less attractive than the original curve. I thought the program was poorly made, but after your response, I see that it's not, that probably the code is fine, and what's happening is that Walk Forward is a very difficult test to pass. Which, in reality, is positive because it eliminates a lot of strategies that appear to be good but wouldn't work with real money. Thank you for your response and for all your videos.

[@AlgoTradingWithKevinDavey]

@@leonjbr you are welcome!

[@leonjbr]

@@AlgoTradingWithKevinDavey BTW teacher: do you agree with the explanation in my previous post?

[@DG-nj1hf]

Hello, Kevin! I looked at results of Pardo Capital.. they are not really that impressive.. doesn’t it put credibility of WF under some doubt ?

[@AlgoTradingWithKevinDavey]

Thanks for the question. I think what it tells you is that walkforward testing is not a Holy Grail. There are a lot of other items that influence final results - walkforward is just a part of it.

[@DG-nj1hf]

Your reply makes perfect sense .. But it means that such a prominent figure as Mr. Pardo is lacking those "a lot of other items" .. Hmm .. I'll keep my R&D going on .. Anyway, apart from that - some mention that Monte Carlo is also quite misleading .. such as in this article (e.g. Case 8) - quote 'When a strategy is developed via data-mining, Monte Carlo analysis is not useful in determining whether it is robust or random.' www.priceactionlab.com/Blog/2015/05/fooled-by-monte-carlo-analysis/

[@hermanspiteri9929]

Gold!

[@AlgoTradingWithKevinDavey]

Thank You, and thanks for the photos!

[@hermanspiteri9929]

@@AlgoTradingWithKevinDavey =) my pleasure, sir.