Backtesting Rayner Teos 1689% Bollinger Bands Trading Strategy in Python [Part II] - OVER 1K stocks

[@romp39]

great stuff, but need to incorporate "out of sample" or "forward walking" way of testing. Since we can endlessly play with params and strats to get something that historically has return many thousands of % returns, but when deployed from today onwards may not get the same results. By going back in time and testing a short period to get params then "walking it forward" in time through unseen data, then we can really start to understand the reliability and sensitivity of our strategies. Also agree with others that backtrader (our other premade engines) can speed up dev time a lot

[@Algovibes]

Awesome comment. Thanks a lot for sharing your thoughts! Agree in general but would rather apply these thoughts on changing indicator parameters over time (depended on e.g. a ML model).

[@anthony-jt2mv]

It's a great feeling seeing you combining two or my favorite things...codes and trading. nice job

[@Algovibes]

I am passionate about both :-) Thanks for watching buddy

[@chigstardan7285]

Another week, another amazing video.

[@Algovibes]

Thx a lot :-)

[@mochamadchairulridjal1514]

Another amazing video!

[@Algovibes]

Thanks a lot Mochamad!

[@Tedwarddddd]

Great video. As others have said, it’s unfortunate that this backtest did not include two things. First, I’d love to see this but with the 10 day hold limit taken in to account so we can see a true representation of PnL instead of dismissal of trades with high loss simply because they exceeded 10 days. There were other trades in there which were less obviously out of the profit prediction which also exceeded 10 days but you left them in. Second, I’d love to see this capable of being in multiple positions at one time. If one was too really use this strategy, each trade would have a predefined total account risk associated with it. I’d love to see you test this same strategy with equity tracking such that each trade was sized according to a total account risk of, say, 1% of the account equity at time of purchase. This would allow for a more real world profit prediction as well as allow for parallel trades. Great video, I learned a lot and can’t wait to try some of the tricks you showed here with panda and sqlalchemy soon.

[@Algovibes]

Thanks a lot for an awesome comment Ted. Appreciate sharing your thoughts!

[@newdata]

I maybe missing something calculaton of profits column is sell price - buyprice but what if it is shorting first... or the first row of tradesdf is sell is true as same for all the concat frames

[@pushkarmarathe4348]

Great video.. Learned a lot.. implemented it. I was wondering if it is possible to have constraints such as 1) maximum number of trades at a time, 2) fixed amount of capital allocated per company (ticker). Any insights will be appreciated.. Happy to discuss..

[@Algovibes]

Hi Pushkar, awesome to read. Thank you! And yes those constraints are possible and more than that make sense!

[@pushkarmarathe4348]

@@Algovibes Thanks for the reply. Good to read that. I am struggling a bit with that part of programming. Would it be possible for you create a video on that? If in case you wish to discuss offline, will be happy to discuss and show my results too. Thanks !

[@billc8761]

Hello. I am very happy I stumbled across your channel. I have subscribed. Keep up the good work :)

[@Algovibes]

Thanks a lot for your subscription buddy, happy to have you on board!

[@MrRuedibuechi]

Wie immer perfekt erklärt, danke!

[@Algovibes]

Danke dir :-)

[@abakrobot]

Excellent video, I watched it) if possible also the interval of 1m to...?

[@Algovibes]

thanks mate! yes thats possible :-)

[@saud5133]

A great video and a great explanation, how to update only the new data in the database without having to re-download all the data again

[@Algovibes]

Thanks a lot buddy. I have covered that in previous videos. In a nutshell: Screen for the max date in the database. Then only pull data from this date on and write back to the database again.

[@eliotharreau7627]

Yes Bro , make the test for live market... 👍💚

[@Algovibes]

Sure! Thanks for watching man

[@ahmetbinatlioglu8848]

Great video as always

[@Algovibes]

Thanks mate :-)

[@RichardPeterShon]

snp has been going up in the past 2 decades. whatever strategy you put, so long as you don:t over stretch your margin and minimize your losses, will give you a huge return compounded. what was the draw down? and how about price slips?

[@Algovibes]

Hi Richard, Thanks for sharing your thoughts!

[@dannyibovnik]

Hey @Algovibes, was ist denn deine Bezugsgröße bei 14:37? Kann man Prozente einfach so addieren? Mit welcher Bezugsgröße würdest du das am Ende multiplizieren, um deinen finalen absoluten Gewinn/Verlust zu ermitteln?

[@Algovibes]

Das sind relative Profite. Verkaufspreis - Kaufpreis / Kaufpreis. Ich hab das hier detailliert erklärt: kzbin.info/www/bejne/nIirgqqniZifhLs Lass mich gerne wissen, wenn du noch Fragen hast. LG

[@ismara6902]

Great video! Would love to see it live!

[@Algovibes]

Thx for your feedback! Will surely come.

[@christophkral191]

In your simulation you are buying for adjusted prices and selling for non adjusted prices. You should have also adjusted the Open prices. Results are not reliable at the moment. But the rest is very interesting ;)

[@Algovibes]

That's correct! Should be done indeed.

[@TradersTradingEdge]

Superb, thanks!

[@Algovibes]

Thx a lot for your comment!

[@philipfiguerres7737]

Very nice video and very good explanation. Since you are evaluating 1000 stocks, how does the code account for buys and sells for different stocks in the same time frame in your "matchtrades" function and profit calculations assuming multiple stocks met the required buying and selling conditions at the same time. Thanks in advance,

[@Algovibes]

Hi Philip. thanks a lot for your comment! It does by matching the trades as explained in the previous video taking a vectorized approach: kzbin.info/www/bejne/o6XKo2ybgr2gnbc

[@tejasappana4097]

In the context used in this video, is there really any benefit in using SQL to store the data over something such as a pickle file? I think a pickle file runs faster (about 4-5 minutes for 1000 stocks), takes little memory, and easy to pull from as well

[@Algovibes]

Hi mate, quite a straightforward answer: Whatever gets the job done and whatever is an efficient way. Pickle file is an option - my recommendation is a database.

[@l.carlito7345]

Rayner is a pro. You should knew that and went for a beer instead ;)

[@Algovibes]

As I said in Part I: Rayners videos are mostly very good to be tested as they have clearly defined parameters. It is just a coding exercise and nothing more than that. Cheers 🍻

[@gthausde]

U should backtest on a similar state in the world, in other words, in times where there was a crisis, inflation and maybe war in the world. This would make more sense and would probably yield more usable data, of course for a short period of time in future (few years).

[@Algovibes]

Good point. Thanks a lot for your input!

[@bryan-9742]

would the code membership have potentially this code in class form? I find that my biggest problems right now is taking the function logic into classes. I am trying to improve upon this and would certainly gain value in a membership if that is the case. Best,

[@Algovibes]

The Drive contains the code as shown in the videos. If you want to see a transition from functional to OOP you can check out my Stock recommender playlist (especially the last part): kzbin.info/www/bejne/p3ikhqpsaNWSq80

[@bryan-9742]

@@Algovibes beautiful. let me get some things worked around. I think i'd like to explore a full membership that I'm going to see if my job will pay for. I'm actually building a pairs trading logic in digital assets. The backtester would be great if I can move my logic into class form. I'm wondering if we pay for the $15/month if we can get any of your time or if you charge separately for that. I'm fairly new in python and have been working with it for about a year and a half within a financial context. feel free to DM me.

[@groundingtiming]

Very interesting! is there a sequence that i should follow with your playlist?

[@Algovibes]

Thanks man. You can just follow the Python for Finance playlist. If you are interested in bringing strategies to the live market the crypobot playlist also might be interesting for you.

[@berndsonalkar2545]

Nice Video ! Had one question : You correct the df.Close via df["Adj. Close"] but you didn't do the same correction for the df.Open price ? (used in df['Sellprice'] = df.Open.shift(-1) )?! In my opinion you should use the same correction for open price too --> ala adj_open = open*adj_close/close It's only a first guess, but I think this is the reason for the -0.6239... you mentioned ;)

[@Algovibes]

The 62% are coming from a stock called SRPT which was dramatically dropping after the buying date. You can check the chart for that stock. Anyhow you have a good point here.

[@berndsonalkar2545]

​@@Algovibes Right, have checked it ... huge gap down in SRPT on 2016-01-15. It seems that SRPT have these huge erratic gaps all over the place. Thanks !

[@haidar901]

Hi algovibes, Would you kindly give us a guideline about how to create a resistance & support indicator 🙏❤️

[@Algovibes]

Hi :) Thanks a lot for your suggestion!

[@haidar901]

@@Algovibes thank you for replying ❤️

[@OuranianCyclops]

Here's my idea of how I implemented it: Do a small hull moving average and look for peaks and valleys, basically when the HMA is going down and then when it's going up, and look for changes (this is how you spot peaks and valleys, -1 for going down and 1 for going up) Then in those peaks and valleys take the close price (or open or whatever you're using), then give it a range like -3 and +3 pips from that price Now look for other peaks and valleys that exist within that established range, if you find 2 or more you got a support or resistance.

[@haidar901]

@@OuranianCyclops thank you very much for the idé, I will sit down and try it 👌

[@OuranianCyclops]

@@haidar901 I forgot to mention to loop backwards so your starting peak or valley is the one closest to the current price

[@hankobrat]

Great stuff! Question: Once you pull the full data into the database, is there a way to just update it periodically?

[@Algovibes]

Thanks man, Yea you can e.g. update the DB every n seconds/minutes/hours/days. I have made an example how you could do it in my stock recommendation playlist as far as I remember.

[@pauls064]

I would like to see this with beta as well and a good comparison to the index. I believe the index as a whole is up well over 100 pct in this time frame?

[@Algovibes]

Good point! I have shown in other videos how you can doing a benchmark comparison. Please check out the videos in the Python for Finance playlist.

[@nicolaschiavo6425]

Algovibes you are a true legend! You r my favourite youtuber, insane content! btw i don't understand why my code is the same as yours but i can t read the html table it gives me a lot of errors even if i ve installed all the libraries and the code is the same , could you think of a potential error im doing? i know i m a total newb sorry 😅

[@Algovibes]

Thanks for the kind words. That's awesome to read :-) Hm.. what's the error looking like? As an alternative you can just export the wiki table as a csv/ excel and read it in with pandas.

[@nicolaschiavo6425]

@@Algovibes the error message is pretty long, but i think the sensitive part is this: "ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)" and it has to do with certificates, but i can't find how to disable this verification for pandas methods. Thank you for your help and time! *edit : i figured that out so no need for you to reply!! thanks again you are the best!

[@ふらちゃん-y6i]

For the past month, I have been learning a lot from your videos and tried coding some strategies by myself. I am using stop loss and I have no issues with live trading code (and you did show how to do in a couple of your videos). However, I have not seen you coding stop loss in any back testing code and I would really appreciate if you could show how to do. I tried numpy functions but it is a mess and also tried "for loops" but it bugs all the time. Thank you so much in advance.

[@Algovibes]

Hey man, thanks a lot for watching my videos! I have covered Trailing stop loss here: kzbin.info/www/bejne/jGfdYoB7aJJjlas and I also actually covered stop loss in older videos e.g. here: kzbin.info/www/bejne/qJTCimmomaekjq8 But I am happy to cover a stop loss mechanism in one of my future videos if that's helpful for sure.

[@mauricioocampo8957]

Thanks for th video. Great content. I am just wondering how reliable are the data from the API? most concerned about the volume data and also if you have any video or you know if it is possible to separate the total volume given by the API into buy and sell volume?. Thanks in advance.

[@mencti]

what do you mean with buy and sell volume? volume is the amount of shares traded (shares changing ownership) in a given range of time, so in order to a trade to happen you need a seller and a buyer at the same time

[@Algovibes]

Thanks a lot for your comment Mauricio, data from the API is as reliable as it is shown in the Binance interface. You can cross check the interface with an API call if you feel more comfortable then.

[@vincentouwendijk3746]

Great stuff! Would you say using SQL for similar datasets has any advantage over pickling (for personal use)?

[@monkyspnk777]

You should never pickle data that you’re saving you pickle it I believe it’s called to serialize it but then you should save it in a flat file or SQL

[@vincentouwendijk3746]

@@monkyspnk777 hi andrew, it would be helpful if you can tell me exactly why?

[@Algovibes]

Hi Vincent, the only use case of pickling I came across yet is e.g. storing a trained ML model with it's trained parameters to use it in another script / environment. I have never used pickling for storage objectives and I have never heard of pickling being used for that.

[@monkyspnk777]

@@vincentouwendijk3746 pickling data could result in short term and definitely long term data corruption. It’s kind of one step too far like encrypting your back ups, when you decrypt it and there’s a problem? It’s OK to use while processing but never save it pickled

[@vincentouwendijk3746]

@@Algovibes I use pickling to dynamicly store OHLC-data. My experience is that, after running several tests, that using pickling is many times more faster than using SQL. For the sake of the argument that its more likely to cause damage to data, I do believe that, but my experience so far is that this is very rare.

[@petrb.9596]

1:37 - with importing tickers like this you will have huge survivorship bias in that backtest. Which makes it just theoretical idea with practical coding. Nothing which can be used as real backtest.

[@Algovibes]

Pretty confused about this comment - I am referring to that exactly some seconds before that timestamp. 00:52

[@ertugrulkuscu721]

Das ist ja krass. Du schreibst deinen eigenen Bot und der traded für dich. Wie gut funktioniert das denn?

[@Algovibes]

Krass ist das nicht, aber schon anspruchsvoll. Du kannst dir gerne mal die Videos in der cryptobots playlist anschauen.

[@sweealamak628]

Currently I am developing on python to fire multiple trades into Interactive Brokers via For loop, verifying indicators and price action one by one. Not the best if I want to attempt Basket Trading. This dataframe method could work in a live trading environment but needs to be executed at speed near market close to fully capture the day's action. Seems quick enough. Any plans to craft this BollBandRSI strategy fit for execution instead of backtest? Cheers!

[@Algovibes]

Thanks for you comment again. I already have a bunch of live strategies in my cryptobot playlist. Be kindly invited to check them out! And yes, I am planning on testing this strategy in the live market (crypto again tho) maybe with some amendments.

[@sweealamak628]

@@Algovibes Nice! Looking forward to see how live trading is done via pandas. Have a great day!

[@ergazipp]

You should use existing back trading library such as backtrader. It generate so much more perfomance data including drawdown, sharp ratio and so on. Also plotting the actual backtested portfolio curve is so much more intuitive for the performance of the strategy.

[@chigstardan7285]

Finally, someone else said it! I've been waiting to see him use backtrader and any creative ways he uses it.

[@NocezHD]

Exactly, was thinking about this for long. Doing things with pandas this way should not even be called backtest. The real goal of a backtest is to to provide us the best insights available, both in data and chart format (metrics, benchmarks, analyzers, equity curves…). Evaluate strategies without all of this would never lend to something useful

[@Algovibes]

I have already commented on that multiple times. Taking a backtesting library comes with limitations. Strategies involving e.g. crossovers are quite straightforward to implement, taking a more nested logic as in this example will be close to impossible to design. I will surely cover Backtesting libraries in the future. Anyhow I won't change my approach - there will be a lot more videos without using Backtesting libraries.

[@ergazipp]

@@Algovibes I think what I'm suggesting here is to add more performance data for a better assessment of the strategy. For example win ratio, profit factor are all very important metrics for evaluation of any strategy. You can have your own implementation for these data but why would you reinventing the wheels. There are lots more backtesting framework out there you can choose from. BTW, this is not a very complicated strategy and can be easily implemented in backtrader. Keep up the good work and cheers.😀

[@NocezHD]

@@Algovibes Respect your position but what you’re saying is not true. Libraries like backtrader allows you to literally code almost every strategy you can code without using it, including the most difficult ones (ex divergences based). The main point is that an approach like yours is educational for sure, but it’s not backtesting at all. Backtest a strategy involves reading through the lines of the several metrics deriving from the backtest itself, returns and drawdown are not enough to confirm an hypothesis.

[@Gius3pp3K]

Great video, as always. Isn’t 5% = 0.05?

[@Algovibes]

Thanks man, didn't I say 0.5%?

[@cooltraderf]

Do you see an advantage in storage the data in a sqlite database rather than just flat csv files in a folder? Another great video, thank you.

[@Algovibes]

Yes I do. You would have 1k files or a very large csv. I recommend Database storage but just use whatever you feel most comfortable with. Thanks a lot for your comment btw :-) Appreciate it.

[@MassiveGamersHD]

Why could you not have 2 trades going on at the same time, they might not even be on the same ticker. There's no problem at being in 4-5 stocks at the same time, maybe it's a good week for many stocks in the Russel 1000

[@jerrylim6381]

Agree. I think that because he sorted the dataframe by the buydates, the ticker is scrambled. This make the double trade as he mention is invalid and so with the filtering. Cmiiw.

[@Algovibes]

You can! In the best case you are working with a limited capital which you can assign with amount of x to a specific trade. Say you have 1k bugs and want to assign 250 each trade.

[@janbiel900]

So essentially the index moved from 1,114.86 to 2,612.41 in that timeframe and you decided to just skip trades that would take longer than some timeframe (this is not correct, they should be sold when the timeframe is reached and the loss of the big trade should be in the total). In essence this strategy with fees massively underperformed the index it was trying to exploit? Nevertheless i learned some Pandas tricks

[@Jerryfromtheblocm]

Sounds about right. The flaw with back testing is you have perfect information. you can play around with the data until you have enough bs to make a KZbin video. For example that -62% trade. It would have been a buy due to buy signal, and a sell after 10 days. which would probably have still incurred a loss even if it wasnt 62%, unless it was some odd company collapsing. Again cant tell since ticker wasn't specified. TLDR don't trade lol.

[@Algovibes]

First of all: awesome that you learned something. That was the purpose of the video. No I am not skipping those trades. I am showing that there are actually a handful of trades which are exceeding the 10 day holding period so it is necessary to include the 10 day check in the buying condition. Regarding the "in essence" question: Hard to say but two things have to be taken care of: - survivorship bias - capital allocation (how much capital per trade) - Adding the 10 days max time to the selling condition @Hgnaix: You have some good points and I hope the "bs" is not referring to my video 😛 Anyhow the -62% is an SRPT trade. You can easily get the ticker information by adding this to the loop: for symbol in symbols: df = pd.read_sql(symbol, engine, index_col='Date') df.Close = df['Adj Close'] df['symbol'] = symbol

[@Jerryfromtheblocm]

@@Algovibes No absolutely not, the "bs" isn't referring to this video, but the multitude of other bs trading videos. This video is actually very educational. nevertheless, it still has its flaws (strategy). I appreciate the time taken to respond tho.

[@pawepastuszko8616]

Hi, can the code be downloaded.

[@Algovibes]

Hi mate, yes it can! Here: kzbin.infojoin Happy to welcome you.

[@ventiladorbueno1846]

Thank you

[@Algovibes]

Thanks for watching mate :-)

[@tomvsn9350]

Good video but test it in the real market with real trades and post your results ;)

[@Algovibes]

Thanks man. I will! (for Cryptos tho) I already made a tutorial on setting the trading bot up if you interested here: kzbin.info/www/bejne/fJ2vYotnbNiqbtE

[@refaamoe]

First

[@Algovibes]

🏅

[@user-io8zt8kj6i]

The sp500 outperformed this lol

[@Algovibes]

Without the intention to defend the strategy: The S&P500 was rising by 132% since 2015. Your claim is not correct. Besides that I would rather compare the Russel performance as a benchmark even tho both indices are highly correlated.

[@kunedroid3446]

Why are you limiting it to only one open position at a time? Is it on the strategy? It goes from 600 trades to < 100 (in 6 years data, btw)... I don't see the reason why you wouldn't open a second position in a different stock if it matches your conditions/strategy (you might have a balance limit but using 1000 stocks to keep a single position seems unreasonable)... Maybe I missed something? It seems like you are treating the whole 1000 stocks data as a single one.. Cheers

[@ivanfitzy]

maybe unreasonable but i think for the purposes of the video he is seeing what would happen if he were to use his entire account balance for each trade

[@kunedroid3446]

​@@ivanfitzy I see, but it doesn't really makes sense mate... this is not how anyone would operate (it would be crazy, IMHO) and it would not make sense to monitor/track 1000 stocks to only invest in one, again - my opinion. There is not much advantage. It would also be interesting to see how many days has it been out of the market, because we are looking at a strategy with 1-day entry (flip) condition. If we have

[@Algovibes]

I am showing both extremes here: 1. Entering every trade and 2. Only enter if we are not in a position. I am not working with capital allocation here and assume that we have either endless funds in the first extreme and only able to enter a single position in the second scenario. What you stated would be quite an interesting question: What would happen if we allocate a certain amount of capital - that might be covered in future videos. Hope I could clarify here!

[@kunedroid3446]

​@@Algovibes I am not sure mate... I am no expert in pandas but I have used backtest.py and backtrader in the past but have a custom/simpler class today. But for instance, you concat the whole dataframe of 1000 stocks without specifying any keys/verify/ignore_index - what happens to trades in the same day and different stocks? (That could be the reason for your 62% loss and others, btw - trying to calculate profit on different stocks/prices). I think you need to track the stock along with the trades, maybe using the keys argument is a good idea. Then you sort the whole thing by date, how are we keeping the information about what entry is for what stock? I am happy with the % profit without thinking on capital allocation, usually not that relevant for backtesting IMO. If you have to travel back in time to change the trade history, it is a good sign that something is wrong on the backtest approach. As I said, I am no expert in vectorization but run it through backtest.py/backtrader for a few of these stocks gives quite a different result (combining 1000 stocks on any backtest libs I know of is a pain and I didn't do it). It might be easier to run it one by one and try to extrapolate from there but quite painful I guess.

[@Algovibes]

@@kunedroid3446 Not the intention to be rude or a wiseacre but you are claiming wrong things here: The concat function is just creating a dataframe where all matched trades are being appended after each other. It is not possible that different stocks are calculated as they are already matched. Please watch Part I again where I am going over what the function is doing. If you want to know which stocks are traded you can simply add this line to the loop - there is no need for a key as every iteration is a particular stock: for symbol in symbols: df = pd.read_sql(symbol, engine, index_col='Date') df.Close = df['Adj Close'] df['symbol'] = symbol

[@aceofwaters]

Bollinger Bands are garbage compared to Nardaya Watson Envelope, late to the party and always out of range. Feel sorry for retail traders still using, you can do better.

[@Algovibes]

Not quite sure if you are serious. Can you confirm? 😄

[@aceofwaters]

@@Algovibes lets just say over 2 years of less than 54%w is garbage when you change one indicator and it goes up to over 65% with about 1/2 the drawdown. You try it, this is way old news for me

[@sancaili7427]

wonder if anyone can help me, at 9:37 I tried but it shows Exception: inputs are all NaN in applyindicators(df) 5 df['Upper'] = df.SMA_20 + 2.5 * df.stddev 6 df['Lower'] = df.SMA_20 - 2.5 * df.stddev ----> 7 df['rsi'] = ta.RSI(df.Close, 30) ~\Anaconda3\lib\site-packages\talib\__init__.py in wrapper(*args, **kwargs) 30 for k, v in kwargs.items()} 31 ---> 32 result = func(*args, **kwargs) 33 34 # Series was passed in, Series gets out; re-apply index _func.pxi in talib._ta_lib.RSI() Exception: inputs are all NaN

[@Algovibes]

Please just double check that part in the video again. I am using a different syntax here (ta.momentum.rsi). Should be solved with that.

[@sazamhashmi]

Your amazing, may God bless you with health and wealth. Keep the amazing work coming. And pls if i could contact you in some way.

[@Algovibes]

Thank you so much man, wishing you all the best!

[@mawhealth5343]

you should defo make a discord server, hmu i can help make it i got alot of experience