Algorithmic trading in Python: Cointegration and pair trading

How to implement the logic of cointegration and statistical arbitrage in Python? Today we are building from scratch our own trading bot based on cointegration and pair trading, utilising data parsed from Yahoo Finance and Python statistical packages, and looking at how we can simulate a simple real-world pair trading strategy in Python.
Don't forget to subscribe to NEDL and give this video a thumbs up for more videos in Python!
Please consider supporting NEDL on Patreon: / nedleducation

[@NEDLeducation]

You can find the .ipynb for this video and some additional materials here: drive.google.com/drive/folders/1sP40IW0p0w5IETCgo464uhDFfdyR6rh7 Please consider supporting NEDL on Patreon: www.patreon.com/NEDLeducation

[@slothner943]

Once again... Amazing!

[@Jtking3000]

Truly the best platform around for distant learning in business, finance, economics and much much more.

[@NEDLeducation]

Hi Jason, and many thanks for such kind words! More videos in a similar spirit are on their way :)

[@_stxf5354]

Holy shit!! The best channel I have stumbled across for algo trading.:) Cheers!

[@pasduroc5422]

Wahou, Incredible ! Thank you for sharing so much information !

[@BlackSwan-sq2iw]

Wonderful tutorial. Thank you.

[@andrewtate7938]

Much love from London bro you are amazing!

[@paulmuller7788]

Wow i am so glad i found your channel. You have really good content and finally a serious algorithm trader

[@Kelevra921]

Amazing tutorial!!! Thank you very much!!!

[@prisiv]

Awesome video, very clear and intuitive to understand. looking forward to more python algorithmic tutorials!

[@NEDLeducation]

Hi prisiv, and many thanks for the feedback! Will definitely make more videos on algorithmic trading in Python in the near future!

[@riccardoronco627]

as mentioned in the prev video, if you subtract the returns each day, you ASSUME that the notional size is the SAME. So each day you need to compensate for the notional changes on both stocks and put them identical (or nearly identical if you use a minimum threshold). Thank you for your amazing work!

[@denisbaranoff]

Awesome! You have to demand fee for your perfect job and perfect english for instance in patreon 👍

[@deniswolf1846]

Good afternoon! Your channel gets more interesting with each new video! The topic of Python programming is very relevant and extremely useful in the world of finance. It will be cool if there are more similar videos!)))

[@NEDLeducation]

Hi Denis, and many thanks for your feedback! Planning to do more videos on similar topics with Python code tutorials in the nearest future, so stay tuned! :)

[@quant-prep2843]

@@NEDLeducation and please get rid of excel, lol its hard to understand from excel

[@tratkotratkov126]

@@quant-prep2843 please don't get rid of Excel as it is great prototyping tool which facilitates the transition to the new concepts !

[@makarioss]

thanks alot!!!

[@onda4165]

Great video! Thank you. The variable "a" in minute 13:15 is defined as Average(S2-b*S1). Maybe I am wrong but I think it should be implemented with 2 averages, like: Average(S2)-B*Average(S1).

[@NEDLeducation]

Hi, and glad you liked the video! As for your question, the two formulae you presented are equivalent, so it can be implemented both ways. Hope it helps!

[@jadecapital]

awesome video! Would have been more cooler if you added kalman filter

[@ericboulet1060]

Hi @NEDL, truly great content, thank you. I was hoping you could do an equivalent video to this one for Excel. I've watched the original Excel cointegration video, and as many people pointed out it relied on in-sample data. Thanks!

[@NEDLeducation]

Hi Eric, and glad you enjoyed the video! I might do it at some point, however all the sample manipulations that are very easy to do in Python are quite cumbersome in Excel. This is the very reason why the original Excel video showed a simplified technique.

[@alimbhanwadiya]

OMG! Why do you only have 6k subss!!??

[@algoudemy7437]

Hi @NEDL Thanks for this very illustrative video and code... I had a clarification to seek re this line of code, which I was hoping you would help resolve : gross_return = signal*returns[stock2][t] - signal*returns[stock1][t] This portfolio return calculation assumes we take equal weight and opposite positions on stock1 and stock2? However the initial stock prices and subsequent positions at the time of the signal may not be the same? Eg the first time signal is non-zero, the prices are respectively 140.22 for stock1 and 66.67 for stock2. Wouldn't the gross_return calculation for the portfolio then have to use a weighted equation, weighed by the positions?

[@avinashmishra6783]

Hello, great video as always. I have a request for a video of Kalman filter for cointegration in python.

[@lyubomirgrozdanov914]

Thank you Savva! Do you have something for energy market?

[@user-ju2mn7qx1r]

Good job!!If i just want to implement long strategy,which lines can i change it?waiting your adivise hopefully,😃

[@rajeshmanjrekar3614]

great video, but would it be right if to take the adjusted price rather than the Closing price, thanks a tonne once again for the video!!!!

[@MinerH2O]

Hi. Great video on python and cointegration. Thanks m you. Question, in your simulated trading, are you trading every day? OR, only when the signal changes from long stock1/short stock2 to long stock2/short stock2? I'm assuming you only trade when the signal tells you to change positions. please help. thanks

[@ik4rus2k]

i don't get the calculation of the gross_return: "gross_return = signal*returns[stock2][t] - signal*returns[stock1][t] " Why do we subtract the return on stock 1 from the return on stock 2? If we short, this position should (in the best case) also generate a positive return. In addition, I still have the question of whether, if the "signal" does not change, we hold the positions until the signal reverses. Is the gross_return then calculated as the return over the period in which the signal was the same? thanks in advance

[@huntergiles]

Thanks for the video, subscribing to this channel is a no-brainer! One question, when calculating gross returns, would you not multiple stock1 by (1/beta)? My thought being, if the residual is the difference between stock2 actual and estimate values, then (1/beta)*residual is the distance from stock1 actual and estimate. For example if you take a $x long position on stock2, you would also take a (1/beta)* $x short on stock1. I'm new to this so I may be thinking about it wrong. Thanks again!

[@NEDLeducation]

Hi Hunter, and thanks so much for the comment! Glad you are enjoying the channel. As for your question, yes, the procedure you outlined also works to achieve market neutrality.

[@user-uc6wo1lc7t]

Видео с фин-образовательной точки зрения очень хорошее, но с точки зрения программиста... Я бы хотел намекнуть на паттерн "стратегия", помогло бы сократить if'ы, да и банально вынести определения функций за пределы цикла, незачем итак медленному питону постоянно переопределять функции. Да и в целом считается хорошим тоном всё-таки использовать Iloc при работе с датафреймами, хотя так как вы всё делаете используя индексы - легче, правильнее, БЫСТРЕЕ использовать нампай аррейки, банально перед основным циклом сделав raw_data = raw_data.to_numpy() и обращаться по тикеру через индексы (к примеру, tickers[0] заменить на 0, market на 2). Я понимаю, что, скорее всего фидбека от вас не получу, но если хотите - могу написать код о котором говорю. Опять же, я не хочу показаться "наезжающим" на вас, чисто с точки зрения кода попытался дать советы. В целом, если бы попал на ваш канал пораньше, когда я только разбирался в этом - вы бы мне безумно помогли, приятно что есть такие каналы!

[@nimbusdodger]

Thank you for posting this and your other detailed videos. Regarding the cointegration test used in this video, would it be more accurate to use 'Adj Close' rather than 'Close' prices? It seems that any dividends and splits not accounted for in 'Close' prices would affect the calculations. In a test run of your code using Adj Close, net returns are cut in half. Maybe Adj Close should be used for the cointegration test but Close should be used for simulated trades returns.

[@NEDLeducation]

Hi, and glad you are enjoying the channel! Adjusted closing prices are basically total return indices with dividends reinvested. While it does not make a big difference for short-term trading, over longer time periods the difference accumulates. I would then suggest simulating trading based on adjusted closes, and running cointegration tests on simple closes. Hope it helps!

[@User-qs4ok]

@@NEDLeducation Hi NEDL, thanks for the video. 1. You suggest "trading based on adjusted closes, and running cointegration tests on simple closes". Why not adjusted closes for both? 2. What about weekends? Do you think one should delete non-trading days or fill them with the previous days? I would be interesting in your opinion, you are clearly more clued up than me!

[@RCNNNNCR1357]

We have the times that we'll enter a trade however when will we close our position?

[@daniellopes559]

Hello! Congrats for the real good and interesting content created here. I ran your code and I found out a strange behaviour with the "fair_value" variable, sometimes it seems that is not well computed. For instance, if you take as start date of your pair "2018-12-31" and if your print the" fair_value" variable, as well the variables used for its computation, you will have weirds values as of 16/0/2020 and 18/03/2020. Are you able to understand why? Thanks. Daniel L.

[@NEDLeducation]

Hi Daniel, thanks for the comment and glad you enjoyed the video! Great question! During the turbulent market periods (18/03/2020 being perhaps the most turbulent day on recent memory) previously cointegrated pairs can become not cointegrated. This would lead to a rather small t-stat and, as the algorithm tries to calculate optimal b and a coefficients when there is little room for improvement, the fair value calculations can generate very high or very low values. This does not affect the trading signals generated by the strategy as such occurs only when t-stat is small, and the t-threshold guarantees the algorithm would not trade in such instances. Hope it helps!

[@bryan-9742]

Hey this is a great video. I think this is a great way via Co-integration to gauge the optimal trade location but what about trade frequency? I heard that mean first passage of Time is a good measure? Something with using an AR(1) process to optimize the trade duration and inter-trade interval? Reason I ask as I was playing around with this in digital assets and ranked my pairs via Kendalls Tau. From their I applied your technique however I realized through playing around with the window I would get very different results. I think I need to optimize the trade frequency component somehow. Your thoughts are greatly appreciated.

[@NEDLeducation]

Hi Bryan, and glad to hear you are using the video to inspire your own algorithmic trading! Overall, these ideas sound good, and it is true the results depend a lot on the rolling window length. The most straightforward approach would be to use backtesting and see which windows give the best results historically, or use KPSS test for stationarity as it favours shorter windows naturally.

[@silence5623]

Excelent Video. I was just curious as to whether this takes into account the spread ?

[@NEDLeducation]

Hi, and glad you enjoyed the video! As for your question, it is unfortunately impossible to retrieve bid/ask prices from the package I use, but these can be incorporated in the transaction cost calculations, just adding the typical bid/ask spread to the fee percentage.

[@josebelbute5812]

Awesome video, really helpful. Did you ever make the video about weighting various pairs? I tried searching it but can't seem to find it.

[@NEDLeducation]

Hi Jose, and glad you enjoyed the video! As for your question, you would most commonly invest equally in a several number (for example, top-10) most cointegrated pairs (lowest t-stats) that can be profitably traded (based on the deviation from equilibrium). However, the issue that different pairs can be held for varying time horizons is non-trivial and to best of my knowledge rarely discussed.

[@josebelbute5812]

@@NEDLeducation I wasn't expecting such a fast response, thanks a lot. Do you have any sort of discord server where ideas can be shared?

[@Gallaris]

Hello, thank you very much for this video it is very instructive. Could you lead me to some papers that explain this strategy? Or that you have used for this? I would like to understand better the details. Thank you! and keep it up with the great videos.

[@NEDLeducation]

Hi, and thank you so much for the feedback! As for your question, these has been considerable amount of research on practical application of cointegration to pair trading at least since the late 90s, the two papers I can point towards from the top of my head are Alexander (1999) and Krauss (2017). Hope it helps!

[@thaizaloiola7538]

just one question: when you establish the functional form for making the ADF test and say that you are not going to consider drift nor constant (no constant no drift unit root test kzbin.info/www/bejne/oKe9YamreJ6igM0 ), don't you think that you are imposing conditions over the data? I mean, don't you think that would be better to first look upon the data and see in which way does it behave? It could have drift for instance in the real world...

[@NEDLeducation]

Hi Thaiza, and thanks for the very good question! The drift and constant terms are generally useful in unit root tests, however for pair trading it would mean the dynamic equilibrium changes with time, so one stock is expected to continuously and predictably become cheaper or more expensive in terms of the other. As this is quite unrealistic, I opted to avoid coding the drift and constant terms in my unit root equation. Hope it helps!

[@oleksitkachenko4747]

Can you do video about johansen cointegration and VECM model to pair trading? I watched your videos they are great, but now everybody use johansen/VECM or KalmanFilter or ANN (LSTM) with some transformation of data.... Nobody use OLS....

[@Gustavo-bi4hv]

Does not make more sense use Beta Neutral operations instead of cash neutral when using co-integration? And a topic that a don't see anyone talking about is beta rotation. Since its a variable, the value can change and affect the beta neutral operations as the days went. Whats is your opinion about this?

[@NEDLeducation]

Hi Gustavo, and thanks for the excellent question! Conceptually, it does and can be a good addition to the strategy. One can even try and estimate the cointegration in abnormal returns of pairs net from their market risk exposures and trade based on those. Personally however, I do not feel it is too useful as 1) betas are a parameter subject to estimation error/noise, and this strategy already relies on several parameters being estimated with reasonable precision 2) if Dickey-Fuller test shows a high negative t-stat over the course of a year (as it does in this code), it does imply the discrepancies related to differing market risk exposures of stocks in the pair are low enough to disregard. I do acknowledge though that some industry experts disagree, and both approaches (cash neutral and beta neutral) are warranted. Here is where finance becomes more of an art than a science. However, I might do a video on beta adjustments for pair trading at some point in the future so thanks for the suggestion and hope it helps!

[@kemalduzkar4044]

Hi NEDL, First of all, thanks for your effort. When I try to apply the method you showed by using google colab, I get an error - KeyError: 'K' - which is caused from the line of; res1 = spop.minimize(unit_root, data[stock2][t] / data[stock1][t], method = 'Nelder-Mead') What may be the reason? Do you have any suggestion? Best regards

[@NEDLeducation]

Hi Kemal, and glad you liked the video! As for your question, cannot be 100% sure without seeing the code and full error report, but it seems that the error is associated with dictionaries. Would recommend running the code in Jupyter to see maybe it is just the Google Colab issue. Hope it helps!

[@kemalduzkar4044]

@@NEDLeducation Works well with Jupyter! Thanks for your advice. Looking forward to see your next project

[@KerouacsAccomplice]

Which stock pairs do you enjoy running through this program @NEDL? :-p

[@NEDLeducation]

Hi, and thanks for the question! Pair trading using cointegration is generally most applicable for stock pairs from the same sector, of similar size, and that are relatively liquid. Think JPM/BAC, PFE/JNJ, XOM/CVX, or, if you are feeling fancy, ROO.L/JET.L :) Hope it helps!

[@billlee9630]

According to your code, signal = np.sign(fair_value - data[stock2][t]) and gross_return = signal*returns[stock2][t] - signal*returns[stock1][t]. However, you are only able to get data[stock2][t] at the end of date t. How can you compute signal at the beginning of date t? If you cannot compute signal at the beginning of date t, how can gross_return be signal*returns[stock2][t] - signal*returns[stock1][t]?

[@NEDLeducation]

Hi Bill, and thanks for the question! This is due to returns on the day being calculated using future data. It does not make the code forward-looking as future data is not used in decision-making, but it simplifies indexing quite a bit. Hope this makes sense!

[@anttraders5883]

Unit root regression

[@maciejscibor]

8:08 - Why do you use simple returns here and not the log returns? Aren't log returns more appropriate for further compounding?

[@NEDLeducation]

Hi, and thanks for the question! Here, you can use either, I use simple returns but compound them later (for example when analysing cumulative performance) using product functions, while for log-returns I could use cumulative sums.

[@maciejscibor]

@@NEDLeducation Thank you for your answer 😁

[@yoofoo2620]

First of all, thanks for your video. And I have a question, (my English is not very well), we retrieve data from 2019-12-31 to 2021-03-08, and we use this centense "np.append(data[stock][1:].reset_index(drop=True)/data[stock][:-1].reset_index(drop=True) - 1, 0)" to calculate the daily return and then append a 0 to the last day. I think we should put the 0 as the return of the first day not the return of last day. I think the first return should be the return of 2020-01-02, because only when market close at 2020-01-02, can't we know the close price that day, and can't we calculate the return that day. So the first return should be the return of 2020-01-02, not the return of 2019-12-31. Since we don't know the close price at 2019-12-30, we should let the return on 2019-12-31 be 0.

[@NEDLeducation]

Hi, and thanks for the comment! The algorithm acknowledges the fact we cannot know what the price is by simply lagging decision-making appropriately, good point though!

[@yoofoo2620]

@@NEDLeducation Thanks for your reply!! I think I get it.

[@vladvol855]

Добрый день! Интересное видео! Посоветуйте пожалуйста, как решать проблему коинтеграции двух акций? Какой оптимальный период времени для оценки коинтеграции стоит принимать? Ведь на разных участках данный показатель может меняться. Спасибо!

[@NEDLeducation]

Добрый день, Владимир! Спасибо за комментарий и интерес к видео! Выбор оптимального периода оценки - это тот момент, где количественные финансы становятся скорее искусством, нежели наукой, и существует множество конкурирующих подходов. Разумеется, можно решить проблему с помощью бэктестинга - какая стратегия показывала наилучшие результаты в прошлом в зависимости от подвыборки. Лично мне нравится довольно красивый метод, где проверяется стационарность линейной комбинации с помощью одновременно тестов Дики-Фуллера (который часто показывает стационарность на больших выборках) и KPSS (который, наоборот, показывает стационарность на малых выборках). Так можно выбрать оптимальный объем данных, когда оба теста не противоречат друг другу. Но опять же, в этой сфере какой бы то ни было истины, высеченной на каменных скрижалях, не существует.

[@iglstivens]

Отличные у вас курсы, видел ваш совместный курс kzbin.info/www/bejne/gHjSl395o9yZjs0 на канале SF Education, у меня вопрос, как можно на Python написать формулу, которая бы в реальном времени высчитывала бы количество/объем покупаемых или продаваемых акций конкретной компании ежесекундно?

[@NEDLeducation]

Привет, и спасибо за отзыв! Вполне можно выгружать объемы на минутных свечах с yahoo finance в реальном времени, для секундных готового решения из головы сейчас не придумаю. Пример кода для выгрузки поминутных объемов Apple: yfinance.download('AAPL', period='1d', interval='1m')['Volume']

[@iglstivens]

@@NEDLeducation Спасибо, т.е будет видно как покупку так и продажу? и как я понимаю эти данные можно посмотреть за предыдущий период как собственно и цену и все остальное? единственное если делать анализ за предыдущий период то эта строчка должна быть немного другой? я новичок в этом поэтому столько вопросов)

[@chrischoir3594]

Why do people use Python? it totally sucks

[@henrikheffermehl5493]

Getting an error msg when coding this: File "", line 17 return = res.params[0] / res.bse[0] ^ SyntaxError: invalid syntax Where is my mistake? #initalising arrays gross_returns = np_array([]) net_returns = np_array([]) t_s = np.array([]) stock1 = stocks[0] stock2 = stocks[1] #moving thru the sample for t in range(window, len(data)): #defining the unit root function: stock2 = a + b*stock1 def unit_root(b): a = np.average(data[stock2][t-window:t] - b*data[stock1][t-window:t]) fair_value = a + b*data[stock1][t-window:t] diff = np.array(fair_value - data[stock2][t-window:t]) diff_diff = diff[1:] - diff[:-1] reg = sm.OLS(diff_diff, diff[:-1]) res = reg.fit() return = res.params[0]/res.bse[0] res1 = spop.minimize(unit_root, data[stock2][t]/data[stock1][t], method='Nelder-Mead') t_opt = res1.fun b_opt = float(res1.x) a_opt = np.average(data[stock2][t-window:t] - b_opt*data[stock1][t-window:t]) fair_value = a_opt + b_opt*data[stock1][t] #optimising the cointegration equation parameters if t == window: old_singal = 0 if t_opt > t_threshold signal = 0 gross_return = 0 else: signal = np.sign(fair_value - data[stock2][t]) gross_return = signal*returns[stock2][t] - signal*returns[stock1][t] fees = fee*abs(signal - old_signal) net_return = gross_return - fees gross_returns = np.append(gross_returns, gross_return) net_returns = np.append(net_returns, net_return) t_s = np.append(t_s, t_opt) #simulating trading print('day '+str(data.index[t])) print('') if signal == 0: print('no trading') elif signal == 1: print('long position on '+stock2+' and short position on '+stock1) else: print('long position on '+stock1+' and short position on '+stock2) print('gross daily return: '+str(round(gross_return*100,2))+'%') print('net daily return: '+str(round(net_return*100,2))+'%') print('cumulative net return so far: '+str(round(np.prod(1+net_returns*100-100,2))+'%') print('') old_signal = signal #interface: reporting daily positions and realised returns plt.plot(np.append(1,np.cumprod(1+gross_returns))) plt.plot(np.append(1,np.cumprod(1+net_returns))) #plotting equity curves

[@NorthwestKastaways]

Paste this: #initialising arrays gross_returns = np.array([]) net_returns = np.array([]) t_s = np.array([]) stock1 = stocks[0] stock2 = stocks[1] #moving through the sample for t in range(window, len(data)): #defining the unit root function: stock2 = a + b*stock1 def unit_root(b): a = np.average(data[stock2][t-window:t] - b*data[stock1][t-window:t]) fair_value = a + b*data[stock1][t-window:t] diff = np.array(fair_value - data[stock2][t-window:t]) diff_diff = diff[1:] - diff[:-1] reg = sm.OLS(diff_diff, diff[:-1]) res = reg.fit() return res.params[0]/res.bse[0] #optimising the cointegration equation parameters res1 = spop.minimize(unit_root, data[stock2][t]/data[stock1][t], method='Nelder-Mead') t_opt = res1.fun b_opt = float(res1.x) a_opt = np.average(data[stock2][t-window:t] - b_opt*data[stock1][t-window:t]) #simulating trading fair_value = a_opt + b_opt*data[stock1][t] if t == window: old_signal = 0 if t_opt > t_threshold: signal = 0 gross_return = 0 else: signal = np.sign(fair_value - data[stock2][t]) gross_return = signal*returns[stock2][t] - signal*returns[stock1][t] fees = fee*abs(signal - old_signal) net_return = gross_return - fees gross_returns = np.append(gross_returns, gross_return) net_returns = np.append(net_returns, net_return) t_s = np.append(t_s, t_opt) #interface: reporting daily positions and realised returns print('day '+str(data.index[t])) print('') if signal == 0: print('no trading') elif signal == 1: print('long position on '+stock2+' and short position on '+stock1) else: print('long position on '+stock1+' and short position on '+stock2) print('gross daily return: '+str(round(gross_return*100,2))+'%') print('net daily return: '+str(round(net_return*100,2))+'%') print('cumulative net return so far: '+str(round(np.prod(1+net_returns)*100-100,2))+'%') print('') old_signal = signal #plotting equity curves plt.plot(np.append(1,np.cumprod(1+gross_returns))) plt.plot(np.append(1,np.cumprod(1+net_returns)))