HUGE SCALE Backtest of Intraday Cryptocurrency Data using Python & SQL

[@christopherchen416]

Loving the humour 😂. "Moment of silence" "did I waste your time? Maybe.."

[@jodhvirsingh8677]

Let me be very clear....you never waste our time man...u teach us invaluable things👏👏

[@Algovibes]

Thank you so much Jodhvir!

[@hoapham7389]

it's only matter of time until you get more subscribers, lots of small things gets repeated over the past videos and it gets more familiar. I love it!

[@Algovibes]

Thanks for your motivation and your kind words man :-)

[@andreasschulze3563]

I‘ve been expecting this! Das hab ich mir neulich genauso gedacht! Richtig guter Ansatz! Thx

[@Algovibes]

Thx :-)

[@inspiration2292]

Recently found your channel, love your down to earth, clear style! big help

[@Algovibes]

Thank you very much my friend. Very happy to read that!

[@jancan9168]

Sehr interessantes Video. Eine coole Idee wäre nun ein Loop über diese Funktion mit unterschiedlichen TimeFames um zu schauen, wo der beste Profit erziel wird und wo der break even liegt. Mach weiter so

[@Algovibes]

Danke dir! Ja, genau das ist der Punkt. Einfach, dass man ein paar Daten hat und mit denen dann n bisschen rumspielen kann. Was du meinst, wäre z.B. eine sehr interessante Frage.

[@rraul]

This video is unbelivable. Awesome. As always, pure gold content. Thanks

[@Algovibes]

Thank you very much rraul!!

[@goodyonsen77]

Sounds like a huge meteorite touchdowned in Frankfart @23:27, sorry Frankfurt. Weird.

[@user-iy2qo6no2d]

Superb, the fee is my biggest problem so far. When we talk about the big crypto winners you will find a few lucky miners, early investors but most likely exchange founders and originators of cryptos. (Talking about BIG BIG Money)

[@HermanWillems]

big crypto winners trough algo's love to hide in the shadows :)

[@Algovibes]

Good point. Thanks for sharing your viewpoint!

[@marmalade12319]

Super helpful, much appreciated

[@Algovibes]

Happy to read, thanks a lot for watching :-)

[@walduss7939]

Interesting video! Akso, being a newbie in pyrhon, I am impressed how you manage the dataframe. If I try to do the same , my code will be 100x larger!

[@Algovibes]

Thanks mate. You will get there with the time. There is alot of improvement potential for me as well. Don't worry about it.

[@willbrenton8482]

Does simply subtracting costs account for the fact trading fees reduce buying power over the backtest period? I feel like this is still overstating? Perhaps this is a log property? Would appriciate any insights or resources. Thanks!

[@andrewyoung3135]

the fact you wrote out coins = (ect) ..... sub on the spot !

[@Algovibes]

Thanks for your support man. Glad it is helpful!

[@cryptolicious3738]

great video ! happy holidays. why did u leave out the trailing stop loss ? thats one of the best features in a strategy imho

[@Algovibes]

Thanks mate. Will covering TSL in one of my future videos surely.

[@kia4now]

Hello AlgoVibes, Im a big fan and have watched most your videos. Do you share the code anywhere for this, Im stuck at the part of building the DB and Im getting a lot of errors, do you happen to have a community chat anywhere. That would be super helpful, thanks for all your hard work

[@Algovibes]

Hi man, thanks a lot for your support and your kind words. I am usually sharing the code based on the engagement on a video.

[@trader8322]

Thanks for the valuable content. can we run this code 24/7 ? if so how we can do it ? do we need to host it in any server or git hub ?

[@Algovibes]

Welcome man. Can you elaborate on why you would run this 24/7? You mean like regularly updating the Database?

[@MrHamish1998]

Love your videos, you've taken me from a noob at coding to fairly good in a year so I can't thank you enough! Just an idea for you: maybe do a back-test but include leveraged tokens/futures could be interesting? I ran through this back-test and made a few changes, adding the ability to short trading leveraged assets with x5 leverage. The returns were enough to absorb the commission but I'm sceptical it's too good to be true.

[@Algovibes]

Hi mate, thanks for your kind words. Love to read that! :-)

[@TheLantani]

Im getting closer and closer!!! Tomorrow is test time

[@Algovibes]

Awesome, let me know how it worked out :-)

[@TheLantani]

@@Algovibes got a rough week but i will give it a try when the planeta alignes

[@leventbozkurt9796]

Thank you. Well done.

[@Algovibes]

My pleasure! Thanks a lot for watching :-)

[@gregd1218]

What I get from your videos usually is how data manipulation should be done properly. I tend to solve this problem the wrong way because I don't know how to use panda properly. I'm learning a lot thx. One question though, why didn't you do the download with multithreads? Wouldn't it had drastically cut the download and insert time?

[@Algovibes]

If you are getting anything out of the video that's nice! Thanks man. Didn't download with multithreads because of this: www.binance.com/en/support/faq/360004492232 Maybe one is not touching the limits but feel free to test it out yourself!

[@gregd1218]

@@Algovibes yeah that's what I was thinking, API limit. I think I'll give it a try, for the sake of it. Looking forward your next video!!

[@thomasniedermayer5956]

Nice Video! Shouldnt the cost be cost = (1-0.00075)^trades? Because if you e.g have 1000 dollars and take away 5 percent 2 times you have 1000*(1-5/100)*(1-5/100) = 1000* (1-5/100)^2

[@Algovibes]

Hi and thanks a lot for your comment. It depends if your investment amount is fixed. If it is not you are right but if it is you are paying steady fees. In your example the amount would shrink to 950 and the next 5% are calculated on the 950.

[@anders21karlsson]

Great video, thanx

[@Algovibes]

Thanks for watching Frankenstein :>

[@kekotalaverano]

I get that you use sqlite to simplify all, but if you use the stream data I guess you can have problems with concurrence. Awesome video btw. Grettings from Spain

[@Algovibes]

Thank you man! Can you elaborate on concurrence?

[@kekotalaverano]

@@Algovibes Yeah as far as i know sqlite its a non threat-safe database, i mean u can only write or read on one threat at a time. So if you are using a datastream to write, the threat writing its blocking the database, so cannot be readed. It´s not concurrence it is concurrency. Sorry for my bad english. xD

[@anshmiester78900]

Thank you so much for making these videos

[@Algovibes]

Thank you for checking them out!

[@financialblocks]

Hi, stellt Binance mehr als 500 Werte zur Verfügung? Dachte 500 wären beim Spot Max.

[@Algovibes]

Hi, kurz gesagt: Nein. Binance arbeitet mit gerichteten Requests. Mehr Hintergrund zu limits hier: www.binance.com/en/support/faq/360004492232

[@random_birch_forest]

This video is "GODLIKE" (in computer games terms). Well done and thanks a lot!

[@Algovibes]

Haha :D Played a lot of Games in my youth - and still do occasionally. Thanks a bunch for your comment.

[@3rdpartylegends]

Hi Algovibes could you make a video on how to fetch the data from tradingview along with indicators

[@Algovibes]

Not on my list but I just checked, there is a trading view python API, so would be quite interesting. Noted it, but as said, can't promise if I will tackle that.

[@3rdpartylegends]

@@Algovibes ok thnx for info

[@johnwarblock9898]

Hey buddy, i really respect you for your content and your type video with code. Can u share with me, which strategy in trading the best for 30-60 minutes trade with Machine learning, in your opinion?(sorry for my language, English isn’t my nativ language )

[@Algovibes]

Thanks for your kind words man. Well you have everything to test out some stuff 😛

[@random_birch_forest]

Why does Binance return the time for the start of the candlestick and not the end of the candlestick? E.g. Closing price of '1m' candle of 18:59 timestamp is equal to the closing price of '15m' candle of 18:45 timestamp. (Hope this explanation does make sense). Is this done to be able to report unclosed candles, e.g. when the request happens in the middle of a '15m' candle, we still get some data from the last candle which has not closed yet?

[@Algovibes]

Can you pass me a timestamp in the video where you see that?

[@random_birch_forest]

@@Algovibes at 5:29 if you pass '15m' into getminutedata you will see that the time of the candle is the starting time and not the ending time. This can be found out by comparing the closing prices of 1m and 15m.

[@diaboloshots]

Hi Algovibes! Firstly, thanks for the amazing content you're putting out. Its great to see a code focus and I absolutely love how you take the time to explain all the detail either here or in a previous video ... much Kudos! My question is, how would you approach developing a backtesting algorithm if the sell condition depended upon the return of that particular trade. So for example lets say I wanted to buy at midnight every day, and then sell when the return of that trade was 1%. In a live trade this isn't too hard to keep track of, but how would you back test it when the sell relies on the return calculation which is then 'reset' after each trade. Also you could end up holding for 3 days or so which means the buys in the middle would not occur. Hopefully an interesting one for you! 😊

[@diaboloshots]

To slightly add to this. I think the problem lies with, how to determine that we are in a trade if the buy condition is only 'true' for one or two rows. How do we 'latch' this condition with vectorisation? 🙂

[@Algovibes]

Thanks a lot Alex! Appreciate your comment. An interesting question indeed. The more granular your data is, the better you can do this kind of calculation. What you could e.g. do is to take a livestream containing live prices and store them in a DB (I actually did this in a lot of my cryptobot videos - which can be found in the cryptobot playlist). With that you can apply the selling logic better than just take the minute granular data distorting the "real" returns.

[@progtuts6071]

we are iterating coin dataframes and assume we are taking position for every signal, how can we iterate all dataframes at once and go minute by minute?

[@Algovibes]

Can you elaborate on your question? Been quite some time since I published this video. In the best case provide me some timestamps where you have questions on.

[@progtuts6071]

@@Algovibes ​here we are going coin for coin, could we somehow go for example 1h for 1h for all coins ? Because in live trading you are looking at the whole market at once . Just asking if you have any ideas?

[@Golapeofficiel]

Hi, I have done some backtest with yfinance library, but when I tried it in real time with binancesocketmanager, it didn't work at all, I suppose that dellay causes too much price divergence (was doing trading on 1m scale) :(

[@Golapeofficiel]

Actually you speak about that at the end

[@Algovibes]

Yep, exactly.

[@SaverioZanco]

Always good job, seriously:-) I'm study your video in this day for more concept of stock data, SQL maybe the best and is the "next level", it semmes Timescale DB offrer much power! Anyway it's my opinion just i want shared with you:-) Good jobs and very appreciate for you work:-)

[@Algovibes]

Thanks a lot for your kind words man. And also thanks for the suggestion! Didn't work with Timescale DB yet but have heard good things.

[@meetshah4800]

Is there any Datafeeds or Library to obtain data of Stock Indices for 1min time interval(Over 5 years)??

[@meetshah4800]

free

[@Algovibes]

Would be nice :D But I wouldn't know of something like that. Only free data is cryptocurrency price data on that granular timeframe.

[@raydan18]

Do you take requests to make robots for money?

[@Algovibes]

I currently don't. Maybe in the future!

[@BOSprodz]

Could you please explain how to use our 1 minute dataframe to create 15m, 1hr etc dataframes?

[@Algovibes]

Hi mate, correct me if I am wrong but didn't I do that in exactly this video? :D

[@BOSprodz]

@@Algovibes True, I didn't watch to that part! But doesn't the resample simply skip over the data in between? I was talking about creating a new dataframe with all correct fields for other timeframes (I did manage to do this myself eventually, using the .agg() method to apply sum/min/max etc to the columns). Btw, I think a video where you do a complex strategy (even if it isn't profitable) would be really educational! I'm trying to play around with it myself but I'm not sure how to make the program "consider" much more factors at the same time (and it also has to "forget" them after they are irrelevant) Thanks again for everything you do!

[@teenspirit1]

What commission and slippage did you apply?

[@Algovibes]

Quite confused with that question as it is exactly discussed in the video...Can you elaborate? Thx in advance!

[@Septumsempra8818]

How do we build a schema for balance sheet data AND price data? s/o from South Africa

[@Algovibes]

Either use MySQL as e.g. in my Stock recommendation playlist or as subtables in a SQLite database.

[@somajuice666]

What if i wanted to use Bollinger Band strategy?? Then i cant just hold?

[@Algovibes]

Exactly. Then you can't use this approach but need to implement a logic that the upper or lower band was hit. I have actually covered that here: kzbin.info/www/bejne/boHdgoadrJWGpq8

[@sazamhashmi]

How can i get in contact with you?

[@Algovibes]

Here! How can I help you?

[@vijax21]

All your tutorials are LONG --> Buy & Sell. could you please help do a tutorial for short selling backtest? it would be intersting to see what profits can be made on certain strategy. thank you

[@Algovibes]

Maybe I got your wrong but I am actually covering a lot of videos on short strategies (e.g. Momentum, Reversal). Even in the video I am stating how you would design the function when you want to include shorting.

[@vijax21]

​@@Algovibes I am sorry! I certainly didn't get to watch all of your tutorials. but thank you for pointing me to topics. I shall check it out now. Thank you again! Cheers!

[@chapmansbg]

I was of the opinion that binance only gives 7 days for 1 minute data? Am I missing something here?

[@tayaamme9954]

With binance you can download 1 minute data from the beginning of the listing (on binance) of a coin.

[@chapmansbg]

@@tayaamme9954 I have hit limits before

[@Algovibes]

What you are referring to is yfinance. There you can only go back 7 days for intraday.

[@chapmansbg]

@@Algovibes oh, I rarely use yfinance but probably transferred that thinking from there without testing.

[@themechanicaltechnologist6121]

Hello! I'm a beginner using the binance API and I'm getting stuck just trying to make a program access my binance account using the API keys. I wrote a test script of a trading strategy that takes live data and then prints buy, sell, wait to buy, or hold. If it decides to sell, it also prints the buy price. I got the program logic to function how I want it to, and now I want to actually connect it to my account and let it start to buy and sell crypto. However, for some reason, I keep getting an error that says either my API keys are wrong, my IP is invalid, or I don't have the required permissions. I've tried multiple API keys, my IP is unrestricted, and I gave the API permissions to do everything it should be able to do. I also get this message same message when I wrote a simple script just to view my basic account info. I'm not sure what I'm doing wrong, any help would be appreciated!

[@Algovibes]

Hi mate, first guess would be: Do you fulfill the requirements of legitimation? Binance is requiring a higher level of authentication for API actions. I needed to go through the KYC process.

[@chapmansbg]

Shouldn't you only calculate the percentage return on the difference between the buy price and the sell price?

[@Algovibes]

Yes, but the Close price is the best guess. How can we exactly know the buyprice? I have talked about that problem in the very end of the video.

[@chapmansbg]

@@Algovibes your method is pretty smart. I have also wondered how to separate the close and buy prices in an elegant way. It is always so cumbersome and brute force.

[@khawajadanyialahmad45]

Hi sir I want to make fiver gig for trading bots can you tell me which bots i can make for my client on freelancing platform

[@Algovibes]

Hi man, I don't really get the question. Can you elaborate?

[@wethebelievers260]

Sir I want a binance bot can you plz help me with it??

[@wethebelievers260]

@Algovibes I am your subscriber sir from past 3 months. Can I get your mail ID so I can send you details?

[@Algovibes]

Thanks a lot for your support man. How can I help?

[@wethebelievers260]

@@Algovibes sir I want you to create a bot for me. As I don't know anything about Python or coding. I want your mail ID so I can send you detailed logic of the bot.

[@wethebelievers260]

You are ready to pay for the project sir. I have to discuss logics of bot with you

[@meetshah4800]

Yahoo finance is unable to fetch data of 1min more than 7 days I want solution

[@Algovibes]

Solution is pretty much straightforward: There is not a good datasource for stock data fulfilling your requirements.

[@itayhilel2168]

Great video, now let's lose some money!!

[@Algovibes]

Haha :D

[@henrifritsmaarseveen6260]

you know what is big tick data from 30 FX pairs from the last 20 years ..

[@Algovibes]

Haha :D I hope you got interesting insights? Anyhow you can scale the crypto data here. You can theoretically go back years.

[@JoseAltagracia]

…………………………………………..…………Maybe 😂

[@javierloa9197]

Who said Germans ( assumption) don't have a sense of humor. 🤣

[@Algovibes]

Your assumption might be right :P

[@meetshah4800]

Please Help🥺

[@Algovibes]

Sure, but how? 🥲

[@meetshah4800]

@@Algovibes I had one data downloaded I just want that into datetime format

[@meetshah4800]

@@Algovibes Please check comments on crypto video of 1min