Async Price Data Load with aiohttp and asyncpg

[@parttimelarry]

I'm starting a new channel on AI at youtube.com/@parttimeai, please subscribe! Source Code: github.com/hackingthemarkets/timescaledb-aiohttp-asyncpg

[@yourpersonaldatadealer2239]

Async is hard to teach but you did a great job. For the newbies, one thing to remember that had me scratching my head a few years ago is: don't use synchronous code in an async function (e.g. don't use dataframe.to_csv(), instead use an async equivalent lib like aiofiles). If you don't do this then you end up going back to synchronous code running speeds.

[@therishabhpatel]

Thanks for the tip. I was dealing with a similar issue few months ago.

[@bouzie8000]

Very valid tip

[@ballyoracle]

the diff concept is very well explained ... got every bit for it .. you are already explaining it very well Thank you Larry !!!

[@TuranInsight]

The pieces of information you teach here are so valuable- thank you for doing this larry!

[@raghavendranayak7378]

such cool and interesting videos

[@michaelstreck0925]

I can't believe this video slipped by me until today. I need to up my PTL stalking game! Thank you for all you do!

[@Rickynator13]

Hey Larry I want to start saying thank you for all the knowledge that you share man. I have been following this series and I have been learning a lot Have a great and safe rest of your weekend!!

[@parttimelarry]

Thanks, glad you have enjoyed!

[@CameronCrosby2468]

Hey Larry, great video! I see your subscriber count going up :) You deserve 100 times as many as you have now!

[@stephenambrose7224]

This is great Larry, thank you for sharing and pickin on the guitar (if that was you). I did something similar with multiprocessing and mysql.connector.pooling which took a dataframe broke it into chunks based on the number of cpu cores and processed the df in parallel. This will be very helpful to get a snapshot loaded to a table for quick analysis.

[@iamvazu2768]

Thanks for all the KT larry, this is gold 🥇

[@binup28]

Hey Larry - Great stuff!! Thank you. Could you do one for alphavantage? Thanks in advance!

[@bouzie8000]

Great lesson

[@degenviking6068]

dude, is your site down? Was going to send you a should out for a whiskey for this video. Thanks for the awesome stuff man

[@JalilAsghar]

Just awesome!

[@brianrowe1152]

This is great, except Alpaca discontinued polygon. I'm trying in postman to get the REST API to return something, but with their current documentation it is just not working and the message is just Not Found. Maybe the URL is wrong, but they have no examples on their site of how to get bars from the REST API its all the API. Maybe I just write an example, that use concurrent futures, but I'm not sure I'll know how to the async pg working with that. Also maybe I need to see the next video, because I really wanted to see how to keep the prices up to date, not do a 1 time load. Thanks again, these are great.

[@AlexCarter881]

love this video - thanks :)

[@AnikPatelIsABaws]

Hey Larry, could you make a video on how to do this with the new Alpaca API and not a HTTP request?

[@richinvest9067]

Great, thanks Larry

[@kylehammerberg3875]

Sadly, as far as I know, this is no longer possible without paying for an API. Still a great video, though. Thanks, Larry.

[@mezseyoum5148]

Hey I noticed that this is not added in the repository yet. I would love to see the code. I'm not using RESTful API I'm using the TWS API of interactive broker. so I'm making all the necessary adjustments. So far I've been following this series and everything has been working great for me but I'm having a hard time wrapping my head around the concept of "co-routine". I would love to have the code in front of me so I could get a better intuition. Thanks for the content.

[@parttimelarry]

Just posted along with the SQL for the video after this one: github.com/hackingthemarkets/timescaledb-aiohttp-asyncpg

[@mezseyoum5148]

@@parttimelarry Perfect!! Thank you so much!

[@DavidCVdev]

hi, the poligon api key requires payment any alternatives?

[@shrimal]

Great video! Does "connection.copy_records_to_table" still work? I keep getting "" error on call to write_to_db.

[@shrimal]

still works. looks like the TypeError was coming from incorrect datetime format for "dt". Those are tough with Z, no-Z and so many formats.

[@thabangtsajoa716]

this is very infomative

[@ballyoracle]

this is awesome thank you bro

[@grizthegriffdog9939]

just amazing work

[@eeropar3543]

@Part Time Larry I'm new to algorithmic trading. I figured commenting here would be the best way to contact you. Since alpaca broker isn't available in Europe, Finland I don't know which broker I can use to set up a trading bot. Interactive Brokers is too complicated for EU use and setting up API. Can you help me, thanks in advance!

[@parttimelarry]

I think IB is still probably your best option. I think ig.com may be an option as well, but I haven't explored it to have an informed opinion.

[@skeletonmasterkiller]

You can use httpx instead of aiohttp, it has requests like structure but async

[@parttimelarry]

Thanks, I saw httpx mentioned in the FastAPI docs, been meaning to try it.

[@HitAndMissLab]

yeah, but data provider is going to block your connection if you send too many requests. How do you work around that? can you use proxies?

[@muhammadumair8126]

video is amazing but my only issue is API rate throttling. How can we implement rate throttling with aiohttp, asyncio?

[@samgeorgesullivan]

Hey, thanks for another great video! I have been trying to follow along using binance api instead and can't work out what the equivalent binance url would look like? Any help greatly appreciated

[@colinmaharaj]

Looking to make my own gap scanner with IB

[@ubaidulkhan]

Does async leverage threads to achieve the parallelism?

[@adityawadnerkar8682]

Google has recently launched its AutoML Tables product for using it in time series forecasting.. can it be used to predict the stock prices? Can you share it in next video if possible... thanks

[@parttimelarry]

Hadn't heard of this, will put it on my list. Definitely won't be the next video though, there are probably 100 other ones I need to make first :)

[@milprodin81]

awesome vid, Larry! thx! will this method work equally well with alpaca_trade_api instead of straight up session.get(url)

[@brijeshkhunt1857]

but, i think polygon.io has api call rate limit.....

[@parttimelarry]

Do you have a funded Alpaca brokerage account? You can substitute the url of your choice if you prefer another data provider. Code and concept will be 99 percent the same for other providers that provide a REST API

[@eeropar3543]

@@parttimelarry I'm new to algorithmic trading. I figured commenting here would be the best way to contact you. Since alpaca broker isn't available in Europe, Finland I don't know which broker I can use to set up a trading bot. Interactive Brokers is too complicated for EU use and setting up API. Can you help me, thanks in advance!

[@benheynderickx1729]

@@eeropar3543 hi yes it's possible to scribe and obtain access to the alpaca account , I just did this excercise this weekend and works fine (I'm from Belgium so it should also work from Finland) ..

[@saranganrajendren2485]

Hi Larry. Awesome work on this and your previous videos .Big Fan.Can you try to code swing low and swing high .Thank you in advance for your help.

[@Krisler12]

Hi! Please post a video on how to detect and plot to chart patterns like butterfly, double bottom, head and shoulders etc. in python. Be aware that this harmonic patterns are not randomly they have pivot points in range of Fibonacci. Thank you so much in advance!

[@theodorelee4383]

How much memory does this take?

[@FinGeek4now]

Awesome video! Gives a few ideas for my ML algo-trader =) Any plans for a multithreading video with perhaps also using the async as well?

[@laminegaye9073]

Hi Larry! Thanks a lot AS USUAL!! Great stuff! Would it be possible to rewrite/demonstrate populate_prices for IB (ib_insync). I tried with ib.reqHistoricalDataAsync inside of the get_price function but I am not getting that much of a boost, at least with the fetching bit (the db writing bit seems just fine. Am probably missing something... Again thanks a million!!

[@parttimelarry]

Yes but it could be a while. I have a lot of videos I still have in the works. I definitely want to do more with IB on here as I feel I have talked about Alpaca a bit too much at this point.

[@alexandremondaini]

Hey Larry could you wrap it up your video series about a topic on a series of sequential videos on youtube ? I jumped from populate_stocks to populate_prices but I can't find a video of you populating the etf table. cheers, mate

[@JellyBean012]

It's a shame this video is no longer useful due to the fact that Alpaca discontinued polygon. Would be great if you could release another approach as this series is great, but sucks I realized this after going through the 7 previous videos

[@parttimelarry]

The video is still useful. The most important concept is that it shows how to use Async IO, aiohttp, and asyncpg. The API request is the most trivial part since it's just a different URL being requested.

[@JellyBean012]

​@@parttimelarry Yea - not your fault both platforms now charge a fee for the new APIs. I guess its possible with pyfinance but would take a while to download for all the symbols

[@retrofutur1st]

I don't know what to do with half a million records :O