Getting all company pricing data in the S&P 500 - Python Programming for Finance p.6

[@alpharomeo401]

Sentdex is one of a kind, really appreciate his lessons. For the new people like me, I had to make a few changes in order to make script functional: On terminal: pip install fix-yahoo-finance Add the following to your import modules list: import fix_yahoo_finance as yf "from pandas_datareader import data as pdr" instead of "import pandas_datareader.data as web" yf.pdr_override() Use: "df = pdr.get_data_yahoo(ticker, start, end)" instead of "df = web.DataReader(ticker, 'yahoo', start, end)" Add: ticker = "str(ticker).replace('.','-')" right before the "tickers.append(ticker)"

[@matthackin]

Thank you!

[@John-dl5pw]

just so people know now it is pip install yfinance --upgrade --no-cache-dir import yfinance as yf github.com/ranaroussi/yfinance/blob/master/README.rst

[@freetube7767]

Thanks a zillion !!!! *** `fix_yahoo_finance` was renamed to `yfinance`. *** Please install and use `yfinance` directly using `pip install yfinance -U` More information: github.com/ranaroussi/yfinance

[@erikpantzar8127]

So thankful❤️

[@lividpudding8565]

Thanks a lot!!

[@porlando12]

Watching the stock_dfs directory fill up was the most satisfying moment of all time. Great video!

[@melancholybrocoli9868]

I wish I found this sooner. I have watched 5 already. Thanks so much for uploading this stuff man.

[@waynewatson7970]

Sentdex, if that is what I should call you. Thank you so much for sharing. That is such a cool thing that people share what they know. All the time you have spent of your own learning all these things and your willing to share. Thanks again.

[@savirien4266]

Working on trading software myself and slowly going through your new series when I have time. It's nice to see someone actually cover the fundamentals of financial computing. It's also nice to validate some things are on the right track, at least for my own program. I had one suggestion though, why use pickle? Why not place your data into some flavor of an SQL database? Then you can design your functions to simply update that information as time goes on. This way would benefit us later on as our program becomes multi-threaded. Anywho, thank you for these videos!

[@kuatroka]

Great knowledge sharing! thanks. Throwing two ideas for topics: 1. How to update the data every day with only missing bars (daily in this case) and not trigger full load every day. 2. How to use TA-lib library for example MACDEXT and use it as filter for the stock screener.

[@alexrodgers2922]

Hey I think I can help with that, I wrote a method that updates the csv files by appending all the information between the current day and the last day the csv files have on them. I put the file on github if you want to take a look. github.com/AlexanderRodgers/stocks/blob/master/finance_for_python.py

[@kuatroka]

Thanks Alex, very neat code.

[@mr.youtube7195]

Sentdex! Can you do a tutorial on using google api with python? Uploading docs, sheets, etc. You have the best Python tutorials by far

[@liangyumin9405]

tensorflow?

[@zcrib3]

Wikipedia uses "." instead of "-" in their list. Had to translate "." to "-" so it would get past Berkshire Hathaway. Just for anyone running into this.

[@superdonkhalil]

how do you do that? i get the same error

[@zcrib3]

Added this into the initial save_sp500_tickers for loop before it appends the tickers: mapping = str.maketrans(".","-") ticker = ticker.translate(mapping) You will need to delete list file and rerun save_sp500_tickers function.

[@DiptiranjanHarichandan]

It's still not able to pass Berkshire Hathaway - pandas_datareader._utils.RemoteDataError: Unable to read URL: ichart.finance.yahoo.com/table.csv?s=BRK.B&d=1&f=2017&g=d&ignore=.csv&c=2000&e=8&b=1&a=0

[@zcrib3]

You need to rerun the save_sp500_tickers function. As you can see from the link it still is looking for BRK.B not BRK-B

[@DiptiranjanHarichandan]

Thanks Mate!!! it worked :)

[@asivolobov]

Yahoo changed API so there are some runtime errors in current version of code. To repair: 1. All information is here: pypi.python.org/pypi/fix-yahoo-finance 2. Install fix_yahoo_finance using pip: $ pip install fix_yahoo_finance --upgrade --no-cache-dir 3. Import fix_yahoo_finance into your code (add this at top of your file after "import pandas_datareader.data as web"): import fix_yahoo_finance 4. Change a line with 'yahoo' string to: df = web.get_data_yahoo(ticker, start, end)

[@bigeteum]

thanks man, you saved the day

[@kmillanr]

As soon as I installed the fix_yahoo module I started getting this error: from pandas.core.common import PandasError ImportError: cannot import name 'PandasError' any idea as to what might be causing it?

[@Anthonypython]

Kevin Millan check and make sure you have python version >= 3.4, pandas version >=0.18.1, and pandas_datareader version >=0.4.0 if this is the case just uninstall both pandas/pandas_datareader and try again, some how it isn't able to import PandasError which means A it doesn't exist or B Denied permission?(pretty sure it would say it though if that is the case)

[@EvilSpeculator]

Works just fine, thanks a ton.

[@kmillanr]

I uninstalled everything and reinstalled it using conda

[@gehtomacgyver]

Full code, working as of 4-19-2020. Corrects for those who kept getting an error that "no data fetched for symbol MMM (or another stock ticker ) using yahoodailyreader" import bs4 as bs import datetime as dt import os from pandas_datareader import data as pdr import pickle import requests import fix_yahoo_finance as yf yf.pdr_override def save_sp500_tickers(): resp = requests.get('en.wikipedia.org/wiki/List_of_S%26P_500_companies') soup = bs.BeautifulSoup(resp.text, 'lxml') table = soup.find('table', {'class': 'wikitable sortable'}) tickers = [] for row in table.findAll('tr')[1:]: ticker = row.findAll('td')[0].text.replace('.', '-') ticker = ticker[:-1] tickers.append(ticker) with open("sp500tickers.pickle", "wb") as f: pickle.dump(tickers, f) return tickers # save_sp500_tickers() def get_data_from_yahoo(reload_sp500=False): if reload_sp500: tickers = save_sp500_tickers() else: with open("sp500tickers.pickle", "rb") as f: tickers = pickle.load(f) if not os.path.exists('stock_dfs'): os.makedirs('stock_dfs') start = dt.datetime(2019, 6, 8) end = dt.datetime.now() for ticker in tickers: print(ticker) if not os.path.exists('stock_dfs/{}.csv'.format(ticker)): df = pdr.get_data_yahoo(ticker, start, end) df.reset_index(inplace=True) df.set_index("Date", inplace=True) df.to_csv('stock_dfs/{}.csv'.format(ticker)) else: print('Already have {}'.format(ticker)) save_sp500_tickers() get_data_from_yahoo()

[@jaydnsmiley]

im getting an error --> ModuleNotFoundError: No module named 'fix_yahoo_finance'

[@andrepinho6730]

Thank you!

[@gehtomacgyver]

@@jaydnsmiley Hello, do you still need help? If so, I'll take a look at it shortly.

[@gehtomacgyver]

@@andrepinho6730 Anytime

[@gehtomacgyver]

@@jaydnsmiley Hello again, as I recall I had to download the module in the python terminal. I'm on my phone currently, so I can't give you an exact download command. Hopefully this gets you in the right direction for now, but if not I'll follow up sometime today.

[@rap2back]

Heads up for everyone watching recently, wiki may have changed the table formatting so you have to type the beginning as: for row in table.findAll('tr')[1:]: ticker = row.findAll('td')[1].text.replace('.','-') He's got a ('td')[0] but you need ('td')[1] for the file to save the stock abbreviations, ('td')[0] saves the full titles and it doesn't work for the next step.

[@ryanflanigan1153]

says name 'table' is not defined

[@boratarhan]

Harrison thank you for all your great contributions. Are you planning to extend this tutorial series to real time data? Getting tick data, updating last ohlc data, visualizing it, saving the data in a db? It would be great if you could either share your opinion about how to efficiently do this or put together a demo.

[@bread639]

I don't know if you're getting the same error as me but next to line: >> ticker = row.findAll('td')[0].text you'll need: >> ticker= ticker[:-1] to get rid of the ' ' character

[@uncommonsense6022]

why does the last character need to be excluded in our case but not sentdex?

[@THEARTISTPASCAL]

ticker = row.findAll('td')[0].text.replace(" ", '').replace('.','-')

[@lhma1981]

Thanks a lot

[@michaelschneck6161]

I added yfinance and the program worked-Halleluah

[@stephenhobson1618]

Many thanks for taking the time to make and share your knowledge, you inspired me to learn Python. Trouble is I've hit a wall, regarding accessing non US exchanges. I'm in the UK and so, want to access London Stock Exchange (FTSE All Share etc) stocks. Any hints or ideas would be appreciated. Thanks again and I can't wait to see what you'll do next

[@realking4918]

Great video series, love it!

[@nokkreload]

Great video and Channel too! I'm so glad to you for your explanation!

[@dustinwi]

My twist on the solutions for "Keyerror: 'Date'", extra white space in the ticker names, and "." instead of "-" in the Wikipedia page: Keyerror: 'Date' - Another user suggested changing the end date to "today" but hard coded the date in his example. Anyone watching it later could run into the same issue. I used these as my start and end so it will theoretically always work, and still gathers roughly the same amount of data used in this video. end = dt.date.today() start = end - dt.timedelta(days=int(16*365.25)) Extra white space and wrong separator in the ticker names - I did a one line update, instead of the suggested multi-line if statements mentioned below. Use strip to strip off leading and trailing whitespace. Use replace to replace the "." with a "-". I prefer it all in one line. Just my preference. Change this: ticker = row.findAll('td')[0].text To this: ticker = row.findAll('td')[0].text.strip().replace(".", "-")

[@RafzLima]

I've changed the Date to that and I'm still getting the "Keyerror: Date" specifically when I arrive at the BRK.B, any idea why? Edit: Ok i figured it out as I wrote this, it was the . instead of - problem

[@dustinwi]

@@RafzLima glad you got it!

[@ZakharovInvest]

a little update. Whoever said to add if ticker.find("_"): ticker = ticker.replace("_","-") print("trying to read modified {}".format(ticker)) he was right. But also try to pull the data from google instead of yahoo. it may work for you perfectly. This worked for my case

[@ZakharovInvest]

RemoteDataError: Unable to read URL: www.google.com/finance/historical?output=csv&startdate=Jan+01%2C+2000&enddate=Dec+30%2C+2016&q=LMT Cant figure it out. Why it cannot get LMT ticker. Tried both yahoo and google. Still stuck with it. If anyone knows what might be the reason I will be grateful for help

[@ZakharovInvest]

For some reason these 3 stocks could not download and caused an error. Just remove them from your list in save function and then you will be able to download others. Hope it will help someone who has the same problem that I had tickers.remove('LMT') tickers.remove('NWL') tickers.remove('NBL')

[@ryank7889]

I have been using Google and I also had this issue. I found that google prefers NYSE:LMT for LMT and these other stocks. Try this code: tickers = ["NYSE:LMT" if x == "LMT" else x for x in tickers] tickers = ["NYSE:NWL" if x == "NWL" else x for x in tickers] tickers = ["NYSE:NBL" if x == "NBL" else x for x in tickers]

[@thedscarves]

Where did you place this in your code?

[@ryank7889]

I put it right before the "with open" section in the saveSP500Tickers() method. It essentially cycles through the tickers list and replaces "LMT" with "NYSE:LMT" and so forth. That way when it adds the tickers to the .pickle, it is correct. Here is the entire method: def saveSP500Tickers(): resp = requests.get('en.wikipedia.org/wiki/List_of_S%26P_500_companies') soup = bs.BeautifulSoup(resp.text, 'lxml') table = soup.find('table', {'class':'wikitable sortable'}) tickers = [] for row in table.findAll('tr')[1:]: ticker = row.findAll('td')[0].text mapping = str.maketrans(".", "-") ticker = ticker.translate(mapping) tickers.append(ticker) tickers = ["NYSE:LMT" if x == "LMT" else x for x in tickers] tickers = ["NYSE:NWL" if x == "NWL" else x for x in tickers] tickers = ["NYSE:NBL" if x == "NBL" else x for x in tickers] with open("sp500tickers.pickle", "wb") as f: pickle.dump(tickers, f) print(tickers) return tickers

[@zanehutchison7304]

For everyone who is getting stuck on the stock CTVA, it's because the stock only opened 3 June 2019. Since there is no data the 'Date' issue occurs. To fix this I just adjusted the end date to be 30 June 2019: end = dt.datetime(2019,6,30)

[@amitdudgaonkar1831]

have you had any issues with using 'yahoo'?

[@amitdudgaonkar1831]

nvm i got it

[@vd_1990]

For anyone running into issues of how wikipedia has some of the tickers with a period and how that causes your function to fail, just need to add the following two lines; for ticker in tickers: ticker_unadjused=ticker if ticker_unadjused.find(".")>0: ticker_unadjused=ticker_unadjused.replace(".","-") if not os.path.exists('stock_dfs/{},csv'.format(ticker_unadjused)): df=pdr.DataReader(ticker_unadjused,'yahoo',start,end) df.to_csv('stock_dfs/{},csv'.format(ticker_unadjused)) else: print("already have the {}".format(ticker_unadjused))

[@qalbepoems]

For anyone who wants their CSV files to be updated the next time you run the program, my code is as follows: for ticker in tickers[:]: if not os.path.exists('stock_dfs/{}.csv'.format(ticker)): df = web.DataReader(ticker, 'yahoo', start, end) df.to_csv('stock_dfs/{}.csv'.format(ticker)) else: temp = pd.read_csv("stock_dfs/{}.csv".format(ticker)) date = [int(temp['Date'][len(temp)-1][:4]), int(temp['Date'][len(temp)-1][5:7]), int(temp['Date'][len(temp)-1][8:])] X = dt.datetime(date[0], date[1], date[2]) Update_start = dt.datetime(end.year, end.month, end.day + 1) if X >= Update_start: print('Already have {}'.format(ticker)) else: print('{} is updating'.format(ticker)) Update_end = dt.datetime(now.year, now.month, now.day) df = web.DataReader(ticker, 'yahoo', Update_start, Update_end) with open('stock_dfs/{}.csv'.format(ticker), 'a') as f: df.to_csv(f, header=False)

[@cesarbodden1162]

This is a very helpful series. Thank you for posting. I have a question, in this series we downloaded the historical data for all the S&P 500 stocks. Is there a way to pull the pricing data for a single day for each stock and then put it in a separate file? This way I can analyze a single day's activity. Thanks,

[@cedriklegrand]

For people doing this with newer version of panda_datareader, if the date is outside the actual life possible data date, it will trow a KeyError : Date. This is a bug that you can see on the github of pandas_datareader. A way to avoid it is to go with a try and except (or copy me :p): for ticker in tickers: print("Scraping: "+ticker) if not os.path.exists('stock_dfs/{}.csv'.format(ticker)): try: df = web.DataReader(ticker,'yahoo',start,end) df.to_csv('stock_dfs/{}.csv'.format(ticker)) except KeyError: pass else: print('Already have {}'.format(ticker))

[@philippeflorence4184]

Hey, thanks for the solution, do you mind explaining why this occurred? Also your solution skips the company, is there any way to not do that?

[@cedriklegrand]

You can check the GitHub of Pandas on this issue : github.com/pydata/pandas-datareader/issues/640 I have not tried it but you could create a list of all the ticker that does not pass the try, and later do a loop with a different start date for those tickers, then merge the 2 list. It is not pretty but could do the job. Preprocessing data is always a pain haha

[@bjornnilsson4648]

Hi, i get this error message: raise RemoteDataError('Unable to read URL: {0}'.format(url)) pandas_datareader._utils.RemoteDataError: Unable to read URL: www.google.com/finance/historical?q=HUM&startdate=Jan+01%2C+2000&enddate=Dec+31%2C+2016&output=csv I get the same typ of error message when I try to pull data from omxs30 list. What could be the problem?

[@bigfoothines]

Hey Sentdex, when pulling my data from Yahoo, an error occurred wherein one of the tickers (cant remember which one) wouldn't load from its URL on Yahoo I changed to google api in the web.DataReader(ticker, "yahoo", start, end), which then ran into difficulties with Endo International plc, so I then changed back to yahoo which got the job done.... any ideas why certain stocks wouldn't pull? got the data now but just curious Thanks P.S awesome tutorials... having started just a few weeks ago on python i've been incessantly watching your channel and I think maybe now I at least have some idea about Python!

[@burntpeppers]

I ran into this issue too. Some of the tickers from wiki had periods in the names, but yahoo uses hyphens, so i just did: web.DataReader(ticker.replace(".", "-"), "yahoo", ...), and this fixed the issue. This may not be the prettiest solution, but it works now.

[@yolzee]

Now they have "_" instead of ".", so I had to add the replacement of underscore print("trying to read {}".format(ticker)) if ticker.find("_"): ticker = ticker.replace("_","-") print("trying to read modified {}".format(ticker))

[@기바랜드]

Thanks for saving the day.

[@joshuabeauchamp4525]

for everyone having the "EOF" error, make sure you have created a 'stock_dfs' folder wherever you have your .py file.

[@durstiespace2270]

If anyone is hitting this in 2021, yahoo finance doesn't use "." in their tickers. Replace any "." with "-" instead. for m_ticker in m_tickers: print(m_ticker) # Yahoo finance uses "-" instead of "." m_ticker = m_ticker.replace('.', '-') if not os.path.exists(f'stock_dfs/{m_ticker}.csv'): try: df = web.DataReader(m_ticker, 'yahoo', m_start, m_end) except Exception as e: print(f'Could not get data for {m_ticker}') continue df.to_csv(f'stock_dfs/{m_ticker}.csv') else: print(f'Already have {m_ticker}')

[@pfblatx2]

THANK YOU! This is exactly what I came here for - wish I could give you more upvotes.

[@iceman13687]

05:50 any help on how to add a list for updating stock data everyday? (Sentdex a BIG thank you)

[@pmunin]

Sentdex, How come you still don't have an autonomous self-training neural network trading for you fulltime? :) Looks like you have all the tools for that.

[@sentdex]

Working on it, currently working on dealing with the whole SEC EDGAR thing, reading company reports for information. I have done lots of automated trading over the years, even beat the market, but automated trading so far has been basically a full time job. It's not enough to beat the market, you need to significantly beat the market, where your earnings compared to market, divided by the # of hours spent...needs to be something nice, and it's not. Something totally autonomous would be awesome, since it would scale unlike me physically...maybe one day.

[@pmunin]

Yeah I remember you talked about it in your previous videos. But I meant fully autonomous neural network, which does not require your time (except administrating it and tweaking parameters), - it learns itself how to trade from its previous "experience", something similar to Q Learning where NN learn how to play game using reward function. As far as I see this problem is equivalent to problem of pattern recognition where patterns are not fixed and evolve in time. With all the knowledge that you have - how realistic you think this task is - to build such NN system? Do you know any companies who have it done or tried to do it but failed? PS: What is awesome that you not only can feed network with pricing data (which most of traders do), but also with sentimental data which you have expertise at - and that will cover both technical AND fundamental analysis.

[@sentdex]

I addressed the autonomous network at the end of my last reply. Q Learning inherently makes use of a fixed set of outcomes, and specifically uses pixel data as input, and lends itself to be used with convnets, because convnets just simply make sense for any image data, which includes games, but this is not the case for finance data. Markets are not set, they do change in time, but there's no reason an online training network couldnt handle for this. The issue really just is simply that finance is not the same as a video game. Videogames are actually very simple, and have very few variables compared to finance and what goes into valuing a company. Like I said, I am working on it, but it's a very challenging problem.

[@kuatroka]

In his series of lectures on Reinforcement Learning, David Silver from DeepMind mentions that Reinforcement Learning is used in Trading. I only started viewing them, but I bet you will extract more valuable info from it. Hopefully it''ll be useful to you.

[@EvilSpeculator]

Sorry to burst your bubble but this is a pipe dream as even systematic trading is very time intensive and requires your full attention. You're basically shifting activities related to discretionary trading for issues related to trading automated ones. Even a completely 'autonomous' self optimizing SVM or ANN based system will require constant human monitoring and the occasional emergency action. The advantage of automation however is that you can run a larger number of systems against each other at the same time. Monitoring activity and occasional corrective action only takes marginally more and you can also hire people if you need some eyes on them overnight. I'm pretty certain that most of you guys already know this but just in case ;-) Me: I have been developing trading systems for eight years plus now professionally and know a good number of people in the industry who do nothing else.

[@lacunalife]

Awesome series - super coaching for us absolute noobs! I have a CSV for all of the Australian tickers and just want to replace the pickle to call the CSV list - can someone please point me in the right direction?

[@theaxel1231]

I really thought he was kidding when he said "if you've slow internet just grab 10 or something like that", then while running the script I realized he was right about spending too much time getting everything.

[@vickys2555]

implement asyncio with aiohttp

[@tahalv85]

Great video (and series). Quick question: is it possible to pull back other historical metrics, e.g. trailing-12 Earnings per Share, through pandas?

[@sentdex]

Not via the Yahoo api. You can through something like the Quandl API, but that fundamental is paid data from Quandl. I do not know of a single source that provides free fundamental data that you could pull locally into your own dataframe. If you find one, do let me know!!

[@liruixue403]

HI Sentdex, very powerful introduction! i do have a question though: when I first tried to download the stock prices, there are around 100 stocks that showed error message. i used try/except to handle the error and save the failed tickers to a list and rerun the same function for the failed tickers and this time it worked for another 88 tickers. i had to redo the failed tickers again and again but still fail to get the data for the following 4:['BRK.B', 'BF.B', 'DWDP', 'STI'].... any suggestions?

[@linaskulbaciauskas7068]

Wikipedia is not reliable source of information. There is not ticker such as BRK.B. only BRK-B finance.yahoo.com/quote/BRK-B/

[@chinzorigtds1796]

Round all columns into 2 decimal places for smaller memory space: df = df.round(2)

[@dr.mikeybee]

Also, note that the column names are different from Quandl with all lower case letters and underscores.

[@edgardolopez6908]

For anyone getting the HistoricPriceStore Error for some tickers, you can skip them with a try/except. (From StackOverflow) def get_data_from_yahoo(reload_sp500=False): ... if not os.path.exists('stock_dfs/{}.csv'.format(ticker)): try: df = web.DataReader(ticker,'yahoo',start,end) df.to_csv('stock_dfs/{}.csv'.format(ticker)) except Exception as ex: print('Error', ex) else: print('Already have {}'.format(ticker)) ...

[@jyoung5256]

thank you!

[@christosroniotis1903]

Hello, Congrats on your amazing work!I modified a little bit the algorithm (S&P100, one year data 2018-1-1 / 2018/12/31) however in the stock_dfs file I get only the first 34 csv's of stocks, in alphabetical order.Then I am getting a huge error that ends with the note 'KeyError: 'Date' .Any idea? Best regards!!!!

[@basvanrooij1020]

It is probably because it wants to read the data from a company with startdate 2018-01-01, which was not yet listed on the S&P at that time (for instance, Carrier was not yet listed by that time)! You can fix that by trying something like this: for ticker in tickers: try: if not os.path.exists('stocks_dfs/{}.csv'.format(ticker)): df = web.DataReader(ticker, 'yahoo', start, end) df.to_csv('stock_dfs/{}.csv'.format(ticker)) else: print('Already have {}'.format(ticker)) except: print('Cannot obtain data for ' +ticker)

[@peterkoppelman6232]

I was getting a lot of errors so I changed save_sp500_tickers. Right before the append statement I added the code ticker = ticker.replace('.','-').strip(). This seemed to fix the problems with wikipedia using a "." instead of a "-" and I no longer had issues with any tickers greater than 3 characters. Then I changed get_data_from_yahoo. I put df = web.DataReader(ticker,'yahoo', start, end) inside a try statement with the error trap (except clause) checking for RemoteDataError. The first time that I ran everything through get_data_from _yahoo 79 tickers went to the error trap. The second time I ran get_data_from_yahoo, the code there were 11 errors (the code caught 68 of the 79 tickers). The third time there were 2 errors (the code caught 9 or 11 tickers). The fourth time there was 1 error. The ticker symbol that still fails is FTV. I can't figure out why the code works some of the time and not all of the time. It bothers me that it's not consistent. I'm using the same list each time. Is this a problem with yahoo or the code? If it's a problem with yahoo, is there a better source of data?

[@peterkoppelman6232]

I made the change the Alexander suggested and still got the same type of errors. I changed df = web.get_data_yahoo(ticker, start, end) to df = download(ticker, start, end) and everything worked. I got information on all ticker symbols. It also appeared to be faster

[@ragavaarajesh]

Hello Peter, I tried to use the df = download(ticker, start, end) code. but I'm getting the error download is not defined. Can you tell me what library we have to use to define download and how to resolve this issue?

[@adamgelaw1382]

Any body else get held up on LMT stock using google? And how would I fix it . Any

[@tongbogeng7348]

Did you fix it already? I got the same problem.

[@ivansim1994]

Hi, I couldnt get my code to obtain data for LMT, but I could tell my program to ignore the exception and keep running the code. Just pass a try and except statement in the loop like so: for ticker in tickers: try: print(ticker) if not os.path.exists('stock_dfs/{}.cvs'.format(ticker)): df = web.DataReader(ticker, 'google', start, end) df.to_csv('stock_dfs/{}.cvs'.format(ticker)) else: print('Already have {}'.format(ticker)) except: print('Cannot obtain data for '+ticker)

[@AhmadRajab1]

Yes, had same problem when extracting from Google finance; I got past LMT (and others NWL, NBL) by just pretending it was already downloaded so that code would return "Already have LMT" and move onto next ticker on list. The problem was that the csv data/page is missing on Google finance for that specific ticker. So just went to stock_dfs, copied another csv file and renamed it "LMT" and then ran the code again. Did this again for NWL & NBL.

[@andantemint]

could not we just get rid of those non-csv-offered stocks when defining save_sp500_tickers() function? Since it returns a list "tickers," we can just remove them from the list using list function: tickers = [e for e in tickers if e not in ('LMT', 'NWL', 'NBL')] I don't know how long it will save me over the future tutorials (I am on pt.7 now) but it works.

[@mgrotheer]

So when I run the script the program sorta stalls out at ANDV. I hadn't seen anyone else run into this issue. I included a one second delay with time.sleep() and reran it and still have the same issue. So I'm able to get 44 of the companies and that's it. I didn't see anyone else mention this specific issue...any ideas?

[@connorbarrick3975]

i have the same issue :/

[@ishandeb334]

Same issue here. I think it maybe an issue with morningstar. I don't know if I can use Quandl.

[@fernandofranca7114]

Hey man! Use this code I created to pass the ANDV stock! for ticker in tickers: print(ticker) if not os.path.exists('stock_dfs/{}.csv'.format(ticker)): if ticker == 'ANDV': pass else: df = web.DataReader(ticker, 'morningstar', start, end) df.to_csv('stock_dfs/{}.csv'.format(ticker)) else: print('Already have {}'.format(ticker))

[@maeldk2]

Thank you I implmented it like this and it works!;) # just in case your connection breaks, we'd like to save our progress! print(ticker) if not os.path.exists('stock_dfs/{}.csv'.format(ticker)): if ticker == 'ANDV': pass else: df = web.DataReader(ticker, 'morningstar', start, end) df.reset_index(inplace=True) df.set_index("Date", inplace=True) df = df.drop("Symbol", axis=1) df.to_csv('stock_dfs/{}.csv'.format(ticker)) else: print('Already have {}'.format(ticker))

[@rednibkb1]

sentex gave you I like this...but counter went from 841 down to 840 seems to me - count should increase ??? Python Programming for Finance p.6

[@chasewillis297]

How can I add the part he was talking about at 5:30? I have downloaded all the data and would like to append new data to my existing files.

[@AndersGustafsson87]

excellent stuff!

[@czhang870]

Hi, guys! I have updated codings for anyone who watch this tutorial recently. And THANKS to Sentdex and predecessors. Thanks for your guidance. import bs4 as bs import pickle import requests import os import datetime as dt import pandas as pd import pandas_datareader.data as web def save_sp500_tickers(): resp=requests.get('en.wikipedia.org/wiki/List_of_S%26P_500_companies') soup =bs.BeautifulSoup(resp.text,'lxml') table = soup.find('table', {'id': 'constituents'}) tickers=[] for row in table.findAll('tr')[1:]: ticker = row.findAll('td')[0].text.strip().replace(".", "-") if "." in ticker: ticker = ticker.replace('.','-') print('ticker replaced to', ticker) tickers.append(ticker) with open("sp500tickers.pickle","wb") as f: pickle.dump(tickers,f) print(tickers) return tickers #save_sp500_tickers() def get_data_from_yahoo(reload_sp500= False): if reload_sp500: tickers=save_sp500_tickers() else: with open ('sp500tickers.pickle','rb')as f: tickers=pickle.load(f) if not os.path.exists('stock_dfs'): os.makedirs('stock_dfs') start=dt.datetime(2018,4,16) end=dt.datetime(2020,4,16) for ticker in tickers: print (ticker) if not os.path.exists('stock_dfs/{}.csv'.format(ticker)): df=web.DataReader(ticker,'yahoo',start,end) df.to_csv('stock_dfs/{}.csv'.format(ticker)) else: print('Already have{}'.format(ticker)) get_data_from_yahoo()

[@duinaart]

Thank you very much, this worked like a charm!!

[@malcolmtan2501]

Hi SentDex! A big fan fan of your work. ive been following u for the past 6 months and my goal is to go through all your tutorials. Also, is it possible to update this (or at least the code in pythongprogramming.net) because im unable to continue with any of these section's tutorial because im not able to access the S&P500 data for some reason.

[@malcolmtan2501]

​ smedia oh! omg it can suddenly run now .__. i ran the exact same code previously which produced an error. It is able to run now for some reason. But nonetheless thank you!!

[@simonromano]

For some reason I am getting a KeyError: 'Date' when the function gets to the EVRG ticker does anyone have any idea why??? Also @stendex, awesome videos

[@josecanizales7584]

Hey Hi! I have the same problem, Did you find a way to fixed?

[@pienuuu6631]

I had the same issue with BRK.B, but it turns out on Yahoo Finance the ticker symbol is actually BRK-B, so I think the YahooDailyReader is recognizing the wrong ticker but didn't grab any data because it's not exactly the same as what Yahoo has

[@anthonycalo4047]

I have the same issue but i get it before EVRG ticker. I'm not sure what to do

[@SuperJarvis100]

I haven't found a fix to the core problem, but I have found a way to continue. When it crashes, see which ticker it crashed on, i.e. the last one before the KeyError:'Date. Then simply remove that ticker from the list by adding "tickers.remove('XYZ')" under the line "tickers = pickle.load(f)". What also worked for me is to use a try catch. With the above method, you manually have to remove every problematic ticker, which gets tedious if there are many. Initialise "df" above the for loop with "df = pd.DataFrame". Then, put "df = web.DataReader(ticker, 'yahoo', start, end)" in the try and "tickers.remove(ticker)" in the except KeyError. Sorry for the long reply, Hope this helps!

[@ashleyez2022]

I have a question. why do we use wb and rb, why not just write and read without the bytes? what role do the bytes play?

[@step7steveX]

Sendex, thank you for your tutorials. very useful! I am using python 3.6 and pandas datareader 0.6.0. / when fetching csv form morningstar API it seams like I am getting throttled, I followed your advise and added a time.sleep(0.5) but I am still stuck.. do you have any other ways or api data sources that could overcome this problem ? thank you!

[@Hungry_Ham]

Here are the tickers that couldn't load for me: [ANDV, BKNG, BHF, CBRE, DWDP, DXC, EVRG, JEF, TPR, UAA, WELL] I would advise writing an if statement that skips collecting those. In the end I obtained 494 ticker csv files.

[@step7steveX]

I have done that. Thank you for the advise!

[@femaledeer]

It would be interesting to see how to load closing prices into to a SQL database and append the database every day with the new closing price.

[@seapanda7887]

I'm pretty sure he has some MySql and Sqllite tutorials too.

[@robfullerton6298]

pandas makes that easy too: Check out pandas.DataFrame.to_sql() Another good optimization is: for each ticker to check what is the last OHLC you have and only pull from Yahoo! (or wherever) the ones you're missing. This is a good safe-guard in case you miss a day for some reason, or for when a new tickers get listed, etc.

[@alexrodgers2922]

I don't know anything about SQL but I did create a function that updates all the csv files everyday. I've posted the modified version of the script created in this video here that has the update method on github. github.com/AlexanderRodgers/stocks/blob/master/finance_for_python.py

[@chegu071]

I was having a problem with ADBE - the wiki table has '12' next to it in the same table cell so it was pulling ADBE12 through as the ticker and looking for that with Yahoo which caused an error. I fixed it by using an if statement to check for ADBE12 then ticker = "ADBE" which seems a bit of a lazy way to do it - is there a better way to fix that? I realise you can't just cut numeric values out because tickers like 3M wouldn't work then?

[@bradengee5093]

To get the csv files to download, I had to remove " if not os.path.exists('stock_dfs/{}.csv').format(ticker):" why would that be? Thank you, im enjoying this series even though I have almost no coding experience.

[@zanehutchison7304]

It's because you have brackets around the wrong section, it should be "os.path.exists('stock_dfs/{}.csv'.format(ticker)):"

[@Ming_Qiu]

for me, inorder to get all the 500 tickets, I have to run the same code again and again. The strange thing is that for the same code, some ticket can not be generated for the first time running, but when i run the second time, it can be generated

[@clanbull3t]

In case someone ran into a problem like me recently: The code breaks right after processing CCL (Carnival Corp.) when reaching CARR (Carrier Global). Looking into the table we can see it started trading 2020-04-03. If our end of crawling is earlier, the code breaks. Fix: set end to a later date i.e: end=dt.datetime(2020,4,30) Warning : your dataset will be much bigger and probably need way more computing power /time .

[@shivajishinde1496]

I have never seen such teaching with great energy so far. you are amazing!!!. God Bless You! I tried same code but getting this error. " FileExistsError(17, 'Cannot create a file when that file already exists')". Could you please help me in this?

[@sentdex]

In what context are you getting that error? Usually files are just overwritten

[@shivajishinde1496]

line " os.makedirs('stocks_dfs') "

[@kwangyangchia9408]

Hi sentdex! I don't know if you still read these comments, but I'm just trying to shoot my shot here. I've received an EOFError saying that I've ran out of input, and while there was a comment that states i should have a stocks_dfs file in wherever my .py file is, I still receive that error. I'm not too sure how to proceed properly, if there is any guidance it would be greatly appreciated! :)

[@porlando12]

It would be interesting to compare these historical stock data against google finance's api. Do you know of an api that provides more metrics than yahoo's ohlc values?

[@samwisegamgee5184]

Hey Sentdex, how big is this series , i mean what is the extent of it. How many more videos are there in this series?

[@sentdex]

I have no idea probably about 20 parts, maybe more, maybe less.

[@samwisegamgee5184]

sentdex Thanks.

[@jaxding]

I am using Jupyter notebook and I am getting an error like below: EOFError Traceback (most recent call last) in () 47 print('Alreader have {}'.format(ticker)) 48 ---> 49 get_data_from_yahoo() 50 51 in get_data_from_yahoo(reload_sp500) 32 else: 33 with open("sp500tickers.pickle","rb") as f: ---> 34 tickers = pickle.load(f) 35 if not os.path.exists('stock_dfs'): 36 os.makedirs('stock_dfs') EOFError: Ran out of input

[@Mr1337Fabulous]

For those getting the error reading BRK.B, you have to replace . (dot) with - (dash) to match Yahoo's format.

[@ColdZera14]

IM GETTING THE SAME ERROR BUT I DIDN'T GET YOU , HOW TO DO THAT

[@robertloop7847]

holy hell I can't wait to try this shit out.... I just need to learn basic programming all of a sudden.

[@joycelynnnelson9316]

Hi! Where was the sp500 saved to? I am using Colaboratory.

[@uditvashisht878]

Hi, I have question that if, i fetch the data daily to create the CSV, will it just append the data of the current date on the already downloaded CSV file or will it replace the CSV file with new CSV having data upto the current date. And if it is not appending the data and replacing the data with new data, is there any way to append it ?

[@broy.]

How can I update the data on a daily basis without reloading it from Yahoo?

[@privateeye242]

I am stuck. I get the sp500tickers.pickle filled with the tickers. I get the directory stocks_dfs filled with the ticker data. When running the Compile_data() module I get the count printed from 10 to 500. But then I get the error: "ValueError: columns overlap but no suffix specified: Index(['MMM'], dtype='object')". I copied another csv file and gave it the name MMM.csv. Still with the same result. Where should I look at?

[@user-ze4qq8mm1q]

Kept getting hung up on CTVA when scanning from Yahoo, only to realize that CTVA was added a year ago so there is no data for it in the January 1, 2000 to December 12, 2016 date range... took me a while to realize so hopefully this comment helps someone out there.

[@sebastiandrozd18]

okay so i spent the last hour trying to fix this lmao. for some reason when i was connecting to the api's if it was yahoo or morningstar it kept saying no results. i changed the for row in table.findAll('tr')[1:]: ticker = row.findAll('td')[1].text where this is a one here use to be a 0 right after the ('td) Having left it how it was it was sending the full stock name to the api not just the ticker so now its taking the 2nd column which is just the symbol. hope it helps! and its working fine on yahoo. just downloaded 100 stocks! thanks for the tutorial!!

[@tejasshah9881]

Hey there.... Can you help me on this? I am getting this error.... How to solve this? RemoteDataError: No data fetched for symbol 3M Company using YahooDailyReader

[@sebastiandrozd18]

@@tejasshah9881 ticker = row.findAll('td')[1].text in the original video code this is set to [0]. therefore it is getting the full stock name.try it with the [1]

[@tejasshah9881]

@@sebastiandrozd18 Yeah thanks man, but I am getting new error. which I am trying to solve... KeyError: "labels ['Symbol'] not contained in axis"

[@inakigoenaga9811]

For the issue of yahoo i used fix yahoo finance and it works, i installed it like this pip install yfinance - upgrade -no-cache-dir and the imported as import yfinance as yf. It now runs correctly, im not a programer so if theres something wrong plese tell me, hope i can help

[@ZectorHolme]

Hi, what if I would like to update the database that I already have? Is there a way to write only the new lines on an already existing .csv file? Thanks

[@thomaselder4076]

1:31 Aren't you worried about cascading failures if you ever have to go back and make changes.

[@henrycwcw]

What to do if I want to get the data for all 500 companies but im stuck due to the variation of list dates? I cant have only 1 start date can I? Because it would return error when the start date is too early, which happens a lot. Would there be a way for me to store the ticker and list_date? Can I index the list_dates with ticker names and then call it during get_data_from_yahoo() function for start date?

[@andrewrooney1]

anyone facing pandas error KeyError: 'Date' while running this? already translated "." into "-"

[@kevinli1070]

Hi, I had the same problem. When I tried to fetch data for DOW, it shows me this KeyError: 'Date' message.

[@jamestedrow1901]

It's because of this line: "end= dt.datetime(2016,12,31)" Any company added to the S&P500 after that date will generate this error. To prevent it, just don't use an end date.

[@1xxxtylerxxx1]

@@jamestedrow1901 Thank you, that helped me. I did this for anyone who wants more clarity. In the get_data_from_yahoo function, I made the end date today (for me). ex: start = dt.datetime(2000,1,1) end = dt.datetime(2019,12,15)

[@epicmonckey25001]

If anyone has any problems with a date error, it is because the stock was added after 2016, or when ever you specified.

[@John-fh2mg]

I get date error when I loop through the list. I don't get it if I manually query the ticker one by one without changing the start/end dates. *shrug*

[@epicmeeltime]

I keep getting this error: pandas_datareader._utils.RemoteDataError: Unable to read URL: www.google.com/finance/historical?q=MMM&startdate=Jan+01%2C+2010&enddate=Dec+31%2C+2016&output=csv Does anyone know the problem? I am not using yahoo because 9/10 times it spits out an error

[@onlyarkasha]

@Night Owl has an answer to this issue if you look above. I also got that error so I updated the dates and changed the statement and it worked. It's because of the way the text is being pulled from Wikipedia.

[@one_lettersandnumbers]

I get this error all the time from yahoo: raise RemoteDataError('Unable to read URL: {0}'.format(url)) pandas_datareader._utils.RemoteDataError: Unable to read URL: query1.finance.yahoo.com/v7/finance/download/AME?period1=946681200&period2=1503871199&interval=1d&events=history&crumb=By7%5Cu002F2Uk16xG I get a few tickers (usually just one), and then it throws the error again. Anyone has any idea how to fix it?

[@dpoort64]

I just added a note at the top - having the exact same issue. Hopefully someone (not a NEWBIE like me, will come along and help us sort this out. Thanks for posting your error too!

[@Salirbeber1]

I have some problems downloading the data from yahoo. When I indicated some dates when there is no data of a particular ticket an exception from external libreries jumps out. The programm continue to run without problems and in fact a file with the data of the particular ticket is save without any information (thats correct). But I dont want to have the exception while the programm is running. I tried some try/except blocks but as the "problem" is from external libreries is very difficult to catch it. The exception is: Exception in thread Thread-105: ... KeyError: 'timestamp' During handling of the above exception, another exception occurred: ... ValueError: ('EGN', 'No data found for this date range, symbol may be delisted') I dont know if someone has had the same problem.

[@mattcostantino346]

I'm new to python and would really appreciate the help here. I am getting an error: KeyError: 'HistoricalPriceStore'. Also, another exception occurs which references many anaconda3 paths and lines within each of the references. Finally, it gives me a RemoteDataError: No data fetched for symbol ... Any reason why I'm receiving this? Is it because of my geography and the georouting by Yahoo. Or maybe too many requests from my IP?

[@shuaixu7299]

I got the same error, have you solved this? Thanks

[@isaakimaliev5584]

No data fetched for symbol MMM using YahooDailyReader

[@nikolezhang583]

hey thank you for the awesome video, appreciate you sharing all this knowledge with everyone. I have a question if you or any of the fellows here can help me with. When I run to grab the SP500 historical price files from Yahoo, I often get error messages for Bad Yahoo URL links, just wondering if anyone else is experiencing this error, and how do you guys deal with it? thanks!!!

[@angeloj.willems4362]

Hey Nikole, I'm not sure what you mean with 'bad URL' but I think you're running into the same problem I have. I'm not an IT expert, but my common sense tells me that if there are thousands if not millions of people all trying to request the same data from yahoo at the same time, there's gonna be some problems. I think we are getting queued and if it takes to long python returns an error. Here's what I do to solve that problem: Add try/except to the function. This allows python to move on to the next ticker if it doesn't get a response from yahoo. When you run the function the first time you won't get all the tickers, so you need to run the function a couple of times. Each time it will take less time, because the function checks to see which ticker data has already been collected. That's the - if not os.path.exists('stocks_dfs/{}.csv'.format(ticker)): - I find that after 4 to 5 times it has collected all tickers in the list. Here what the function looks like: def get_data_from_yahoo(reload_sp500=False): if reload_sp500: tickers = save_sp500_tickers() else: with open('sp500tickers.pickle','rb') as f: tickers = pickle.load(f) if not os.path.exists('stock_dfs'): os.makedirs('stock_dfs') start = dt.datetime(2017,1,1) end = dt.datetime(2017,12,31) for ticker in tickers: if not os.path.exists('stock_dfs/{}.csv'.format(ticker)): try: df = web.DataReader(ticker, 'yahoo', start, end) df.to_csv('stock_dfs/{}.csv'.format(ticker)) print(ticker) except: continue else: print('Already have {}'.format(ticker)) get_data_from_yahoo() Hope this helps :)

[@trevorweir]

@@angeloj.willems4362 Hey, That really worked. I don't know why I am so surprised, I have used try catch blocks with java over the years. Wow, so easy to forget. THANK YOU Angelo. It is adding considerably more processing time, but its running smoothly, right now for me.

[@jamesbevan1]

I ran this code and only got the tickers for 66 companies. Then I added some of the code changes mentioned in the comments as fixes to certain errors. Now I have been able to get 206/505 of the tickers. However, the same two lines of code keep causing error messages. 1) df = web.DataReader(ticker, 'yahoo', start, end) 2) get_data_from_yahoo() What is it about these lines of code that cause me issues? Why does the error message not tell you how to fix it? Update: For some reason I can now get 257/505 of the tickers, however I'm also getting this in the error message: "pandas_datareader._utils.RemoteDataError: No data fetched for symbol ISRG using YahooDailyReader" Is there a way to skip if there is an error? For example if it can't fetch the data for a ticker, it just skips it and continues with collecting the rest, rather than terminating the task?

[@aldoseitlli6034]

When i write in df = web.DataReader('TSLA', 'yahoo', start, end) i get the data but for tickers i cant where is the problem? for ticker in tickers: if not os.path.exists('stock_dfs/{}.csv'.format(ticker)): print(ticker) try: df = web.DataReader(ticker, 'yahoo', start, end) df.to_csv('stock_dfs/{}.csv'.format(ticker)) except: print('Couldnt Reload {}'.format(ticker)) else: print('Already have {}'.format(ticker))

[@aldoseitlli6034]

The problem was that i should had deleted the file before run it

[@alihazem8820]

No data fetched for symbol MMM using YahooDailyReader pls help

[@colinhsu1942]

try it if not os.path.exists('stock_dfs/{}.csv'.format(ticker)): try: df = web.DataReader(ticker, 'yahoo' , start,end) df.to_csv('stock_dfs/{}.csv'.format(ticker)) except Exception as ex: print('fucking Error', ex) else: print('already have {}'.format(ticker))

[@sheldonlavis]

hey sendex i am getting the following error, do you know why? on a mac thank yoU!! EOFError Traceback (most recent call last) in () 38 print('Already have {}'.format(ticker)) 39 ---> 40 get_data_from_yahoo() in get_data_from_yahoo(reload_sp500) 22 else: 23 with open("sp500tickers.pickle","rb") as f: ---> 24 tickers = pickle.load(f) 25 26 if not os.path.exists('stock_dfs'): EOFError: Ran out of input

[@philipbein3586]

I'm getting the same error

[@sheldonlavis]

i believe you have to have the lxml even though i am receiving a RemoteDataError when copying and pasting the code directly into Jupyter

[@Timotheos42]

Heads up for everyone watching recently, Do you know how I would write in the def loop a line of code saying to continue if the ticker is not find in Yahoo. It would be nice and help me correct this error : RemoteDataError: No data fetched for symbol LAK-USD using YahooDailyReader

[@skmn07]

When would u be releasing the further videos in this series

[@sentdex]

7 more parts to be released Monday the 23rd.

[@jameshuangmusic]

if the start and end date do not align with the historical data (i.e. the company was not in the S&P during the time frame, then the pull would not work - does anyone know how to make an exception for these cases?

[@GoredGored]

Thanks sentdex I am lost right by def get_data_from_yahoo(reload_sp500=False): Can you explain the logic behind this function? if reload_sp500: (that is if reload_sp500 is True then the below) tickers = save_sp500_tickers() but why tickers is assigned by calling the function save_sp500_tickers() and then the else? I appreciate if anyone can help explain the logic behind this function.

[@saadahmed9239]

I was able to fix the yahoo api issue by looking at the comments but I have one more problem When I download the stock data most of the files in Stock Df's folder are 1Kb in size, why is it like this any suggestions?

[@jackkoh6174]

I'm having problem with "if reload_s&p500: ", it shows "undefined name". I'm using python 3.6. Do I need to import library to use it?

[@GamersGoneExtinct]

if reload_sp500 == True: tickers=save_sp500_tickers() works for me

[@kennyPAGC]

how come the sp500 table has more than 500 entries? 505 to be exact

[@sentdex]

Because there are more than 500 companies in the S&P 500!

[@kennyPAGC]

oh, that's kind of misleading haha thanks!

[@peterkoppelman6232]

Some companies have more than one class of stock (ie. GOOG and GOOGL)

[@tubehossein]

How can I download the data under the Statistics tab for all the S&P500 companies in Yahoo Finance?

[@Fiddelush]

Got the error "FileNotFoundError: [Errno 2] No such file or directory: 'stocks_dfs/MMM.csv'" when using quandl as a API. Dont know how to come around this issue..

[@clarkjr6]

I am getting the same issue with IEX API. Did you discover a fix?

[@hyakushiki23]

I had the same problem and what I found was I had a spelling error: df.to_csv('stocks_dfs/{}.csv'.format(ticker)) Since I had to type it twice, one time I had "stocks" and the other was "stock" Once I set both as 'stocks', it worked fine. Hope this helps and to anyone else.

[@pawpawprince104]

@@hyakushiki23 wow, I made that exact same mistake and yours was the only comment in here that could help me. Don't know what I would've done without this, thx!

[@danidin77]

But I get 'pandas_datareader._utils.RemoteDataError: Unable to read URL' after a few stocks each time...:( What can be done ???

[@matt_t937]

For some tickers I get that yahoo cannot fetch the data required, how can I solve this?

[@c-spam9581]

The Yahoo api is no longer working. The Google API is but doesn't provide an adjusted close. Do you have any suggestions on how I can move forward with this tutorial?

[@sentdex]

You can just use close instead.

[@c-spam9581]

Thanks for the fast reply.

[@Aerozine50]

There is a module you can download and if you just changed a couple lines you're good to go, it's called "fix_yahoo_finance". I actually got it from the comment section here.

[@c-spam9581]

Aerozine I noticed that solution in the comments as well but was hoping to find another data source before installing a module. Thank you for assisting though. Just another opportunity for someone to find the fix.

[@kunomels]

Google API won't give data for some tickers. MMM for example

[@abluntdaily]

if I rerun the function again will it create duplicate files or continue where it left off? its taking a really long time so I think I might be getting throttled. On my last attempt I was able to get 60 files but then I deleted the folder and it started right up again when I ran the function. I have 256 files now so I don't want to delete the folder but I also don't want to rerun the function if it will create duplicate files. Its been about an hour that I am stuck at 256 files

[@abluntdaily]

Im just going to run it again. I imagine windows should prompt that the file is already there and only copy the ones I need..

[@sentdex]

Windows will *not* prompt you when overwriting files with Python. The reason we have the following code: for ticker in tickers: if not os.path.exists('stock_dfs/{}.csv'.format(ticker)): df = web.DataReader(ticker, "yahoo", start, end) df.to_csv('stock_dfs/{}.csv'.format(ticker)) else: print('Already have {}'.format(ticker)) is precisely for this reason, however. If the ticker data is already there, we wont pull it again.

[@akshatjain8052]

I am getting this error- OSError: [Errno 22] Invalid argument: 'stock_dfs\\MMM .csv' Can anyone explain to me what this error means and what is the fix?

[@gehtomacgyver]

Hopefully, this helps you out, I haven't worked on this in a while. Full code, working as of 4-19-2020. Corrects for those who kept getting an error that "no data fetched for symbol MMM (or another stock ticker ) using yahoodailyreader" import bs4 as bs import datetime as dt import os from pandas_datareader import data as pdr import pickle import requests import fix_yahoo_finance as yf yf.pdr_override def save_sp500_tickers(): resp = requests.get('en.wikipedia.org/wiki/List_of_S%26P_500_companies') soup = bs.BeautifulSoup(resp.text, 'lxml') table = soup.find('table', {'class': 'wikitable sortable'}) tickers = [] for row in table.findAll('tr')[1:]: ticker = row.findAll('td')[0].text.replace('.', '-') ticker = ticker[:-1] tickers.append(ticker) with open("sp500tickers.pickle", "wb") as f: pickle.dump(tickers, f) return tickers # save_sp500_tickers() def get_data_from_yahoo(reload_sp500=False): if reload_sp500: tickers = save_sp500_tickers() else: with open("sp500tickers.pickle", "rb") as f: tickers = pickle.load(f) if not os.path.exists('stock_dfs'): os.makedirs('stock_dfs') start = dt.datetime(2019, 6, 8) end = dt.datetime.now() for ticker in tickers: print(ticker) if not os.path.exists('stock_dfs/{}.csv'.format(ticker)): df = pdr.get_data_yahoo(ticker, start, end) df.reset_index(inplace=True) df.set_index("Date", inplace=True) df.to_csv('stock_dfs/{}.csv'.format(ticker)) else: print('Already have {}'.format(ticker)) save_sp500_tickers() get_data_from_yahoo()

[@gogytdk6215]

This is what I get when i copy paste final code Traceback (most recent call last): File "finance4.py", line 48, in get_data_from_yahoo() File "finance4.py", line 32, in get_data_from_yahoo tickers = pickle.load(f) File "/usr/lib/python2.7/pickle.py", line 1384, in load return Unpickler(file).load() File "/usr/lib/python2.7/pickle.py", line 864, in load dispatch[key](self) File "/usr/lib/python2.7/pickle.py", line 886, in load_eof raise EOFError EOFError

[@JeremAl]

I had the EOF error with pickle. So I switched in the FIRST function: pickle.dump(tickers, f, protocol=pickle.HIGHEST_PROTOCOL) I erased the .pickle file and rewrote it with that method. Then it loaded ok. Also I used google instead of Yahoo which gave me error in the previous videos and here too: df = web.DataReader(ticker, 'google', start, end)

[@rodneysimpson4604]

Hi Everyone, bit new to this and hoping there might be a few Aussies out there. Trying to run same code as sentdex, but on the ASX200 (Australian top 200 companies). Everything seems to work perfectly... tickers list created ok and ticker.csv files created upto PPT.AX.csv (140th csv file ) then it crashes with following error: FileNotFoundError: [Errno 2] No such file or directory: 'ASXstock_dfs/PRN.AX.csv' .....exact error dump below. Very confused to why it would just crash after successfully executing 140 times. hmmmm.... Any help greatly appreciated. Already have PNV.AX ['A2M.AX', 'ABC.AX', 'ABP.AX', 'AGL.AX', 'ALL.AX', 'ALQ.AX', 'ALU.AX', 'ALX.AX', 'AMC.AX', 'AMP.AX', 'ANN.AX', 'ANZ.AX', 'APA.AX', 'APD.AX', 'APE.AX', 'APT.AX', 'APX.AX', 'ARB.AX', 'ASB.AX', 'AST.AX', 'ASX.AX', 'AVH.AX', 'AWC.AX', 'AZJ.AX', 'BAP.AX', 'BEN.AX', 'BGA.AX', 'BHP.AX', 'BIN.AX', 'BKL.AX', 'BKW.AX', 'BLD.AX', 'BOQ.AX', 'BPT.AX', 'BRG.AX', 'BSL.AX', 'BVS.AX', 'BWP.AX', 'BXB.AX', 'CAR.AX', 'CBA.AX', 'CCL.AX', 'CCP.AX', 'CGC.AX', 'CGF.AX', 'CHC.AX', 'CIM.AX', 'CKF.AX', 'CLW.AX', 'CMW.AX', 'CNU.AX', 'COE.AX', 'COH.AX', 'COL.AX', 'CPU.AX', 'CQR.AX', 'CSL.AX', 'CSR.AX', 'CTD.AX', 'CUV.AX', 'CWN.AX', 'CWY.AX', 'DHG.AX', 'DMP.AX', 'DOW.AX', 'DXS.AX', 'EHE.AX', 'ELD.AX', 'EML.AX', 'EVN.AX', 'FBU.AX', 'FLT.AX', 'FMG.AX', 'FPH.AX', 'GEM.AX', 'GMG.AX', 'GNC.AX', 'GOR.AX', 'GOZ.AX', 'GPT.AX', 'GUD.AX', 'GWA.AX', 'HLS.AX', 'HUB.AX', 'HVN.AX', 'IAG.AX', 'IEL.AX', 'IFL.AX', 'IGO.AX', 'ILU.AX', 'INA.AX', 'ING.AX', 'IPH.AX', 'IPL.AX', 'IRE.AX', 'IVC.AX', 'JBH.AX', 'JHG.AX', 'JHX.AX', 'JIN.AX', 'LLC.AX', 'LNK.AX', 'LYC.AX', 'MFG.AX', 'MGR.AX', 'MIN.AX', 'MMS.AX', 'MND.AX', 'MPL.AX', 'MQG.AX', 'MTS.AX', 'MYX.AX', 'NAB.AX', 'NAN.AX', 'NCM.AX', 'NEA.AX', 'NEC.AX', 'NHC.AX', 'NHF.AX', 'NSR.AX', 'NST.AX', 'NUF.AX', 'NWH.AX', 'NWL.AX', 'NWS.AX', 'NXT.AX', 'OML.AX', 'ORA.AX', 'ORE.AX', 'ORG.AX', 'ORI.AX', 'OSH.AX', 'OZL.AX', 'PDL.AX', 'PLS.AX', 'PME.AX', 'PMV.AX', 'PNI.AX', 'PNV.AX', 'PPT.AX', 'PRN.AX', 'PTM.AX', 'QAN.AX', 'QBE.AX', 'QUB.AX', 'REA.AX', 'RHC.AX', 'RIO.AX', 'RMD.AX', 'RRL.AX', 'RSG.AX', 'RWC.AX', 'S32.AX', 'SAR.AX', 'SBM.AX', 'SCG.AX', 'SCP.AX', 'SDF.AX', 'SEK.AX', 'SFR.AX', 'SGM.AX', 'SGP.AX', 'SGR.AX', 'SHL.AX', 'SIQ.AX', 'SKC.AX', 'SKI.AX', 'SLR.AX', 'SOL.AX', 'SPK.AX', 'SSM.AX', 'STO.AX', 'SUL.AX', 'SUN.AX', 'SVW.AX', 'SXL.AX', 'SYD.AX', 'TAH.AX', 'TCL.AX', 'TGR.AX', 'TLS.AX', 'TNE.AX', 'TPG.AX', 'TWE.AX', 'URW.AX', 'VCX.AX', 'VEA.AX', 'VOC.AX', 'VUK.AX', 'VVR.AX', 'WBC.AX', 'WEB.AX', 'WES.AX', 'WHC.AX', 'WOR.AX', 'WOW.AX', 'WPL.AX', 'WSA.AX', 'WTC.AX', 'XRO.AX'] ['A2M.AX', 'ABC.AX', 'ABP.AX', 'AGL.AX', 'ALL.AX', 'ALQ.AX', 'ALU.AX', 'ALX.AX', 'AMC.AX', 'AMP.AX', 'ANN.AX', 'ANZ.AX', 'APA.AX', 'APD.AX', 'APE.AX', 'APT.AX', 'APX.AX', 'ARB.AX', 'ASB.AX', 'AST.AX', 'ASX.AX', 'AVH.AX', 'AWC.AX', 'AZJ.AX', 'BAP.AX', 'BEN.AX', 'BGA.AX', 'BHP.AX', 'BIN.AX', 'BKL.AX', 'BKW.AX', 'BLD.AX', 'BOQ.AX', 'BPT.AX', 'BRG.AX', 'BSL.AX', 'BVS.AX', 'BWP.AX', 'BXB.AX', 'CAR.AX', 'CBA.AX', 'CCL.AX', 'CCP.AX', 'CGC.AX', 'CGF.AX', 'CHC.AX', 'CIM.AX', 'CKF.AX', 'CLW.AX', 'CMW.AX', 'CNU.AX', 'COE.AX', 'COH.AX', 'COL.AX', 'CPU.AX', 'CQR.AX', 'CSL.AX', 'CSR.AX', 'CTD.AX', 'CUV.AX', 'CWN.AX', 'CWY.AX', 'DHG.AX', 'DMP.AX', 'DOW.AX', 'DXS.AX', 'EHE.AX', 'ELD.AX', 'EML.AX', 'EVN.AX', 'FBU.AX', 'FLT.AX', 'FMG.AX', 'FPH.AX', 'GEM.AX', 'GMG.AX', 'GNC.AX', 'GOR.AX', 'GOZ.AX', 'GPT.AX', 'GUD.AX', 'GWA.AX', 'HLS.AX', 'HUB.AX', 'HVN.AX', 'IAG.AX', 'IEL.AX', 'IFL.AX', 'IGO.AX', 'ILU.AX', 'INA.AX', 'ING.AX', 'IPH.AX', 'IPL.AX', 'IRE.AX', 'IVC.AX', 'JBH.AX', 'JHG.AX', 'JHX.AX', 'JIN.AX', 'LLC.AX', 'LNK.AX', 'LYC.AX', 'MFG.AX', 'MGR.AX', 'MIN.AX', 'MMS.AX', 'MND.AX', 'MPL.AX', 'MQG.AX', 'MTS.AX', 'MYX.AX', 'NAB.AX', 'NAN.AX', 'NCM.AX', 'NEA.AX', 'NEC.AX', 'NHC.AX', 'NHF.AX', 'NSR.AX', 'NST.AX', 'NUF.AX', 'NWH.AX', 'NWL.AX', 'NWS.AX', 'NXT.AX', 'OML.AX', 'ORA.AX', 'ORE.AX', 'ORG.AX', 'ORI.AX', 'OSH.AX', 'OZL.AX', 'PDL.AX', 'PLS.AX', 'PME.AX', 'PMV.AX', 'PNI.AX', 'PNV.AX', 'PPT.AX', 'PRN.AX', 'PTM.AX', 'QAN.AX', 'QBE.AX', 'QUB.AX', 'REA.AX', 'RHC.AX', 'RIO.AX', 'RMD.AX', 'RRL.AX', 'RSG.AX', 'RWC.AX', 'S32.AX', 'SAR.AX', 'SBM.AX', 'SCG.AX', 'SCP.AX', 'SDF.AX', 'SEK.AX', 'SFR.AX', 'SGM.AX', 'SGP.AX', 'SGR.AX', 'SHL.AX', 'SIQ.AX', 'SKC.AX', 'SKI.AX', 'SLR.AX', 'SOL.AX', 'SPK.AX', 'SSM.AX', 'STO.AX', 'SUL.AX', 'SUN.AX', 'SVW.AX', 'SXL.AX', 'SYD.AX', 'TAH.AX', 'TCL.AX', 'TGR.AX', 'TLS.AX', 'TNE.AX', 'TPG.AX', 'TWE.AX', 'URW.AX', 'VCX.AX', 'VEA.AX', 'VOC.AX', 'VUK.AX', 'VVR.AX', 'WBC.AX', 'WEB.AX', 'WES.AX', 'WHC.AX', 'WOR.AX', 'WOW.AX', 'WPL.AX', 'WSA.AX', 'WTC.AX', 'XRO.AX'] Traceback (most recent call last): File "C:/Users/Simpson M17X Admin/Finance 3 P38/FinanceASX.py", line 66, in get_data_from_yahoo() # takes long time to run to create all 200 csv files File "C:/Users/Simpson M17X Admin/Finance 3 P38/FinanceASX.py", line 60, in get_data_from_yahoo df.to_csv('ASXstock_dfs/{}.csv'.format(ticker)) File "C:\Users\Simpson M17X Admin\Finance 3 P38\lib\site-packages\pandas\core\generic.py", line 3170, in to_csv formatter.save() File "C:\Users\Simpson M17X Admin\Finance 3 P38\lib\site-packages\pandas\io\formats\csvs.py", line 185, in save f, handles = get_handle( File "C:\Users\Simpson M17X Admin\Finance 3 P38\lib\site-packages\pandas\io\common.py", line 493, in get_handle f = open(path_or_buf, mode, encoding=encoding, errors=errors, newline="") FileNotFoundError: [Errno 2] No such file or directory: 'ASXstock_dfs/PRN.AX.csv'