Making a Modeler S1E2 - Making Your First Web Scraper

[@montespiard4496]

Great steps for me starting out

[@themagictraindriver]

This is exactly what i've been looking to do. It's been daunting thinking about this but this has been a massive help. Thank you so much

[@jackzwart5749]

This is exactly what I was looking for as a beginner ... thank you so much sir.

[@petermayer5350]

Thanks so much for these!!! Scraping has always been my bugaboo.

[@dsizzle49]

Damn this was helpful, I'm here doing a python course wondering why I'm so dumb.

[@rhodesjf1]

Captain Jack, thank you for this and all your great videos. In a world filling w/data web scraping seems like a valuable skill. I heard good things about the python library Beautiful Soup but don't know python. Excel still really powerful. 🙏

[@CaptainJackAndrews]

Yes, I'd definitely encourage viewers who want to get serious about this to move beyond Excel when they are ready. Scraping with BeautifulSoup in Python has a learning curve, but not an impossible one.

[@thandogy]

Love your work cheers from Australia 💪

[@CaptainJackAndrews]

Thank you! Cheers!

[@ss-dr2el]

Never disappointing Captain...thanks.

[@merdenoms2590]

Learned about you through Gill Alexander's show and this stuff is incredible. I set with a note pad out while drinking my morning coffee, you do a fantastic job of teaching and showing too.

[@jackmccormick63]

Thanks for these.

[@CaptainJackAndrews]

You are welcome. More to come!

[@wreckingcrew52]

There should be an O/U on how fast 💨 I click the Iike button on a @Captain Jack Andrews video!!!

[@hamsammer103]

Good stuff. What method do you use when the data source isn't a tidy numeric table? Let's say you wanted to scrape NFL player grades from PFF.

[@CaptainJackAndrews]

You'll need to use something more robust like BeautifulSoup in Python. Much steeper learning curve there but not insurmountable.

[@1jruby]

Is anyone using Excel on a Macbook and having trouble getting web data? I can't figure out if it's an Excel issue or a Macbook issue.

[@tomursano4851]

Hi Jack, thanks again for the great video. Would it make sense to apply the same transformation method as you did for 3PA & 3P% to the standard deviation in order to get a projected std dev rather than simply using the sample std dev in the Monte Carlo sim?

[@CaptainJackAndrews]

Your standard deviation is already a controlled measure of your variance. If you add variance to that you are undermining its role in the projection.

[@saiedkhan716]

Having trouble pulling in some of the data from basketball-reference in excel the same way Capt does. The standings tables can pulled into Excel but Team Stats, Opponent Stats, and Miscellaneous Stats aren't recognized. Is this common? Is this the reason to professionals use R or Python?

[@Foost]

I have the same issue. I assume it is a bug with the website currently

[@CaptainJackAndrews]

You're right. Something seems to have changed at basketball-reference. Not sure if it's permanent or temporary. I'll try to reach out to them to find out more. Did I mention in the video that web scrapers will break? 😉

[@Foost]

@@CaptainJackAndrews Thanks for your help. I guess you were right about them breaking lol. For now I was actually able to find a way around it. I went to the specific table I wanted in basketball reference and click embed table. Which for example pulled up () I then just took the link out of that (widgets.sports-reference.com/wg.fcgi?css=1&site=bbr&url=%2Fleagues%2FNBA_2021.html&div=div_team-stats-per_game). I put that into google sheets via import html which does work. Then I published the google doc to the web and was able to use that link on excel to get the live information working.

[@eastquad]

@@CaptainJackAndrews you did mention that -- ha! I am experiencing the same issue, reached out to the folks at Sports Reference, and the individual responded with: "No we have not made any format change on the site recently. Our site isn't set up to support automatic import systems like that anyway, so I'm afraid I can't provide much troubleshooting on that front." So, either something changed with PowerQuery, or the folks at Stathead changed something without consideration for PQ. Sigh.

[@eastquad]

@@Foost Thank you!

[@joeycarroccio]

Is there any comparable add-ons to power query for Google sheets that you know of?

[@CaptainJackAndrews]

I don't have any personal experience with any, but since there's a large market for add-ons to the Google Drive apps, I would be surprised if someone hasn't tried to fill that void.

[@joeycarroccio]

Thank for the reply and thanks for the great content. Keep it up

[@STree42]

What tool are you using to show your PC screen with you still visible in the lower right PIP?

[@CaptainJackAndrews]

Two separate video feeds (DSLR and Screen Capture) synced together.

[@matthewmagee5117]

Has the site mentioned in the video blocked scraping? Can anyone confirm?

[@CaptainJackAndrews]

They have made some modifications since this video was done in 2020 that make it not as easy as shown. You can still use scraping tools like Python or R, but using Excel is tougher. You can export to CSV and go that route if you'd like.

[@John_K_85]

Any recommendations for Mac users? Looks like power query isn't available...

[@CaptainJackAndrews]

I’ve heard Power Query support for HTML is coming for Excel Mac, in the meantime there is the standard Web Query available from the Data, Import menu. Not as powerful, but should still get the job done.

[@jward7993]

How did you get it on google sheets?

[@joshroberts5661]

I use R programming. Might have to look into python.

[@stevengill8864]

You are 2 for 2 Captain. How did you learn python? I am taking an online class now.

[@CaptainJackAndrews]

Python I was largely self-taught. Followed some online tutorials. Online classes are good as well. There's a lot more to Python that I'd like to learn. I know how to do what I need to do but that's about it. Thanks for watching!

[@MichaelMurray-bp6oo]

Captain Jack is the best

[@1jruby]

I don't think excel on Macbook allows these features. My from web button says HTML and it doesn't load in as your screen showed. dumb Apple.

[@sethsurendonk3165]

When scraping historical data, how far back is actually useful? Say data on the Boston Bruins from 1950.

[@jward7993]

It doesn't work like that for sheets

[@CaptainJackAndrews]

The IMPORTHTML function in Google Sheets has some usability when it comes to scraping. However, you may need to find a 3rd party add-on to do more extensive scraping in Sheets. Check in the Add-ons menu.

[@jward7993]

@@CaptainJackAndrews thank you so much love the videos. I learned a lot. Thank you for taking the time to reach out too.

[@sethsurendonk3165]

When scraping historical data, how far back is actually useful? Say data on the Boston Bruins from 1950.

[@CaptainJackAndrews]

Depends on what you're looking for from the data. If you survived WWII, might as well go play goalie without a mask. 😉