How to scrape SPORTS STATS websites with Python

In this video we scrape a sports stats website by accessing the api endpoint. utilising Postman we can replicate the request that the browser makes and download all of the data into Python and VSCODE. Using Pandas we can create a dataframe with all the player stats ready to be analysed, or exported to a csv file.
www.postman.com/
Thank you to Daniel for sending this over, and if you've never watched any AFL before search for AFL biggest hits on KZbin..!!
-------------------------------------
twitter / jhnwr
code editor code.visualstu...
WSL2 (linux on windows) docs.microsoft...
-------------------------------------
Disclaimer: These are affiliate links and as an Amazon Associate I earn from qualifying purchases
mouse amzn.to/2SH1ssK
27" monitor amzn.to/2GAH4r9
24" monitor (vertical) amzn.to/3jIFamt
dual monitor arm amzn.to/3lyFS6s
microphone amzn.to/36TbaAW
mic arm amzn.to/33NJI5v
audio interface amzn.to/2FlnfU0
keyboard amzn.to/2SKrjQA
lights amzn.to/2GN7INg
webcam amzn.to/2SJHopS
camera amzn.to/3iVIJol
gfx card amzn.to/2SKYraW
ssd amzn.to/3lAjMAy

[@celerystalk390]

I learn something new in every video of yours! Looking forward to learning more from you. Keep up the great work!

[@obinnaihekona1199]

This video singlehandedly helped me with a project that kept me up for 2 nights straight. Thank you so much for your content.

[@JohnWatsonRooney]

That’s great! Im glad it helped

[@canal_vhs]

Awesome job. I'll need several months to learn everything you're teaching us. Thank you!

[@sayantansarkar6433]

Mind blowing explanation and coding.. hats off my friend..

[@brkerd6865]

Easy, fast and understandable tutorial. Thanks for this, It is just what I'm looking for

[@brkerd6865]

I also subbed you are amazing

[@gisleberge4363]

Very elegant and smooth solution

[@fsamobby]

Thanks for the valuable tip! This postman service is exactly what i was looking for but don't even realise d this)) knowing what to ask is sometimes more important than the answer itself)

[@JohnWatsonRooney]

Glad it was helpful!

[@pr0skis]

Awesome video! Thanks for introducing me to Postman. It helped me through a hurdle I've been struggling with for a while haha

[@JohnWatsonRooney]

Glad you liked it!

[@julianhristov5844]

Excellent job John, best tutorials ever. I applying for job, which they want me to do this kind of task and I do it perfect with your expert level knowledge .Thanks again.

[@JohnWatsonRooney]

That’s great!

[@brandonbergstrom4289]

Great video! Took me two minutes of watching this video and solved my problem. Thank you for your hard work and sharing!

[@Epitaxi]

Simple amazing guide. I was able to scrape all player data from the Swedish Allsvenskan fantasy site which I don't think is a public api.

[@JohnWatsonRooney]

Great stuff glad it helped!

[@edsonmatheus7976]

Wow, I'm impressed. 13 minutes of pure master class, this video was so detailed! Got a new subscribe, greeting from Brazil!!!

[@chr1sto7]

I don’t know why you don’t have more subscribers. You’re great man. Keep up the good work. You really helped me with a web scraping project for my parents and you have given me the tools to begin a project at work using a REST api. Really helped me a lot! Thanks!!!

[@ephraimmotho887]

2 years have gone by. Subscribers: 44.1k (and growing...). Consistency is beautiful!!! Well done John

[@stuarthoughton3517]

Another great walkthrough, John!

[@arabymo]

Wonderful explanation! ❤️

[@os2002n77]

Thank you very very much! I learned a lot of new things from your videos

[@tomkmb4120]

Great video man, nice job.

[@ryanwest8058]

This is an Awesome Python Video!!

[@sandutiron8004]

Your videos are inspiring me every single day!

[@arzigogolato1]

Great tutorial! I have been fighting with scraping this sort of stuff for a while, you made my work so much easier! Thanks man :)

[@engineerbaaniya4846]

You have rocked

[@abhinavshah138]

Hey John, can you scrap Walmart (title & price) using bs4? Would love a video on that! Keep up the good work ❤️👌🏻

[@BathingAfrican]

man i love this channel idek how i found it

[@susanwowe7810]

Really a great tutorial. First time to know the code tab in postman and the pd.json_normalize() method. Thumbs up 👍

[@berdjarafik6283]

thanks a lot, if u may please make us a video on how to clean datasets once scraped. especially, when the data set is messy and not well structured (column containing two types of data format; ie : counts & date time). thank you very much.

[@JohnWatsonRooney]

Good idea, would be very useful

[@rakeshchaudhary1525]

Hi John- I have been following your awesome scrapy and webscraping videos. Can you create a video that explains how to build and then deploy large scale web scrapers ? For example: in this case, how to call API for multiple pages ? Considering the video time and API call limits, Even if you show 2 automated iterations, that would be great.

[@kimbanzip]

Thanks ! Great explanation.

[@radgeradge]

been trying a bunch of different approaches on this and this one is definitely the simplest! can i just ask (as im a novice), if im looking to rename the column headers would it be best for me to just use the df.rename function?

[@profoundgenius9726]

Just dropped a 💎!! Subbed

[@meedip07]

thanks and awesome. Kudos!!

[@MatheusSilva-qm3ph]

Very cool the video, congratulations. A question, still to use the "user-data-dir"? I'm not getting, could you help you please?

[@brianschroeder2465]

Thanks for the video! Question is this token going to expire eventually that is used in the request?

[@JohnWatsonRooney]

Yes it will - this sort of scraping is more of a on off get the data and run type rather than something that can be run daily or weekly easily

[@brianschroeder2465]

@@JohnWatsonRooney Cool, thanks for the info and inspiration. I was able to get what I needed with a library and an api. Here's my repo, still have alot of tuning to do any recommendations would be appreciated!: github.com/brianschroeder/NBAGamePredictor/blob/main/NbaGamePredictor.py

[@JohnWatsonRooney]

That’s a great project idea Brian - I’ll definitely going to check out your repo!

[@avexoidavex3577]

How can we automate the analysis of the obtained data? Suppose we need to collect several tables and compare data on some columns and rows. Is it possible to launch your own bot for this?

[@sskinedGUY]

What if the website doesn't return json reponse when inspecting in the Network tab, for example bet365, all the response types are 'plain", any advice ?

[@yajantbhawsar2481]

Hello john , could you please make a video on "desktop scrapping"?

[@zhaofour9833]

This is absolutely fantastic Sir. I wonder if I wanna get the live data, such as live odds and live score, can I still use this method?

[@Cedric_0]

Good video, well explained, but I am stuck at gziped csv response how can I get that csv in python, thanks in advance

[@jacksonmorris6717]

hi new to this, I followed all your code I just seem to have an error at import requests. yellow line under requests. do I need to install a package not sure what's happening. thanks

[@JohnWatsonRooney]

Hey, sure you should have “pip” installed and can run “pip install requests” in the terminal or command prompt

[@tariqanwar8097]

great video

[@coldfya6739]

u saved my life

[@yusufrumi1626]

You are a gem

[@KokeBeast23]

Is there a reason I don't get the option of 'Code' next to 'Cookies'?

[@terranovatech8672]

My API call was only the first 50 players... How did you manage to pull every player? It's strange, the JSON displays 8100+ players total, but when I export to CSV it only gives me 50....

[@JohnWatsonRooney]

Is your json response showing all 8k? If so I’d check your loops and export to see if there is an error there?

[@ereaUA]

Hello! Nice explanation, but I have a question. Could you give me any tips on how to get "scraped" a numeric odds for specific page in live? Say, get 3 values from 1 page every 20 seconds. Maybe there are already some tools exist. I am a newbie in programming. With respect, Ivan

[@lordmo3416]

1:28 the type isn't json for me. It's fetch and when I plug the copied raw text into postman, it throws an importing Curl error to me. The page also makes the request automatically every 10 seconds or so; with a different cf-request-id, set-cookie, etag, and without x-media-mis-token in the response header. Sofascore response is all rather perplexing

[@abiodunogunnaike2688]

God bless you bro, you just read my mind. i think you are using firefox, what extention did u use to preview the response? i mean the red box

[@JohnWatsonRooney]

Thanks! Yes I am using Firefox- no extensions though the response is shown in the inspect element tool by clicking on the get request

[@kwamstercmg8968]

what do I do when there is no api to scrape ?

[@skylucacius8463]

How to automate this process ? I.e. How can I get XHR information programatically (from python script)

[@PredatorsUltimateSoccerChannel]

I’m having a tough time finding a way to do this using C++(i don’t know whether its even possible).If anyone knows how please leave a link with an explanation down below. Thank you in advance

[@KokeBeast23]

I don’t get the option to copy as Curl (windows) only (cmd) or (bash) :((

[@KokeBeast23]

Using Firefox instead of Microsoft SE fixed this

[@kanglee2377]

Hey I wanted to ask if you could share how to scrape the odds from a betting site/a book maker. I can't get it to work for some reason

[@alejandroramos4926]

Hi John, I tried to search for information in a similar way to how you show in the video but on the flashscores site, I don't know about data transmission via json, could you tell me if on that site is the right way?

[@zholl005]

you should check out thunder client extension in vs code. Got me to uninstall postman

[@JohnWatsonRooney]

Yeah I've heard of that, I should look into it more. I like to keep my API program separate though I guess its just what I'm used to

[@eu.pedroleon]

Thanks a lot

[@lifeofreggiewoods]

Cool, now how do I implement these stats to my html website?

[@benhall6327]

How do you get the api-endpoint commands there? my code says cant find the import "requests" and those api endpoints dont show up in the terminal.

[@dailyhustlemoti]

You need to install the requests package. Type pip install requests into your vs terminal

[@ugurdev]

Thanks John, I have done the same process for NBA but unfortunately the code Insomnia generates does not work.

[@JohnWatsonRooney]

I haven’t checked that specifically but it’s possible they have better detection for this method

[@ugurdev]

@@JohnWatsonRooney I figured it out eventually, it is just insomnia not generating the correct code or sometimes their output has broken code. (For example double quotes inside double quotes breaking the string format) Ran into this on 2 more cases. Can work around this with copying request headers from the dev tool though.

[@cuccuu1730]

@Ugur Did you find the stats files on the official NBA website? Because I can't find which file I need...

[@binhole]

how get scrapping bac bo - Evoluting gaming

[@richtay.7]

I need to make a Scrapy from a betting site, but it's a casino, I want to make a Scrapy to collect the information from the roulette rounds, and do an analysis on top of that, I just need to get the result data, is it possible? I'm having a hard time getting response from the site, always giving 403 Forbbiden error

[@JohnWatsonRooney]

Honestly I don’t tent to touch betting sites, they often have the best bot and scraping protection. If you want to analyse data maybe there is a data set already available that someone else has released ?

[@richtay.7]

@@JohnWatsonRooney Like a similar project? I'm trying to look for some here, the site I want to make Scrapy uses frameworks in Js, I'm trying little by little to get information and apply it to my code, at first I wasn't able to communicate with the site, but I got it, but the information I want is in encoded javascript, so I need to make my code

[@hammadalamsiddique3730]

​@@richtay.7use selenium

[@KhalilYasser]

Thanks a lot. You are a rock. Can you share the code please?

[@vanderleimanoeldias1988]

manda o code pa nois

[@NoTengoIdeaGuey]

Easiest way to scrape data: don't scrape the data.

[@bluiish999]

poor explanation

[@andresalcocer4544]

Absolutely fantastic! really loved this tutorial, straight to the point and useful information!

[@IonidisIX]

Fantastic John. Thank you so much mate.

[@ichfan9811]

Thanks man, I really enjoyed your content!

[@JohnWatsonRooney]

Hey thank you very much!

[@dfsed1057]

Heck of a video; been doing data scraping for a while and yet learned several new things here. instant subscriber

[@joilsongb436]

can you do this method on the page b e t 3 6 5 ? I couldn't with the instruction in this video Delete the spaces between the words