Basic Webscraper : Get info from the web with Python

[@JohnWatsonRooney]

Questions about basic web scraping always pop up so I wanted to answer them and help out with this video.

[@Neil4Speed]

Hi John, great tutorial as always. Only addition I would recommend is showing how to take it past the finish line and export to a CSV

[@JohnWatsonRooney]

Yes of course, I have since covered this in my later videos! Thanks!

[@jimmyporter8941]

A great pragmatic intro to webscraping. Thanks!

[@faisalrkhawaja]

Hi John. I have two challenges in my scraping project: 1) the products must first have a search term entered (e.g. the product name or category, etc.), 2) the results are spread over multiple pages (which your video did cover), but the results I need are divided over several tabs.

[@faisalrkhawaja]

Hi John. Total noob here. This is the first of your videos I have watched. It is super cool! Question: Target sites will often change the product list (adding or removing), and I may want to keep the data updated on my end at same time. Is there a way to inspect the landing page to see how many pages need to be scraped, and put that into the code as a reference to the last page number, as opposed to a hardcoded number?

[@daveys]

Worked all the way through this. Great tutorial, many thanks!!

[@merttarm848]

thanks for the video, amazing introduction to webscraping

[@powerquotes9492]

John, your channel is amazing! Exactly what I was looking for. I'm gonna study all your videos and cancel my Udemy course too, as you have better content and for free.

[@venkateshgolla8005]

Excellent..your explanation is awesome.... If possible can you please make an another video, which includes to get data after clicking some buttons in web page(like Radio buttons, list box, button).

[@JohnWatsonRooney]

Hi! Thanks for the comment. In my other video I do this using browser automation - How I use SELENIUM to AUTOMATE the Web with PYTHON. Pt1 kzbin.info/www/bejne/poa4mamAq9hgiJY - around the 10 min mark. I will have more webscraping videos coming up too.

[@shivasuresh5957]

Awesome video John. Thanks! I will now try to learn how to add the data to a .CSV. Would I be on the right path by using a Pandas data frame to do this?

[@JohnWatsonRooney]

It’s definitely worth knowing how to do it with the CSV module, but yes use pandas - I do all the time

[@powerquotes9492]

Just add these lines to save everything to .CSV. It worked for me: import pandas as pd table = pd.DataFrame(book_list) table.to_csv('name_your_file.csv')

[@spearchew]

A* tutorial - and besides the scraping, useful for learning about python more generally. For instance, before today I probably would have created three or four empty lists and appended to each of them individually... rather than simply appending a dictionary, which is much cleaner!

[@paloma6350]

Super useful video, thanks John! New subscriber here

[@JohnWatsonRooney]

Awesome, thank you!

[@aksontv]

Sir I have a question please, sir for title you used find_all and for price just find, please clear this point. thanks

[@sasuwayne]

Thanks alot John! This video made things clear.

[@JohnWatsonRooney]

Glad I could help!

[@xilllllix]

i learned in 19 mins here what i learned in a 9-hour $89.99 udemy course lol!

[@dwiatmokopurbosakti1193]

how to fix this error: GuessedAtParserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("html.parser"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.

[@ammaralzhrani6329]

How to save data that have scraped and organize and transfer to csv file?

[@mellyndaputri6697]

how can I scrab book rating?

[@Offplandevelopments]

Hi mate, here: for x in range(1,50): url = f'books.toscrape.com/catalogue/page-{x}.html' I'm getting: line 7 url = f'books.toscrape.com/catalogue/page- {x}.html' ^ IndentationError: expected an indented block What am I doing wrong? Thanks :)

[@JohnWatsonRooney]

Looks like you missed indenting the lines of code after the “for x in range” part. It needs to be indented to work (4 spaces or tab)