Python Scrapy Tutorial - 9 - Extracting data w/ CSS Selectors

In this video we will scrape quotes from a website and select elements that need to be scraped using CSS Selectors. We will also learn about the tool called as Selector Gadget that is going to make your life so much easier!
Summary
1) Using CSS selectors
Using scrapy shell
response.css('title')
response.css('title').extract()
response.css('title::text').extract()
response.css('title::text')[0].extract()
response.css('title::text').extract_first()
2) Selector gadget on quotes website
3) Selector gadget on amazon
Next video - Extracting data using XPATH
• Python Scrapy Tutorial...
Full playlist - • Python Web Scraping & ...
Subscribe - / @buildwithpython
Website - www.buildwithpython.com
Instagram - / buildwithpython
#python

[@Fmorrell100]

Really great tutorial. He goes through it step by step in order, so you have a clear understanding. That helps a lot

[@onksgk]

You just made my day! From last 2-3 days I am trying to learn web scraping but there are complicated videos on other channels .Today I watched your first 3 videos and then I got it you are going to kill it🔥🔥 and now suggesting that tool ...it became 💎💎. Thank you. You have one more subscriber.

[@Nirmal_rai]

bro really you are so underrated. bro you are teaching so well that i ,a mechanical engineering is doing this like its nothing. keep the hard work on . i love your videos and your teching style

[@devang1956]

You just gave me a breather with the Chrome Extension. Amazing video series! Keep up the good work. You earned a subscribe :)

[@MarsLanding91]

Wow. I was going to wait until the last video to comment but I had to do it now. THANK YOU for these videos! They are SUPER helpful.

[@NirmalSilwal]

your explanations are amazing, very engaging and interesting stuffs

[@debruppaul8239]

Bhiya I don't know how to thank you great job and thanks a lot,you just made selecting piece of cake,thanks again

[@asifmohammed1270]

I wish I found this video much earlier. Just saved a lot of time and effort.

[@echezonaazubike8054]

i have subscribed you nailed it bro i am Nigerian and we loved Indians

[@jerinfrancis4509]

oh my god ! how am I gonna pay you back. YOU JUST MADE MY DAY. Speechless. The chrome extension is damn good bro. Thank you so much for this particular video !!!!

[@buildwithpython]

Glad I could help.

[@zeeshanshani8896]

Hey, great tutorial bhai ! What i get from it is that by using shell command on the terminal we can dynamically scrape data like we do with python request and beautiful soup. Thanks for uploading them.

[@PMDJBMS]

I'm really finding your work helpful for a research project I'm on in the UK. A big thank you for your excellent videos

[@buildwithpython]

Glad I could help.

[@LV7agent]

a really good hands-on tutorial, 10x alot

[@5caioc]

Incredible series!!!! Thanks a lot!! The extension you recommended is extremely helpful

[@danielhoyos6788]

Why am I getting an empty list when scraping Amazon?

[@truverol8205]

wow your tutorial is so great! good job

[@rhn122]

Once again, great tutorial! Clear and straightforward!

[@sgerodes]

Your tutorial is pure magic. Thank you very much!

[@linusjohansson3164]

How do you get the last command in pycharm? Up does not work here for me. I have to write response...etc all over again which is annoying.

[@simasj1]

Nice! The video is so clear, I think you should consider a lecturer carrier! You have a gift to explain complicated things very simply.

[@simasj1]

P.S. www.buildwithpython.com does not work - it says "The account for this site no longer active. This content is not currently available."

[@buildwithpython]

Yeah it's not up. I didn't know people were even checking it out!

[@RobertRoman]

This video is Gold! I'm excited to learn web scraping now :D

[@buildwithpython]

Glad I could help.

[@babuji010]

Nicely explained 👍. Thanks. Have a question. It looks like the "response" object under "Available Scrapy objects:" is responsible for response.css. is that right? There is no "response" object in the list for the web link I try to work on. Any suggestions? Ideas? Please.

[@nathanheath3756]

Subscribed! Very helpful information ! definitely keep these videos coming!

[@whayAl]

many thanks for all your teachings

[@cstech2364]

Thank You So Much Sir 👍👍

[@VigneshSahoo]

Selector gadget is awesome. Thanks mate.

[@web-dev-zargo]

OMG! IT IS WONDERFUL!

[@umerimran3833]

Brother you're outstanding

[@morganv3732]

Pure Gold. Thank you!

[@sgerodes]

The selector tool is magic

[@Abdullahkbc]

this extension is perfect. thank u so much.

[@da_ta]

very exceptional excellent work thanks for doing this

[@167tejaswini]

I have tried below ..but still not displaying here anything >>> response.css(".a-color-base.a-text-normal").extract() [] >>> response.css(".a-text-normal::text").extract() [] >>> response.css("a-text-normal").extract() []

[@buildwithpython]

Did you try it on the example website I gave?

[@ReasonToKeepGoing]

Solved the issue in two different ways, response.css(".a-color-base.a-text-normal::text").getall() and response.css(".a-color-base.a-text-normal::text").extract()

[@mihirthakur917]

Facing the same. Worked with quotes to scrape but not with amazon. I tried it with Flipkart and it worked

[@buildwithpython]

@@mihirthakur917 hey I have a separate video for Amazon in the same playlist

[@CSSuccessGamer]

amazon must have found this video and decided to block scrapers...

[@md.mahabuburrahman8544]

Wow great tutorial

[@anupamasingh1239]

Hey, I'm getting error 404 while scraping the amazon website which you gave. I tied finding solution but was not able to fix it. Can you please help me out on this?

[@alisiraydemir]

Just want to say Thank you!

[@jamezz2181]

how do you remove blank space like and spaces when it just has a bunch of them from it

[@imaduddinsheikh3546]

Thank you so much for your Scrapy tutorials! However at 10:36, I tried running scrapy shell command on the Amazon website, and the response came back with a 503 code. How do I fix this? And, what's the issue behind it? I am running Windows 10.

[@imaduddinsheikh3546]

Nevermind, I fixed the issue. I reduced the concurrent requests in the settings.py file to 1(I also added a user agent for Chrome browser with the latest version in the same file)

[@ItsBen27]

@@imaduddinsheikh3546 THANK YOU!!! Your comment saved me from a lifetime of searching for the fix!

[@jiyarahman2673]

Hi, I am following the code as you guide, but I am getting a Empty list for response.css even for previous video I got empty value can you explain me why?

[@andrejohnv]

Did you get it now? I'm getting empty list lol.

[@naveenkumardongre]

same I too getting empty list

[@web_devs]

Cant scrape amazon... returns empty list >>> response.css(".acs-product-block__product-title .a-truncate-cut::text").extract() [] any help..?

[@yudhvirsingh9909]

same here

[@CSSuccessGamer]

i have same error,

[@CSSuccessGamer]

did u find an alternate way?

[@amitjamwal1985]

Very helpful videos. thanks a lot :)

[@amitkumar-yu6yz]

great video man.ver very thanku

[@bhavyajain2034]

will appreciate your help

[@harshadmanglani1309]

The series is great, although there's something wrong with the quotestoscrape website, it gives me a twisted internet error, works for every other website though. Thanks.

[@DarkScizor]

Hi there, I had a question. I wanted to parse the alt text off of an img. How would I go about this? I appreciate any help you can give!

[@pranaydas3320]

Use attrib("alt")

[@redfeather22sa]

Your very Good !!!!

[@ubaidmanzoorwani7491]

I am trying to scrap data from youtube but it is returning an empty list every time . please tell me what to do.

[@gurjeetkaur3626]

Helped a great....but after half of video .. view not clear

[@souilahmaher7188]

You're great instructor!

[@buildwithpython]

thanks

[@emm5138]

Great video! Thanks a lot!

[@hasnainahmed6706]

It is giving empty list on my pc at 11:12 please help me out.

[@Imperialcodex1]

Thanks man. 💪

[@giotsas]

Great video, just a small correction. In 09:00 you mention [1] is the first index of the list of authors. It's the second index.

[@brendenandrews6965]

How to access the previous commands in the shell.. usually when I'm in the terminal I am able to access the previous command using the up button but in the shell I am not able to do the same as shown in the video.. can anyone help me with this..

[@ThallaSampathKumar]

yes it is fine but it is not working for all websites returning me an empty list

[@bhavyajain2034]

SIR, while running scrapy shell command, terminal is raising a ValueError : invalid hostname: 'http

[@TranLamYoutube]

Amazon's source could change, I can't crawl data, elements are render from script, not from sample Html

[@as-px2mv]

thanks a lot!

[@Jgs8115]

Saved my neck thanks man

[@jss2754]

i have some question of this video. As you know ,scrapy is have a two ways for xpath that is css and xpath. i wonder why are u using css on your video .

[@buildwithpython]

In the next video I use xpath. I just like CSS selectors

[@jss2754]

@@buildwithpython thank u for comment!!

[@Coney_island23]

great!

[@shaikhanuman8012]

bro i am getting 503 error code how could i fix it please tell me brother

[@teo-medesi]

Go out back, find the biggest stick you can find, keep hitting your pc until it works. I hope this helped!

[@shaikhanuman8012]

@@teo-medesi I tried brother due to that I bought new pc (😄 I fixed the error)

[@shaikhanuman8012]

@@teo-medesithanks brother for providing valuable knowledge

[@teo-medesi]

@@shaikhanuman8012 Any time!

[@shaikhanuman8012]

@@teo-medesi tq sir

[@aakanshasingh9680]

scrapy crawl quotes. -> not returning anything. Nothing is displayed on the terminal

[@aakanshasingh9680]

Basically, the function parse is not getting executed. Anything else written outside parse but inside the class is getting executed.

[@jabiraziz1219]

Just Wowww

[@CSSuccessGamer]

10:34 im getting 503 error from terminal for amazon

[@CSSuccessGamer]

it says forbidden by robots, what do i do?

[@aakashkhadka3835]

same here

[@hau_hau_happu_singh]

I am continuously getting a null array, after using selector gadget.

[@danielamorariu6722]

you're probably getting a 503 error, which means service is unavailable. I solved this by specifying a user agent in settings.py and disabling the cookies, also in settings.py . User agent can be Mozzila 5.0 etc etc ( check explanations here: www.scrapehero.com/how-to-fake-and-rotate-user-agents-using-python-3/)

[@Pandazaar]

hey I was trying to follow along this video and I think you can no longer use response.css, because it was removed I guess, the error I get is: AttributeError: 'function' object has no attribute 'css'

[@buildwithpython]

Nope it's not removed. Don't think your scrap is installed properly.

[@Pandazaar]

@@buildwithpython oh, i did cd quotetutorial before opening shell,my bad

[@babuji010]

@@Pandazaar Hey, I am getting same error. Can you explain what went wrong? And solution pls. Thanks

[@Pandazaar]

@@babuji010 just type "cd .." and then open the shell

[@anrm6]

You are god

[@doubled9645]

thx bro

[@buildwithpython]

No problem

[@vigneshsivasubramanian9193]

is it only me or the entire headphones shakes and trembles when he presses his keys think they are feared of him please dont overuse it and give some rest for both u and your keyboard

[@SquaredbyX]

Should be called a css De-selector

[@Bihari_Chaman]

It is not a list its Array

[@Yuri-xx2gi]

It's a list, this is not C, in python they're called like that

[@_thoneeer3220]

wow

[@juann9880]

What if response =403, I can't extract anything?