Intro To Web Crawlers & Scraping With Scrapy

[@muhammedozen2699]

Awesome video. I never thought I'd learn this much in 30 mins. Every second of video is full of useful information. Thank you so much

[@rangabharath4253]

Thank you so much brad. I purchased the Django course on udemy. Awesome content. Congratulations. U will soon reach 1M subscribers. Wow.

[@PythonLearningChannel]

I did a little web scraping a while back-- this video is very timely because I was going to get back to it!! I needed a refresher, thank you!!!

[@PythonLearningChannel]

@@Tolrias Good to know, thank you!!

[@swappy3010]

@@Tolrias thanks. I wanted to know this. Also could you link me to python scraping with headless chrome tutorial? A blog is also fine

[@RameenFallschirmjager]

Right now I'm learning Bootstrap from your udemy bootstrap course. Man, it's amazing! it's not just videos or slides, it has a very comprehensive code examples which accompanies whatever Brad says in the video. Brad did a hell of a job in this course! It's wonderful! I highly recommend it. I've not tried other brad's udemy courses, but if they are as half good as his bootstrap course, I'm sold! This man is a god among us! Love you brad, the great instructor and the awesome family man! God bless you and your family.

[@AmDsus2Fmaj7Am]

Let's scrape ... the scraping blog! I had a good laugh. Your courses are amazing and every now and then we get a good laugh. Keep up the excellent work.

[@tomasjsierra]

man after watching this video and executing this video in just one morning I managed to crawl an entire website in seconds. Thank you!!!!

[@asenchekov]

I was just looking what a crawler is some hours ago. Now logging in to see this uploaded an hour ago! Are you reading the minds of your subscribers? :)

[@leonardol8158]

YEAH, ME TOO. What a coincidence!

[@DennisIvy]

A great content creator that knows what people want. Brads a legend :)

[@abj136]

No no, he's projecting his thoughts into your mind.

[@DennisIvy]

abj freakin brad get out of our heads lol.

[@TraversyMedia]

😊 maybe, i do hear that a lot

[@xzl20212]

for code 'page = response.url.split('/')[-1]'. I thought it should be page = response.url.split('/')[-2] and it works for me. But I donot know why it works with 'page = response.url.split('/')[-1]' in the vedio.

[@tayebsaadi]

Thank you for the instructions, I like how the last minutes made things clear for me...

[@maxwellmuhanda7940]

I was following along with a different site I needed to scap all made sense still always grateful Brad

[@ngsuraj]

I have use scrapy for many web crawling and web scraping projects. However, I still found this tutorial very handy.

[@aneriemmanuel7243]

This video couldn't have come at a better time... Thanks a bunch Brad... God bless

[@justinmean7370]

It's a little out of my league since I am only a beginner coder but it was utterly fascinating! Thank you very much!

[@ericbeard7007]

Your videos always great. A lot of other coding vids built on python talk about simple math for 8 hours and I learn nothing.

[@dev_apostle]

great lesson. After doing some webscraping with selenium, this finally made a lot of sense because I was lost a month ago

[@tyrrelldavis9919]

Bro ur the best, I hate my life but these vids help make it better. I do IT and dev because I like it and because I don't have anything/anyone else for me. Thenk u for helping me learn, Been into python and C# lately, as I re visit JS, it only strengthens my skills, after thinking in New paradigms

[@bassirpechaz]

thanks for your comprehensive description. i think this is good as start point.

[@johnfaulkner5946]

great tutorial, but Im having trouble following along. filename='posts-%s.html' % page fails to number the pages so i just get post-.html and it overwrites itself for page 2, i assume. also tried filename = 'posts-{}.html'.format(page) with no joy.

[@bentraje]

I have the same issue. Did you managed to solve it?

[@AdamEfrati]

@@bentraje I have the same issue, were you able to solve it? is it related to kite? EDIT: Found the problem, you need to replace this line OLD: "def parse(self, response):" with this one NEW: "def parse(self, response, **kwargs)"

[@bentraje]

@@AdamEfrati ah gotcha. didn't solve it. thanks for the reply!

[@quentincaldway]

Ridiculously awesome video! Def an amazing teaching and great start to web scraping with scrapy. Dope Stuff!

[@robpatty1811]

Great video, both compressive and concise!

[@simonetruglia]

thank you so much. My first time with Scrapy and you've been really clear. Great video. Tranks mate :)

[@ProfessorHoffman]

Great tutorial, the copy XPath from the browser was very handy

[@dzenish.2262]

Like => Add to Watch later => Thanks, Brad. :)

[@salimel8802]

Hello brad ! Please could you tell me when would yould you share the front end course for the devBootcamp backend on udemy?

[@TraversyMedia]

After my next course (20 vanilla Projects) which will be released within 25 days or so. I will start working on it

[@chriscastor8328]

@@TraversyMedia Looking forward to them both. Have a phone screen with Amazon coming up and was really worried about my lack of experience with vanilla stuff. What great timing!

[@RahulT-oy1br]

Thank you very much for this tutorial! It's nice, short and crisp!

[@mdjasim3722]

Hey brad m still waiting for your new vanilla javascript course can you tell us when it will available in udemy???

[@XiagraBalls]

Nice and I know you've said there's lots more you could do with this, but one obvious improvement you could make to this is to collect an array or a set of URLs as you go to ensure you don't crawl the same page more than once - as I think that's what this code might end up doing, as it is right now. Right?

[@Chandasouk]

I love me some scraping but I did it with puppeteer and something else for work. My custom API did get blocked a few months later though...

[@tomershechner]

In 8:04, why not use an f-string instead of the old percent sign way?

[@georgestatefield]

Such an awesome tutorial, sir!

[@rishabhkothari1763]

Can Someone Help me I am getting two exception errors while putting the command Scrapy crawl posts: 1)KeyError :posts 2)Spider not Found in Posts Thank you in Advance! (Any Help Appreciated)

[@Vincent.Esders]

Sorry I have the same problem

[@rishabhkothari1763]

@@Vincent.Esders Hey! If you get any Solution, please notify me here in the comment box. Would Really Appreciate the help!

[@madsfynbobergnielsen3310]

@@rishabhkothari1763 sounds like you are making it search the wrong place. are you sure you are set rightfully up with your virtual environment? try going to 'debug configuration' and change the 'source path', e.g. make the last path equal to PostsSpider.py. then it should be able to find the spider. Hope it helps :)

[@whasuklee]

Love your series! Thank you always!

[@_____-ze5ow]

I dont search for this but i am kind of like to watch this so thank you

[@slickgordash]

I'm getting an Unknown command: crawl error which is at 8:49 into the video. I can;t seem to find the error. Any help here?

[@AlessandroBottoni]

Excellent tutorial, as usual. Kudos!

[@ahmadhaidar719]

very useful video super educative and clear

[@josemadarieta3]

so weird that i just started looking at scrapy this morning and boom... this vid drops. question - i cant seem to get vs studio to launch the debugger for a scrapy file. any secrets? thx

[@andig97]

have you tried turning it off and on again?

[@dostontoshpulatov4043]

Great Video simple explanation Thank you

[@bordieit2874]

Very good, please keep doing this tutorial series :)

[@goldenmamba4839]

What about pages that are secured with middleware can you scrap them aswell?

[@thecardtrickstudent3870]

I'm confused regarding why sometimes we specify the attribute name (div) and sometimes we don't while selecting by class. For example: 13:54: No 'div' keyword 18:16: There is 'div' keyword

[@nowieszco868]

The o ly difference is that when we don't select with div, all elements with class will be selected, when we write with div, we gonna select the elements with class, that are divs only. With sayi g with div, you're more specified.

[@thecardtrickstudent3870]

@@nowieszco868 oh oh I see. Thank you for explaining :)

[@emberprime9696]

This is a great Tutorial for crawling data

[@diegocobian8982]

thank you one question why is necessary to create a virtual env?

[@furqanamjad90]

I see Brad's Video I click it even though I don't know what's going on :P . Like it anyways.

[@TraversyMedia]

MeGaZ haha, thanks I appreciate that ❤️

[@hayathbasha4519]

Hi, Please advice me on how to improve / speed up the scrapy process

[@sdwaltersumajit2138]

Thank you for sharing the knowledge.

[@paulshop5580]

I cant seem to make this code work on Python IDLE. upto 22:24 and it gives me the output on scrapy shell but cant make it work in Python IDLE 3.8.2 please help.

[@michelphilippenko2581]

Very clear explanations :-) Thanks a lot !

[@robihamdani5385]

how are you know my thought. i looking for web scrapper and you make a tutorial with this ? are you an alien brad

[@residentjoker]

I may be mistaken but I believe there is already a default method named "parse" that is overwritten here. Nothing wrong with overwriting it but it could cause unexpected behavior for someone that doesn't know.

[@magicmystery4211]

I haven't found any better videos for data structure & algorithm. If you know something please make a vid about it.

[@azamatshaimyerdyen6037]

Love your tutorial man. Thank you. With scrapy can we scrape millions of data with sequenced/scheduled interval to not get blacklisted and keep updating out file?

[@pwchan7443]

How about if I have multiple keywords, for instance, “123”, “apple”, orange” or even with date time, can I use these before crawling it?

[@codewithnacho]

This is sooooo cool! Thanks a lot Brad!

[@alexsandroaugusto5722]

Your video is best, thank you help me a lot!

[@josueanyosagalvez5371]

4:38 When I type 'import scrapy' I get the message 'unresolved import 'scrapy'Python(unresolved-import)' I am using vscode

[@josueanyosagalvez5371]

This solved the issue: www.reddit.com/r/learnpython/comments/a97p09/unresolved_import_warning_vscode/

[@mohsin-ashraf]

Can we also take the user input for the url to scrape in scrapy?

[@Mladen27]

Can you please explain why you used yield on lines 13 and 21 for final version of code? Does this mean parse is generator function in this case? How does this work under the hood?

[@subsoho]

Nice video man ! which extension do you use to see scrapy help in vscode ?

[@fgoerlich2000]

it's called "Kite"

[@alphabeta448]

Hi Brad, thanks for the video. Is Scrapy also able to handle SPAs and specifically with content that is dynamically generated with Javascript?

[@andig97]

Hey man, pls do a course on setting up a bespoke MVC system from scratch with express server , node etc.. going over the MVC fundamentals etc.

[@cooller8888]

thx for this one, helped me a lot

[@trinimafia001]

Is it possible to code this normally like in pycharm or sublime without using a virtual environment?

[@aarongonzales3765]

I think PyCharm automatically creates a venv for your projects..

[@trinimafia001]

@@aarongonzales3765 whenever i try to code this in pycharm i run into issues

[@aarongonzales3765]

@@trinimafia001 Something like package not found? If so, that is easy to fix.

[@waseembarcha6816]

Is there any upcoming course for Vue with TypeScript?

[@VladSuperKat]

kzbin.info/www/bejne/ini6Xq1nl66Kr8k

[@VladSuperKat]

kzbin.info/www/bejne/jaiYaJ-LiJ6XfJY

[@ethanyoung8971]

I'm stuck at 24:50. I run the program and no data returns, and no errors, either.

[@chandlerbing8164]

you're really doing good job ... keep it up buddy... joey says

[@RodneYSSantamarina]

This is great content @Brad, would it be possible for you to explain some basic topics of SEO, I feel as engineers we often lack those skills, right now I am going through the pain. Again very grateful for every piece of content you put out there.

[@zeneto2157]

I have 2 dozens of sites with jobs in europe. I would like to crawl and scrapp several data sets from it. Is there a way to do this in a generic matter to get it all at once ?

[@sujalkhatiwada3572]

Wow! It would be great if you make JIRA course and Agile development, Love all your courses here in udemy, keep going sir.

[@nathanlewis42]

sujal khatiwada you don’t need a course in Jira. If you don’t know it you are in some ways lucky.

[@sujalkhatiwada3572]

@@nathanlewis42 but why, JIRA is used in industry

[@abdullahalkurdi6845]

Scrapping with js, just in the perfect time for me

[@abdullahalkurdi6845]

Thank you brad

[@Booyamakashi]

You clearly watched the video if you think its in js.

[@abdullahalkurdi6845]

Booyamakashi I hadn’t watched the video by the time I commented, I honestly would’ve loved it more if it was a JavaScript, I still like the video as long as brad made it.

[@utopictown]

scrapy is python lib lol

[@abdullahalkurdi6845]

neesyler you’re right, scrapy and beautiful soup are python, puppter is JS

[@themovielookout]

Glad to be here

[@niteshsethi4091]

Can you make a scrapping tutorial in js? There, maybe, so many persons who are looking for web scrapping tutorials in javascript.

[@jamshin6646]

Hi, Is it a vscode extension to see a document at 4:57 How can I use that of 'docs'?

[@jamshin6646]

I found the answer myself;marketplace.visualstudio.com/items?itemName=kiteco.kite

[@dev_apostle]

wish you did a whole series on this

[@Kngdmio]

This is great. Any plans for a Python video that calls an external API and fills models?

[@Expert_Muffin]

Quick question, using xpath insead of css when generating with yeild creates in json file differently, I mean it puts all the titles first, then the dates and so on. It's a there a different sintax that I need to use?

[@Expert_Muffin]

For me yield doesn't do the same when is generating, is just putting all the text under a single tag for each section.

[@L4zzA]

lets say you need to distribute this program to some people and they don't know python. How do you package this project up into an executable that can be run by double clicking or via command line passing arguments?

[@robinc.6791]

Hello! I want to do some web scraping but to find info on a certain thing. So normally, I would use a search engine to find the urls then from there, find the data I need. How would I automate the process of obtaining the URLS? The websites are pretty much the same ( I only really end up using 4 or 5 five websites with the data being a specific spot on the site). I would really appreciate any suggestions! Web scraping is such a good tool, but I need to automate the URL gathering process to accompany the Web scraping

[@srijitdas2207]

cant we create a spider using genspider? or we need to do it manually. I want to do scrape using scrapy in jupyter lab. How can i do that?

[@dean6046]

Awesome! Thank you Brad!

[@umerimran3833]

Sir I have watched the while series but I got one question how to bypass 423 status code as the user agent and proxy pool isn't working

[@jayvr.]

What is the purpose of Making virtual environment ?? Please explain.

[@jg9193]

Some python packages conflict with other python packages, (or their dependencies may conflict), or you may have older projects that depend on older versions of a package, and maybe some even require you use an older version of Python. Virtual environments let you import the packages you need for a project, and use the versions you need and want (for both packages and Python), without having to worry about messing things up for other projects. It's generally a good idea.

[@dgloria]

So it ends the text as it bumps into an apostrophe in regex? Congrats to the almost 1m subscribers!

[@mahinkhankishizade804]

You are literally the best

[@Scuurpro]

My VS doesn't show any of the scrapy folders. But when I cd into my project name and do Tree it shows the folders.

[@sivasubramanianramanathan6945]

Hello Brad, When 20 vanilla Projects course will release.. Waiting for that.

[@TraversyMedia]

To be safe, I will say within a month. Most likely sooner though

[@gradientO]

Can you do a video about *unit testing* ? Please

[@mingzhu8093]

Does it support SPA web app such as Angular?

[@dietermitplatten]

8:41 I don't understand how that works. He declared a starts_url array and then doesn't use it?

[@imenkhiari261]

it is used to tell the command "scrapy crawl posts" where to get the data from. It's like the variable name that you don't used in the code but you use it in the command line.

[@taimoor722]

do u have its course ?? or playlist where are other scrapy videos

[@darkphoenix4273]

In vs code how do you execute the python code in the terminal. Like when he starts the for loop?

[@i_mnikhil]

What if a server block your crawler, how can I overcome with this issue

[@MikeNugget]

Next video: How to overcome captcha with Scrapy :)

[@davyroger3773]

What is the setup on your developer tools on chrome?

[@harissaleem8688]

Can you please make a video on this concept recently i learned java, But in javascript it is difficult for me to understand that suppose we have let a = "Brad".length; then console .log(a) result will be 4 ; so, how we are accessing the dot length thing or maybe like let a = "Brad".toUpperCase(); how this function we are calling because " Brad " it is not a reference variable because in java we use the reference variable them put dot to call any thing inside the object of that class in javascript where is this coming from ???? Please make a video that will be all for me ane it will others also to build concept .

[@U32-w7f]

Also tried a scraping with a node app. I don't know why but the performance was really different from this Scrapy.

[@ryangoodwin9115]

when i move my mouse over Spider in the class it doesnt show the docs link for me to click, any reason? im using windows

[@MrWkirsten]

Hi Ryan, this is because you need to install the Kite extension to your IDE

[@hibald8351]

Thanks for your explanation......but how can do that (crawl ) for my website which built with WP and js ? if some one help me in that

[@dazzlinghwa]

What if the Website is heavy on JS? and how to manage the robot.txt that explicitly disallows Scrapy? :/

[@3ckortreat]

i am getting error 'str' object has no attribute 'css' 19:00