This will change Web Scraping forever.

[@danielcave9606]

Thanks John, great to see you enjoyed the tool. People should 100% still be learning scrapy and web scraping mechanics so to understand whats going on under the hood and to be-able to customise any AI/ML solution. Understand the rules so you know how and when you can break them! Back when I was a contractor, charging by the project ,I quickly learned that any technology that could help me work a lot faster and deliver more projects - even at a slightly lower margin - was always worth it.

[@mbalam11]

@john thanks for the information. It looks similar to octoparse. Can they export the data in platform specific format? for example Shopify accept data in certain formats where the variants (color/ sizes/ style) go with different line while maintaining the product id same on multiple lines. I am just trying to discover that are the data exported are ready for upload on platform based or formatting need to be done.

[@MrMantas211]

Did the aii transfer to other languages?

[@FolioCharm]

I'm interested in automating media (video, image...) upload/download, are there any python libraries that tackle such tasks or do you have any tutorial on this subject?? Thanks.

[@JohnWatsonRooney]

There are but it’s not something I’ve ever done I’m afraid! I know it’s possible though

[@silkogelman]

Playwright has FileChooser for file uploads

[@silkogelman]

Playwright has FileChooser for file uploads. You might want to look into that.

[@MT-zm9dl]

Whats the solution for sites that require an account and bans the account after certain requests even if I scrape manually by clicking each link.

[@danielcave9606]

It depends on the terms you agree to when you register for the account. Zyte does not allow scraping behind logins where you explicitly agree to terms of service that prohibit scraping and/or your intended use of the data.

[@graczew]

I start scraping project for learning purpose, mainly python. You can use python for feee and this is something what i love in coding. Now we enter the interesting times when ai will do all for us. Unfortunately not for free. If you need data this is brilliant if you want to have fun you can do spend time learn and try do that on your own. I wish to get something what i can hoste on my homelab or some vm and use. But as much as u learn you understand that how complex is your project to get results. You will need other services anyway so why not use all in one witch get job done?

[@tvcodemate]

You are great 👍🏽 I have question, i am going to make videos about web scraping. My question is that does it ok with KZbin rules? I mean i want to make tutorials about scraping imdb or amazon. KZbin doesn’t have problem with this? Thanks

[@JohnWatsonRooney]

Yeah go for it, I’ve got lots of web scraping videos and others do too

[@tvcodemate]

@@JohnWatsonRooney perfect, thank 👍🏽

[@einekleineente1]

So as an absolute beginner in web scraping, should I still try to do everyting myself first? Will say, learn As much as possible and only then go down the AI route? Or go straight to the AI solution?

[@JohnWatsonRooney]

Depends on what you want out of it. If you want to learn how it works do it all manually, then once you’ve got to a good point you’ll understand where services like this fit in and how and when you would want to use them

[@einekleineente1]

@@JohnWatsonRooney excellent! Thank you!😃

[@amirsohail2368]

Hey john, Can you please make a video to scrape desktop application data.

[@kingofmontechristo]

Translation: "Game over."

[@JohnWatsonRooney]

Adapt :)

[@omkarpatel1634]

I scrape a m a z o n, 4 times, every day, so $0.20 x 40(40K Requests) x 4 x 31 = $992/month. I am doing it for a monthly cost of $25. So, it might be good for someone who wants data only once.

[@ulrikpedersen6112]

Sounds good. Do you use specific software or? Thanks.

[@omkarpatel1634]

@@ulrikpedersen6112 Python and Proxy

[@bakasenpaidesu]

Yabeeeee

[@TheJFMR]

I like this dynamic, to promote different tools created by companies

[@JohnWatsonRooney]

Thanks I appreciate it

[@alexdin1565]

zyte == very expensive

[@JohnWatsonRooney]

For most sites I didn’t find it much more if at all than good proxies