A Comparison of Proxies - Rotating IP with Python Scrapy

Learn how to use free and paid Proxies in Python Scrapy.
Links in the video:
Source Code: gist.github.co...
Free Proxy: github.com/hya...
Not so Expensive Proxie by ScraperAPI: dashboard.scra... - use promo code for a special discount: coderecode10
Premium Proxies By Crawlera (now Zyte): www.zyte.com/?...
Check my latest courses: coderecode.com/
Disclaimer: These are affiliate links.
~-~~-~~~-~~-~
Please watch: "Making Scrapy Playwright fast and reliable"
• How to make Scrapy Pla...
~-~~-~~~-~~-~

[@codeRECODE]

If you like the vide, *Like* and *subscribe* !

[@ASOUE]

Very helpful. Many other tutorials were not helpful with integrating proxies into scrapy, but this was exceptionally clear. ScraperAPI is very easy to use too.

[@codeRECODE]

Glad it helped!

[@tanercoder1915]

long waited tutorial! thank you!

[@codeRECODE]

Hope this helps :-)

[@thewilltejeda]

Thanks for the heads up about scraperapi

[@codeRECODE]

Happy to help

[@noushadkhan1647]

Thank you, sir. Please make a video on crawlera too asap.

[@carloscampos7709]

Great video! Thanks for the insights

[@NguyenHung-do7gc]

👍👍👍 Great video! Thank you

[@suchanachakrabarti6783]

Thankyou for the tutorial

[@codeRECODE]

You’re welcome 😊

[@ataimebenson]

This was very useful Thanks. You mentioned you will do the Crawlera video also

[@codeRECODE]

Yes, it's been in my list for some time now. Will get it to it.

[@ataimebenson]

@@codeRECODE Nice, would be looking forward to it. Really appreciate

[@Karymemontolis]

The rotation feature provided by Zeus Proxy allows me to change IP addresses seamlessly, maintaining a dynamic online presence that reduces the likelihood of being flagged by platforms like Twitter.

[@codeRECODE]

Yes, most proxy providers have this feature

[@Emily_Entropy]

I've gone through several of your videos and picked up some great tips, thanks! Since you seem to be heavily focused on scraping, I wonder if you've looked into ML-based scraping. Instead of manually coding heuristics, let the machine learn how to find the relevant data!

[@codeRECODE]

Thanks Tim. ML based videos are in the plans. I really appreciate you brining it up though.

[@riv4wid77]

Hello Upendra, thank you very much for sharing, I take this opportunity to ask you if you know of any alternative as good as this one, but for selenium?

[@lordmelbury7174]

Excellent video my good man!

[@codeRECODE]

Thank you, my good man! :-)

[@lordmelbury7174]

@@codeRECODE I get this, it only changes on the 5th attempt: { "origin": "80.239.199.101" } { "origin": "80.239.199.101" } { "origin": "80.239.199.101" } { "origin": "80.239.199.101" } { "origin": "80.239.199.101, 207.74.82.103" } I've checked and triple checked my code matches yours. Any suggestions?

[@codeRECODE]

Strange! Contact support team

[@riv4wid77]

Upendra can you recommend any companies that do really cheap proxy rotation services?

[@abdd9878]

Can i use this for cpa, i am zero knowledge with coding, how to use it, i do not understand please

[@codeRECODE]

This video is meant for Python developers, so not having any knowledge will be a problem here.

[@atultanna]

Can we use these for scrapebox

[@hayathbasha4519]

I'm always getting this error 2021-10-02 06:44:24 [scrapy_proxy_pool.middlewares] WARNING: No proxies available. 2021-10-02 06:44:24 [scrapy_proxy_pool.middlewares] INFO: Try to download with host Please help

[@codeRECODE]

Free proxies are really unreliable. Check scrapy-rotating-proxies package. This one can be used with a custom proxy list.

[@786komail]

hi sir i need rotating residencial proxy unlimited help me sir

[@rostranj2504]

when developing a project should I set up proxying before hitting the target's website?

[@codeRECODE]

Write your code without proxy first. Once you are sure that everything is working, then add proxies.

[@tanveermehmood333]

when i implemented the same code and ran it. It is displaying different websites for proxies (IP obtaining purpose) but it is printing same IP every time. Could you please guide me what is the reason.

[@codeRECODE]

Are you using the same code? Which proxy did you try? Or some variation?

[@raisulislam4161]

Hello Sir, scrapy-proxy-pool is no longer working. What can we do now?

[@codeRECODE]

and that's precisely the reason I recommend getting one of the cheaper proxy providers. I will see if I can find something else, but sooner or later, they will all stop working.

[@JnWayn]

Is it fair to conclude that some some web servers are just so adamant about not getting scraped, and good at implementing it, that no amount of coding is ever gonna get past their security?

[@codeRECODE]

That is true

[@adityapandit7344]

I tried the same but it's totally failed. Not getting any output.

[@codeRECODE]

Did you try running the code from the link in the description? If yes, what is the exact error you are getting?

[@adityapandit7344]

@@codeRECODE failed because website has a captcha for verification. Nd giving 307 error!

[@codeRECODE]

Will chek. Anyways you can use 2captcha kind of service to solve that

[@OP-qs2kt]

What counts as an API call? Is that every new page that gets scraped?

[@codeRECODE]

Every HTTP Request is one API call, doesn't matter whether it is an GET, POST, or result is HTML, JS, image. Hope it helps :-)

[@Rajeshgee10]

Do you have any idea about Google 2 fact authentication login based website scraping?

[@codeRECODE]

If it is a code on text message, you can use the web client for messages and read the messages.

[@shubhammishra2267]

Hello Sir, how to scrape the website which are having reCaptcha

[@codeRECODE]

2captcha.com

[@ataimebenson]

@@codeRECODE Please can you make a video on how to solve recaptcha's using this ?

[@codeRECODE]

​ @Ataime Benson - solving Recaptcha is a sensitive topic. The presence of any sort of captcha means that the site is discouraging from being scraped. For _educational purposes_ I can tell you that there are many services like 2captcha that can solve Recaptcha for a very small amount of money.

[@ataimebenson]

@@codeRECODE Thanks

[@ashish23555]

Sir, does free membership for proxies helpful

[@codeRECODE]

Yes if you want to experiment and are okay with slow proxies. Any paid proxy will be lightening fast as compared to free ones.

[@lxvi4322]

Hi. How to bypass Zyte?

[@codeRECODE]

Not sure I got the question.

[@riv4wid77]

Hello Upendra, thank you very much for sharing, I take this opportunity to ask you if you know of any alternative as good as this one, but for selenium?