Django + Celery + Selenium to Scrape Anything with Python

[@CodingEntrepreneurs]

Thanks for watching! Here's the code: github.com/codingforentrepreneurs/Web-Scraping-with-Django-Celery Grab the free Udemy course: www.udemy.com/course/smarter-web-scraping-with-python-ai/ 00:00:00 - Welcome 00:02:23 - Tools 00:06:07 - Rapid Fire Django + Celery Integration 00:19:06 - Setup Project 00:21:23 - First Look at Selenium with Jupyter 00:25:08 - Proxy Scraping with Bright Data + Selenium 00:32:08 - Helper Functions for Bright Data Module 00:35:55 - Django + Jupyter Integration 00:39:00 - Parsing Data with BeautifulSoup4 00:52:02 - Amazon Captcha + Prepare Parser Helper Functions 00:58:30 - Saved Scraped Data with Django Models 01:10:51 - Celery Task to Offload Scrape-to-Save 01:22:29 - Schedule Scraping Events with Celery in the Django Admin 01:33:16 - Thank you and next steps

[@emikenester9174]

Really enjoying the series! Does HasData offer any API that simplifies scraping complex pages like Amazon?

[@amitavasengupta5580]

Awesome!!!!!!!!! Was waiting for a celery video from the Master himself!!!! Had a tough time with celery now it's a time for a celebration

[@CodingEntrepreneurs]

You got this! 👍

[@DiegoSilva-dv9uf]

Valeu!

[@CodingEntrepreneurs]

Thank you

[@abolfazldaneshvar1441]

What a good tutorial! Keep going man

[@CodingEntrepreneurs]

Thank you!

[@banka990]

You are my only hope to make a tutorial on integration of Django with Next.js and demonstrate different types of rendering - SSR, CSR and SSG. Please

[@diogos7233]

I hope he sees your comment. I wish to learn this too.

[@oludelehalleluyah6723]

Your prayers were answered

[@yssplays2620]

The best youtuber i ever seen

[@kaanatsever2333]

If you say we are scrapin, then we are scrapin

[@CodingEntrepreneurs]

😂😂

[@dgiri2333]

1.Sir do text to (sql or ORMS) ask our questions and get result that way Course can you plan sir like Langchain sql to openai /2.geodjango related ask questions and get answers from our database

[@gamingLiberationFrontOfficial]

sir can you please one on Nextjs Django and Docker

[@edwardmike7523]

I learnt alot, thanks so much. But why did you use Django, if you need its model you can use Pydantic, SQL-alchemy, PostgreSQL

[@MuhammadRizwanManzoor-f1n]

The reason is simple. They don't have Django Admin feature. Here the idea was to control everything through an interface. Imagine you are providing this all configured scraping functionality to a Non-Programmer client who can easily scrap and control events from Django Admin UI.

[@MehrdadMahmoodpoor]

great, can you tell me the name of font that you are using in vscode?

[@aashayamballi]

thank you!

[@adirnoyman2231]

I’m trying to ping the redis-cli but getting this error: “redis-cli zsh command not found”. The venv is active and Docker is running the Redis image. What can be the problem?

[@CodingEntrepreneurs]

You need to install the redis-cli on your machine. It’s different from what you mentioned. The Python redis client should connect to your docker instance of redis without the redis-cli

[@alexdin1565]

at the right time because I got blocked a lot hhhhhh please can you make a video on how we can deploy Django with docker to server. I'm one of your udemy Django core if you can update the content

[@CodingEntrepreneurs]

Thanks for the suggestions. I do need to update Django Core. Thanks for the feedback

[@alexdin1565]

how much Bright Data costs for 1000 scraped URL?

[@not_amanullah]

thanks ❤

[@sheikhhassan7899]

Bright site link isn't working for me 🤔 anyone who is facing this issue ?

[@CodingEntrepreneurs]

How do you mean it’s not working?

[@philtoa334]

Nice.

[@ddkinput]

Celery and it's like still useless for heavy tasks and we still split out code in half or more to be Celery compatible. wasting time..

[@Deus-lo-Vuilt]

wow

[@darylhunt9070]

nice advertisement