How To Webscrape Indeed Jobs with Selenium July 2023 | Python with Considerations

[@MrFuguDataScience]

Thank you for watching feel free to reach out for content ideas or suggestions. Here are a few videos to give you ideas Join this channel to get access to perks: kzbin.info/door/bni-TDI-Ub8VlGaP8HLTNwjoin ▶ NATURAL LANGUAGE PROCESSING | NLP BASICS | SENTIMENT ANALYSIS WITH VADER | USING PYTHON: kzbin.info/www/bejne/pZLWe6GBrrN5h9U ▶ TWITTER API WITH PYTHON | NLTK BASICS (NLP): kzbin.info/www/bejne/oYTCYYWEmbqdqLM ▶ NLP Using R-Studio: kzbin.info/www/bejne/ionLdp9tgbmEgdE ▶ R-STUDIO: LABEL & ONE HOT ENCODING WITH MULITPLE EXAMPLES: kzbin.info/www/bejne/kIXIeH2mhLilh9E ▶ REGULAR EXPRESSIONS (Regex) for Parsing ADDRESSES using Python: kzbin.info/www/bejne/pIPUaX2Vlr5-ipI ▶ CONVERT NESTED JSON TO DATA FRAME WITH PYTHON CREATE FUNCTION TO STORE NESTED, UN-NESTED DATA: kzbin.info/www/bejne/fIeodIemls2tebM

[@AleemAhmed-g3n]

Wonderfully Explained the whole notebook code. Thanks for making a better understanding who are also looking out to know this stuff!

[@MrFuguDataScience]

Thank you for the feedback.

[@mastershifu5799]

Thanks for posting this, really appreciate you interacting with your community and trying to solve the issue!

[@MrFuguDataScience]

I appreciate the feedback thanks.

[@mus5125]

Rly cool !

[@mkii5333]

Great video. Easy to follow and I feel like you explained everything clearly. Was searching for ways to accomplish this without using a proxy server, so I'm excited to give this a try!

[@MrFuguDataScience]

You're welcome! Hope it works for you. There is a side note, you may need to change the driver for chrome setup depending what version of Selenium you have. If this is the case for an error, check this video I made a week ago. kzbin.info/www/bejne/b6enaYt4m75sr9k

[@MrFuguDataScience]

For some reason I see a reply for your notification but the comment doesn't show here. Maybe you deleted comment. You said there is a no element exception for finding jobs I think. without seeing your code I need to try and trace your error. 1.) Did you import this? from selenium.webdriver.common.by import By 2.) You may need this now but I don't know # Locate elements on page and throw error if they do not exist from selenium.common.exceptions import NoSuchElementException 3.) is the driver a problem? It wouldn't seem so if you get that error. from selenium import webdriver from selenium.webdriver.chrome.service import Service service = Service() # this is the important line of code! options = webdriver.ChromeOptions() options.add_argument("--headless=new") driver = webdriver.Chrome(service=service, options=options) driver = webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()), options=option)

[@MrFuguDataScience]

I tried it and got the error. Now are you seeing chrome 115 error also with it? I changed to Selenium 4.11 and also changed the selectors and same error. I tried to do an explicit wait as well. Now when you run this did you get a cloudflare popup checkbox? I am curious if the website is blocking some content even though the div class or id are visible. I need to investigate but this may take time. If you get a solution please contact me in the mean time

[@GreenEyedYeti]

Thanks for the content

[@MrFuguDataScience]

Let me know if there is any content that you are interested and thanks for watching

[@soundaryanaga2322]

Can you help how to handle TimeoutException problem

[@MrFuguDataScience]

try to add explicit wait when you enter the page. the page needs to load contents first before you start scraping. Try to add something like this after you do the request for the webpage: import time, then add time.sleep(and some integer here) but, you may be getting blocked i am unsure. Try this webpage: blog.apify.com/python-requests-timeout/

[@Hemanthk-f4s]

hi, Mr fugu, the code you shared is not working for the location "Saudi Arabia", when I entered the location as Saudi, it asked Captcha to redirect which I don't know how to do. Do you have any suggestions for this?

[@MrFuguDataScience]

I noticed some of this happening to me as well last week. I am curious is the layout for the webpage when you look at the tags the same? Next, I need to see if there is an option for the captcha. I do know that you can use a click button in Selenium. But I am not sure if the information is hidden for this on purpose to avoid scraping this button or not.

[@dawnstar24]

I've trying to make a automation bot that'll auto apply for jobs on indeed. But the and dom hidden elements are bother some. I've also tried to find a way so that the script loads one of my own browser Instead of the default one. But it still doesn't work Also its very to even find these stuff because you dont know what they are called

[@MrFuguDataScience]

yeah they changed this last year. So frustrating. Here is a way I can suggest kzbin.info/www/bejne/iYexgYOAlJJlipY

[@sukdeb5315]

from India and you are speaking local American English Please try to normal English

[@MrFuguDataScience]

I will put subtitles hope that helps you. I am AMERICAN, and that is how I speak.

[@dawnstar24]

​@@MrFuguDataSciencehonestly its understand. I'm indian too. Btw you sound like this guy kzbin.info/www/bejne/Y5_XeJmErLiZnpIsi=DASkWc06jQAEcDv1 Joke XD

[@dawnstar24]

* understandable

[@matattz]

What? Do you understand how RUDE this comment actually is? Are you out of your mind? This content from MrFugu is beyond great and people like you have the audacity to complain that this man speaks in his own language..... make it make sense. Mr Fugu, if you read this, from the bottom of my heart, thank you for every video you uploaded and please, don't let these people get in your head!