Indeed Jobs Web Scraping Save to CSV

Let's scrape some job postings from indeed.com using Python. I will show you how to work with pagination, extract job titles, salary, company and summaries from the site and save as a csv file for excel.
-------------------------------------
twitter / jhnwr
code editor code.visualstudio.com/
WSL2 (linux on windows) docs.microsoft.com/en-us/wind...
-------------------------------------
Disclaimer: These are affiliate links and as an Amazon Associate I earn from qualifying purchases
-------------------------------------
Sound like me:
microphone amzn.to/36TbaAW
mic arm amzn.to/33NJI5v
audio interface amzn.to/2FlnfU0
-------------------------------------
Video like me:
webcam amzn.to/2SJHopS
camera amzn.to/3iVIJol
lights amzn.to/2GN7INg
-------------------------------------
PC Stuff:
case: amzn.to/3dEz6Jw
psu: amzn.to/3kc7SfB
cpu: amzn.to/2ILxGSh
mobo: amzn.to/3lWmxw4
ram: amzn.to/31muxPc
gfx card amzn.to/2SKYraW
27" monitor amzn.to/2GAH4r9
24" monitor (vertical) amzn.to/3jIFamt
dual monitor arm amzn.to/3lyFS6s
mouse amzn.to/2SH1ssK
keyboard amzn.to/2SKrjQA

[@franklinokech]

Great tutorial John, just a quick fix on the for loop, you forgot to add the i to the extract function: Made the changes to this for i in range(0, 41, 10): print(f'Getting page {i} ') c = extract(i) transform(c)

[@JohnWatsonRooney]

Great thank you!

[@sosentreprises9411]

Hi everyone, I had the following error message : File "/Users/admin/Downloads/Test.py", line 36, in transform (C) File "/Users/admin/Downloads/Test.py", line 22, in transform summary = item.find('div', class_='job-snippet').text.strip().replace(' ','') #trouver sommaire et remplacer new lines with noth' UnboundLocalError: local variable 'item' referenced before assignment ANY HELP ?

[@Why_So_Saad]

This channel has helped me a lot. Everything I know about web scrapping is thanks to John and his to-the-point tutorials.

[@vvvvv432]

That's an excellent video for the following reasons: -- the flow of the tutorial is really smooth, -- the explanation is excellent so you can easily adjust the classes that existed in the time of the video to the current ones -- and the iterations are detailed so every step is easy to understand. Thank you so much for this video! Greetings from Greece! 🇬🇷

[@JohnWatsonRooney]

Awesome, thank you very much!

[@rukon8887]

John amazing tutorial and skills, love the way you slip in sometimes different method on going about. Hope you getting big bucks for your expertise. Keep them video coming.

[@mrremy8]

Dude, thanks so much. You deserve much more views and likes. I didn't understand scraping one bit before this.

[@afrodeveloper3929]

Your style of code is so beautiful and easy to follow.

[@igordc16]

Worked flawlessly, I just had to edit a few things, like the classes and the tags. Nothing wrong with the code, just the indeed website that changed since you posted this video. Thanks!

[@JohnWatsonRooney]

Awesome thank you!

[@dmytrodavydenko7467]

Great tutorial! Nice and easy flow of code! As a beginner programmer, I really enjoyed this video! Thank you a lot!

[@user-rj5hd7hh8p]

Hey. Seriously. Thank you. I just downloaded soft and I can CLEARLY see why your vid was recomnded. You're an aweso intro into

[@hi_nesh]

Honestly, This channel is marvelous. It has helped me a lot. 'a lot' is even an understatement

[@kmgmunges]

keep up the good work those lines of code and the logic is sure fire.

[@davidberrien9711]

Hello, John. I have just started learning Python, and I'm trying to use it automate some daily tasks, and web scraping is my current "class". I really enjoy watching your workflow. I love watching the incremental development of the program as you work your way through. You are very fluent in the language, as well as the various libraries you demonstrate. I am still at the stage where I have to look up the syntax of tuples and dictionaries... (Is it curly brace or brackets? commas or colons?) so I find myself staring in amazement as two or three lines of code blossom into 20, and this wondrously effective program is completed in minutes... I am envious of your skill, and I wanted to let you know I appreciate your taking the time to share your knowledge. I find your content compelling. Sometimes I forget to click the like button before moving on to the next vid, so sorry about that. I just have to go watch it again, just to make sure I leave a like... Your work is very inspiring to me as a noob. I aspire to the same type of fluency as you demonstrate so charmingly. Thanks again.

[@JohnWatsonRooney]

Hi David! Thank you very much for the comment, I really appreciate it. It's always great to hear that the content I make is helping out. learning programming is a skill and will take time, but if you stick with it, things click and then no doubt you'll be watching my videos and commenting saying I could have done it better! (which is also absolutely fine) John

[@eligr8523]

Thank you. You saved my entire semester!

[@OBPagan]

You sir are a true legend. This taught me so much! I really appreciate it!

[@JohnWatsonRooney]

Thanks!

[@dewangbhavsar6025]

Great videos. Very helpful in learning scraping. Nicely done. Thanks!

[@theprimecoder4981]

I really appreciate this video, you thought me a lot. Keep up the good work

[@JulianFerguson]

I am very surprised you only have 1500 views. This is one of the best webscraping tutorials i have come across. Can you do one for rightmove or zoopla?

[@thenoobdev]

Heheh already at 42k today 😁 well deserved

[@vijayaraghavankraman]

Sir I became a great fan of u. Really interesting. A great skill to explain things in a better way to understand. Thanks a lot

[@JohnWatsonRooney]

Thank you!

[@jonathanfriz4410]

Like usual very helpful John. Than you!

[@sayyadsalman9132]

Thanks for the video john! It was really helpful.

[@MyFukinBass]

Damn this was top quality my man, thank you!

[@caiomenudo]

dude, you're awesome. Thank you for this. Nice guitars btw

[@JohnWatsonRooney]

Thanks Fabio!

[@alexeyi451]

Great job, neat explanations! Thanks a lot!

[@sujithchennat]

Good work John, please use the variable i in the extract function to avoid duplicate results

[@Eckister]

your video has helped me a lot, thank you!

[@lebudosh2275]

Hello John, Thank you for the good work. It would be nice to be able to be see how the job descriptions can be added to the data collected from the webpage as well.

[@irfankalam509]

Nice and very informative video. keep going!

[@visualdad9453]

Great tutorial! thank you John

[@benatakaan613]

Amazing content and teaching style! Thank you.

[@JohnWatsonRooney]

Hey thanks very kind of you

[@aliazadi9509]

I just did webscraping on this website and youtube recomanded this video!🤣

[@thecodfather7109]

Thank you 🙏🏼

[@sanketnawale1938]

Thanks! It was really helpful.

[@lokeswarreddyvalluru5918]

This man is from another planet .....

[@jakepartridge6701]

This is brilliant, thank you!

[@loganpaul8699]

Such a great video!

[@nikoprabowo6551]

I think its the best tutorial!!!! big thanks

[@ritiksaxena7515]

really thanks for this wonderful work

[@ibrahimseck8520]

I couldn't thank you enough for this tutorial...I am following a Python course on Udemy for the moment, and I found the section on web scraping incomplete...I followed this tutorial and it's brilliant...The indeed page is quite different including the html code, but the logic stays the same...I will put my code in the comments, it might be of interest especially for people using indeed in french

[@oximine]

Any update on your code bud? I'm trying to scrape indeed right now and the html looks very different than what's in the video

[@cryptomoonmonk]

@@oximine Yea, indeed changed their code. I had a rough time figuring it out. the job title is no longer in the 'a' tag in the new html. At 9:13 in the video you need to use: ... divs = soup.find_all('div', class_='heading4') for item in divs: title = item.find('h2').text print(title) ... Reason being, Indeed now has the title of each job title within an h2 element which is in the class starting with heading4. So the code searches for the class heading4, once it finds it will search for the 'title' item in the h2 element Just look at the html and see where the "title" of the job search is in the new code. One thing is for sure, once you figure this out and understand it, you understand what's going on.

[@michealdmouse]

@@cryptomoonmonk The code works. Thank you for sharing.

[@ashu60071]

Thanks 🙏🏻 you so so much. Actually can’t thank you enough.

[@gabrielalabi4385]

Thank a lot, really helpful. I'll love to see how to automate applying to them 🤔🤔🤔

[@anayajutt335]

Ima download it thanks for sharing!!

[@ramkumarrs1170]

Awesome tutorial!

[@stuarthoughton3517]

Awesome , John!

[@JohnWatsonRooney]

Thanks Stuart!

[@dominicmuturi5369]

great content hopefully more videos to come

[@martpagente7587]

Very thankful to your videos John, we support your channel and your popular now in youtube, I wish you can make video also scraping LinkIn or Zillow website, these are demands from freelance sites

[@JohnWatsonRooney]

Sure I can have a look at the best way to scrape those sites

[@expat2010]

@@JohnWatsonRooney That would be great and don't forget the github link when you do. :)

[@glennmacrae3831]

This is great, thanks!

[@ertanman]

GREAT VIDEO !!! Thank you very much

[@daniel76900]

really, really good content!!

[@yazanrizeq7537]

You are awesome!!! Def Subscribing

[@kammelna]

Thanks John for your valuable efforts In my case I wanna scrape data inside each container where there is a table of info then loop over every link in the page So I need to click the link of first job for example and get data from a table and so on so forth for the rest of the page It would be highly appreciated if you could consider similar case in your next vids. Cheers

[@ALANAMUL]

Thanks for video....realy useful content

[@JulianFerguson]

I know you mentioned using a while loop to run through more pages. Could you give an example of how this might look like?

[@anthonyb5625]

Great tutorial thanks

[@ansarisaami5196]

its so helpful brother

[@datasciencewithshaxriyor7153]

bro with your help i have finished my project

[@JohnWatsonRooney]

Thats great!

[@Dev-zr8si]

This is amazing

[@kamaleshpramanik7645]

Thank you very much Sir ...

[@rob5820]

Cheers! I'd love an updated version of this. It seems They've changed it. I have a project due soon for which I'd like to scrape Indeed as the project is a job search app.

[@JohnWatsonRooney]

Thanks, I did a new version not that long ago the code is on my GitHub (jhnwr)

[@rob5820]

@@JohnWatsonRooney Unreal. Thanks for the quick reply too.

[@AtifShafiinheritance]

really good for lead generation ty

[@lbayout2775]

perfect class

[@Didanihaaaa]

very neat!

[@tenminutetokyo2643]

That's nuts!

[@SamiKhan-fd8gn]

Hello John, great video but unfortunately I keep getting 403 from indeed instead of 200 so not working for me.

[@prashanthchandrasekar1026]

Thank u so much🙏

[@hassanabdelalim]

Hi great , but follow the same steps but i get 403 response not 200 , any help

[@AC-sk1mz]

How would i pull the underlying link embedded in the title for each job posting into a variable?

[@rajuchegoni108]

Hi John, how did u customize the output path, i tried so many experiments but it did not work. can u help me with that?

[@looijiahao2359]

hi John , great tutorial , how would you add the time function in this particular set of code .

[@eligr8523]

Hi. How can I scrape multiple pages? Can I just define another function to scrape another page? Ideally I would like to add all the information to one database using sqlite.

[@syedashamailaayman8242]

In the same website...If we have the contents in an 'a' tag instead of the div tag what do we do? coz the 'id' is different for all the 'a' tag. I want to scrape all the 86 pages that has the content in their 'a' tag. Please Help!

[@alibaba2746]

Can u please teach us how to Automate or Scrape Facebook too. Thank u again bro for ur valuable teachings. GBU

[@alexcrowley243]

It seems though that no matter what I set the range for the pagination in the f string for the url, I can only return 15 results, similar to this video. Do you have any advice for this?

[@JohnWatsonRooney]

Yes i made a mistake in my code - the "c = extract(0)" should be "c = extract(i)" so we get the new page from the i in range() loop!

[@absoluteRandom69]

Hello John, I'm not able to crawl the website because of captcha. Who should I handle it?

[@misfitcodes4069]

Great video, that's a sub from me!

[@joxa6119]

Why i dont found the card in the div? I found it in a=tag which doesnt have the serpJObCard

[@julianangelsotelo4757]

I got a 403 on my status code, does anyone know any potential solutions? Thanks!

[@saifali4107]

Hi John, Thanks for this wonderful video. I am following the steps but struggling with getting Company reviews the same way. Can not seem to find the right div class. Could you please help there.

[@Palvaran]

This is fantastic, thank you for this. I am trying to learn how to code and had a question on the locations field. When I nest "location = item.find('span', class = 'location') between the title and company lines of code it appears to only partially populate the fields with the location data. Additionally, the fields contain extraneous information such as the metadata in it. If I try use the text.strip() it gives an error of AttributeError: 'NoneType' object has no attribute 'text. Any ideas on what to do for the last portion of code? Thanks!

[@Palvaran]

For those wondering, you can add location by using this variable. "location = item.find('div', 'recJobLoc').get('data-rc-loc')"

[@hanman5195]

@John- Can you please prepare script to capture complete job description for specific role like data scientis or technical account manager.

[@technoscopy]

Hello sir if there are no page numbers my url is same for all time for every data how to scrape them I have to get the new data from the Drop down so how to do it.

[@FadeOutLetters]

Amazing video! Do you have a copy of the code you used in this video anywhere?

[@Free.Education786]

How to GRAB job listening email addresses to e-mail CV in BULK at ONCE???. Thanks

[@GudusSeb]

Any idea how I can render/display the response data on a browser using HTML instead of saving it into CSV? Your Aswere is much appreciated. Thanks.

[@ajinkyapehekar8985]

I hope this message finds you well. I wanted to reach out and let you know that I've been trying to interact with your video, but I keep receiving a 403 response instead of the expected 200 response. I have checked my code, and it seems that I am setting the User-Agent header correctly to mimic a browser request. However, despite these efforts, I am still encountering the 403 error. I wanted to ask if there's anything specific I should be aware of or if there are any additional steps I need to take to ensure proper access to your video. I appreciate your time and any guidance you can provide to help me resolve this issue. Thank you for creating such valuable content, and I look forward to your response.

[@harkoz364]

I have an error when I try to issue an HTTP request with the get function of request by putting a second parameter to it but when I remove this second parameter which contains my user-agent it works, is this already happen to someone? 3:40

[@shayanhdry6224]

god of scraping

[@samiulhuda]

Can't get the 200, tried lots of mimic headers, cookies. But no results. Any advice?

[@gihonglee6167]

I followed your guide and edit a few lines of code so that I can scrap the whole job description. It worked well, but after 15 pages or so, I faced a captcha page and was unable to scrap anymore. I watched your user-agent video and changed the user-agent, still no luck. Is there any way I can scrap again?

[@therealwatcher]

how were you able to get the full job description? Doesn't the url changes for each selected job id

[@shrutitiwari4068]

Sir, how we scrap the data when the same class present

[@ishantguleria870]

what if i want to scrap next 5 pages data what will be the code ?

[@CodePhiles]

Good Job, but in the loop you forgot to add "i" in extract function , the data were replication of first page, thanks a lot, plus more option to make location and job titile as parameter as well

[@nathantyrell4898]

can you explain where to add the i in the extrct function? im dealing with this very problem right now

[@CodePhiles]

@@nathantyrell4898 see at time 18:43 in line#35 just make it ..... c = extract(i) instead of c = extract(0)

[@therealwatcher]

Do you know how I could extract the full job description ? since the url changes based on the selected job.

[@jt23ice]

I like your tutorials, they are concise, complete, helpful and useful. How can we add something like the link for the job post? I really struggle with selectors, even with tool plugins. Any advice or best references for purposes of scraping? My impression from a lot of docs out there is CSS can get in the weeds and ya need a PhD or something. Here are my fails..Couldn't determine if it was a "static" identifier, or one that is an incremental, sja0..n , etc. #WebAddress = item.find('sja1').text.strip() #jt added this , no go #WebAddress = item.find('sja1') #jt added this , no go #WebAddress = item.find('span', class_ ='sja1').text.strip() # compiler says it aint text #WebAddress = item.find('turnstileLink') # whole bunch o' nope WebAddress = item.find('div', {'class' : 'turnstileLink'}) #fail number 972 lol job = { #ColumnName : value 'Title': title, 'Company': company, 'Salary': salary, 'Summary': summary, 'WebAddress': WebAddress #web link for the job #jt added this

[@raph6709]

Thanks

[@harshrohilla6151]

How do i save xpath element data in a csv file ?

[@imanolbelausteguigoitia5510]

You are so crear and Good. Do you hace courses?

[@hanman5195]

@john - i am getting cookies msg like "We use cookies to personalise content and ads, analyse traffic and improve our services. We also use essential cookies to inform employers of clicks to their jobs." Please help me to get rid of it.

[@arsalraza3997]

GREAT. Can you tell me how to go inside these jobs urls? how to get jobs urls!?

[@munimovi]

When i try to check the response of glassdoor website it says response 403. what to do now?

[@rahulpurswani809]

can someone please explain, why do he use header agent ? what is the use of it ?

[@InsightpediaOfficial]

Can anyone explain following error requests.exceptions.ConnectionError: ('Connection aborted.', PermissionError(13, 'Permission denied'))