Web Scrape Text from ANY Website - Web Scraping in R (Part 1)

Web Scraping in R is super easy and useful, and in this video I scrape movies from IMDb into a data frame in R using the rvest library and then export the data frame as a csv, all in a few lines of code. This method works across many sites -- typically those that show static content -- such as Yelp, Amazon, Wikipedia, Google, and more.
Part 2: • Web Scrape Nested Link...
Part 3: • Web Scrape Multiple Pa...
Part 4: • Web Scrape Tables - We...
SelectorGadget for Chrome: chrome.google....
Code: github.com/abh...
Music by: LAKEY INSPIRED

[@AbhijeetSinghs]

Learned more from this than from other 1hr long videos. Thanks for making this video.

[@-Exen-]

This is the first coding project I've ever completed. Your tutorial was extremely intuitive, thank you!

[@TheMunishk]

Some people can explain things in a neat and simple manner. This video does that

[@Unbox747]

You have such a calming voice and such clear explanations!

[@kathrynwang4081]

I thought it was really helpful that you explained what the rvest functions are doing! Thank you!

[@deltax7159]

great video, been a SAS user for a while but really getting into R, your videos really help, thank you!

[@alvaromartinez3310]

Excellent tutorial, I've been searching for this long time. Thank you so much, bro. Here you have a new sub

[@rahulraghavan1894]

Amazing tutorial. Quality content!! Subscribed immediately after I saw this one tutorial. Hats off for the good work.

[@willykitheka7618]

I can't believe you have taught me web scraping in 8 minutes! Thanks a heap! Ooh, I subscribed!

[@josephfife8946]

Such a great video! Thanks for putting this together! I love how clear and concise you were with each part! When I was following along I decided (for personal use of scraping imbd) the content rating (G, PG, PG-13, R ect...) was important, but was having some issues adding it to the table since not every movie (content) rating was available. This is what I ended up doing to get around that issue, in case anyone else finds this useful. #Part 1 Select content rating and a variable that does not change (This one was ended up having text of "Rate this") get_rating = page %>% html_nodes(".rate , .certificate") %>% html_text() #Part 2 Make a for loop that adds in 'Not provided" when a movie does not have a rating i = 1 is_null = "Rate this" content_rating = "Rate this" count_rate = 1 for(i in get_rating){ if(get_rating[count_rate] == is_null) { content_rating

[@barankaypakoglu7643]

Very clean explanation. Super useful stuff! thank you for this

[@haoranliu8204]

This is the all time best tutorial!

[@silvestrecamposano6317]

Thank you for the very simplified explanation that we are able to understand.

[@loganlloyd3581]

This is very well done and helps out a lot, thank you!

[@scpbm]

You've just helped me save time, as I am gathering data from different websites. Thanks a lot!

[@dataslice]

Great to hear! Thanks for watching!

[@bastih9816]

I don't comment often but this is so good quality content mate

[@hcrnn7518]

Thanks, Man..It's so easy to learn from your videos..and I needed this for my work in the office..You have no idea how much time this has saved me..A subscribe and thumbs up from me!!!!!!!!!!

[@antxnioo]

i never coded in R. this made it look so easy. Thank you!

[@bunnyhei]

Thank you very much! Your great tutorial video straight to the point!

[@fabienneraier1140]

A great tutorial, I got it to work right away! Thank you so much! :)

[@jean777-p2t]

Thank u very much! i learning to use R Studio, and its my first time in practice Web Scraping. I really so' happy :D

[@jean777-p2t]

Greeting from Argentina

[@thecardigancardigand]

Thank you! Very useful and clear explanation.

[@Ricefield88]

Thank you! I’ve tried python and mostly failed but this tutorial worked!

[@evanglaser6517]

Super helpful and concise, thank you!

[@giannispets]

Thank you for the tutorial. Very nice and on to the point with blah blah

[@terraflops]

i was hmm, okay, hope this is easier than bs4 in python, and just using the chrome extension with the name variable code .... AWESOME!! that was so easy! Thanks so much

[@hayekri]

I hope you get more subscribers b/c this is a very effective overview! Thanks!

[@dataslice]

Thanks!

[@dimasprasetyawardanadana7682]

Your video is so helpful. Thanks a lot!

[@eloscarc5782]

Wow, what a great explanation

[@Cx787]

Thanks ! this is really helpful. One question about the data, I usually work with spanish web pages and the text have special characters such as á, é, í, ó, ú. These characters do not appear in the CSV file (they appear different as A', Ä, etc). Any idea how to solve this? I used to replace each one manually lol

[@SC-bi6my]

One of the best video in youtube.

[@christianberntsen3856]

Very nice! However, on some pages the "read_html(link)" gets stuck in an infinite loop. Any idea why?

[@joaquincarrascosa91]

Great video, do you know how i could scrape the entire text from a website ? I was thinking of using it to make wordclouds as shown in your other video.

[@sub4morebysquawk427]

You got me scraping the world wide web. Thanks!

[@sub4morebysquawk427]

@dataslice, i was trying to do this with facebook and google search, like i was searching for dentists in the area, and wanted a list and contact number out of them.. But i only show the div part, of the inspect..

[@raj-nd6kz]

lol at Lagaan being in the list, one of my favorite movies

[@deepflare1]

I just try your script and run. and why it show this Error in data.frame(name, year, rating, synopsis, stringsAsFactors = FALSE) : arguments imply differing number of rows: 50, 47 any package that maybe I don't install? or important step I forgot to do it?

[@deepflare1]

and what R version do you use? why my R in the console is not show the data that I scrape ?

[@dataslice]

You're getting that error because the four vectors you pass into data frame need to all have the same size, and it seems like one or more of the variables is less. I'd check what name, year, rating, and synopsis are equal to and see if you're missing data points somewhere

[@ac6852]

You are a freaking legend! Thank you for this awesome video!!!!!!!! xoxoxo

[@buffaloperformanceandanaly1431]

Awesome video, thanks for sharing! Is there a way to read in images? Thanks!

[@retrolu1]

That looks so easy, thank you for that

[@mrk9076]

Hi everyone! Just a question: why my SelectorGadget don't put the code when I highlighting the text is just show "#main a" which is not the code. Anyone can help me please?

[@ali5t4ir]

Thank you so much for this, well explained!! I have tried this on a website & I get "Error in open.connection(x, "rb") : HTTP error 405." - usually in Python I think they use Hearder or User Agent to bypass this - is there any way to incorporate this in R please?

[@arshammikaeili7408]

This is the best Good quality Best way Not too long Fantastic 👌🏼👌🏼👌🏼

[@Jason-ot3fu]

Hi DataSlice, thanks for the great tutorial. I was wondering why when I type "View(movies)" I can see the synopsis values but when I export it to CSV, I can't see the synopsis values in the CSV file.

[@dataslice]

That’s an odd issue - are you sure the synopsis values aren’t there and just hidden? What command are you using to write to the csv?

[@kavitakamatdivekar5152]

@R for students | Dr. Fahad synopsis values are there, just increase the size of excel cell row, you can see it.

[@retrosak1977]

Such a great video 👏👏👏

[@saminba9111]

Hi, i have a question about your video, suppose that I extract the CSV file from a webpage for the engine capacity of different make/models of the cars. now I have make/model and engine capacity . should I then manually search in the CSV file to find each make/model engine capacity related to my dataset? i mean after scrapping, should I manually find data in the CSV file?

[@hineshpatel7076]

hi great video, super useful. Are you able to do a video on scraping behind a login page ?

[@ammarparmr]

Informative video!! I just have a question, How to add a random delay time to avoid blocking

[@johnbuhl7863]

What do I do if the name field is empty? I followed along with your example and had no issues, but when I tried doing what I needed it for I couldnt get any values in "name"

[@FroFoLife]

What if you can't select individual data elements on the page?

[@eduardobustamante1797]

This is the best tutorial, thank you so much

[@TheApexsha]

Hey, I tried to do this exactly for youtube videos but the columns have 0 characters. Would you know why? Thank you.

[@ignaciomorenobasanez3821]

I encountered the same error, but when I tried another page, it worked well. I believe the package does not function directly with pages built using JavaScript.

[@jasonarchimandritis1183]

This is great thanks! Curious can this be used to scrape a youtube search result (I tried and couldn't get it to work, but ran your imdb code and it worked fine, not sure if it has something to do with the youtube search code or something) Thanks! :)

[@dataslice]

Yes, unfortunately this method will only work for sites where the content isn't generated dynamically after the page loads (e.g. KZbin). To scrape KZbin, you'd likely need to use the RSelenium library which allows for more advanced web scraping techniques

[@jasonarchimandritis1183]

@@dataslice Gotcha thanks so much! I will check that out! Any chance you'll put up a Rselenium tutorial anytime soon? ;)

[@dataslice]

@@jasonarchimandritis1183 I've got a lot of video ideas in the backlog including RSelenium, so hopefully soon!

[@buraktiras93]

Great content, thanks! Waiting for your new videos!

[@ucabcd7003]

Thanks!! I follow your code here, but i does not work, I'm so neofit ... does this plataform allow scrapping? or maybe I made something wrong?

[@gizl1]

Great video!!

[@lifefaithworks]

Hello, great video! How do you scrape the next page.. etc to the end

[@ahmedfaraz9813]

Thanks a lot Just one question. On my page some of the movies are missing IMBD ratings and hence when i ran the command "(Error in data.frame(name, year, rating, synopsis, stringsAsFactors = FALSE) : " arguments imply differing number of rows: 50, 41" what to do about it?

[@previncoin8592]

My IMDB page has 41 titles as confirmed at the top. All columns return 41 elements except (year) which returns 43, this causes a mismatch: "Error in data.frame(name, year, rating, synopsis, stringsAsFactors = FALSE) : arguments imply differing number of rows: 41, 44" This is because the first 3 entries in (year) are: [1] "IMDb user rating (average)"(2)"Number of votes"(2)"Release year or range" I cant see where this is coming from as there are no extra highlights in the gadget selection, is there a way to return only numbers for year)?

[@kavitakamatdivekar5152]

years= page %>% html_nodes(".lister-item-year") %>% html_text() will work

[@hm.91]

Great video! Thank you very much

[@ogclinton4780]

Great video. Would this work if i want to get data off of a website say number of views and visitors of a website or organization site?

[@celmywall]

First, great tutorial! Thank you. I had a problem creating the data frame because I have a different number of rows in some objects (45 or 50), so this is the reported error: Error in data.frame(name, year, rating, synopsis, stringsAsFactors = FALSE) : arguments imply differing number of rows: 50, 45. Any suggestion on this? Thank you

[@tainafelippe4842]

Hello! I love your videos, very easy to understand even for ppl who have English as a second language like me. Unfortunately when I tried to replicate this script, theres a problem in line 10, when I print line 10 to see its content it shows "character [0]" instead of the information that appears to you (the names of the movies). I tried using both your example and other websites but the problem remains, has anyone else had this issue? Thanks!

[@retro527]

you have such a nice voice 🥺❤️❤️❤️

[@yashs1999]

So helpful, thank you so much!

[@HadesTimer]

how do you deal with this if you don't have a data frame with the same number of rows? This one lined up but it would be easy to get data from a page like this that doesn't.

[@fk-xj5oj]

thank you. thats very helpful

[@jonplaud]

I got the webscrapping part down but the data.frame keeps showing up as an error. I keep getting Error in data.frame(name, year, rating, synopsis, stringsAsFactors = FALSE) : arguments imply differing number of rows: 51, 50

[@DudeGuyWho]

Excellent content! How can I download a multiple tab xlsx file into R from a URL. I know how to merge the tabs together once saved locally, but would like to read them in directly from URL into R.

[@DudeGuyWho]

Awesome content! Can you help me understand how to download a multi-sheet xlsx workbook from URL into R? It's only two tabs and I do know how to merge the tabs into a single dataframe once downloaded.

[@DappuDon]

What if I am in a situation where I want to search imdb. How do I handle search engine?

[@nth.education]

wow, this is so so cool

[@igorl7274]

Great tutorial, but I still not able to get text from dynamic webpages. Do you have any intention to do some tutorial for this? Thanks!

[@dataslice]

Yeah unfortunately dynamic pages are a little tougher and require a package like RSelenium. I've got a lot of videos ideas in the backlog, but I do hope to cover it eventually!

[@jacobmooslarsen1134]

@@dataslice i would love this tutorial

[@user-tg6qk1il4u]

@@dataslice Please make a video on the dynamic web scraping! I’ve tried everything on Google and nothing works

[@fleetwoodayisi9308]

is there a way to accont for items with a missing variable for example movies that have no cast so that the final output does not result in a dataframe error?

[@SteashEdits]

I ran the code for the title and worked perfectly fine. After I added the same code to get the year, neither year or title worked anymore giving me an error: “ no applicable method for 'xml_find_all' applied to an object of class "function" “

[@nancyachiengodhiambo9727]

thanks so much, waiting for scraping multiple links

[@dataslice]

Hey Nancy -- the rest of the series is up! Part 2 is here: kzbin.info/www/bejne/e2TTd3WmatSDi5o and 3 and 4 are in the description as well. Thanks for watching!

[@manu3939393]

Mhh, I'm getting "Error in open.connection(x, "rb") : HTTP error 403." if I do this in R for the page I want. Using your Google Sheets Tutorial works, however. But since I need nested links that's not really useful. Any ideas?

[@marvelousmike79]

How do I return values that are N/A? I am trying to scrape Indeed and some postings do not have the same variables e.g. salary.

[@walrexx_2370]

thank you for the great tutorial

[@nathasyapramudita6312]

is there any similar addons like SelectorGedget but in Firefox?

[@ymdec95]

Hi I tried loading the library (rvest) and library (dplyr) it shows an error saying there is no such package. What should I do?

[@dataslice]

What's the error you're getting? Did you install.packages("rvest") and install.packages("dplyr") beforehand?

[@ymdec95]

@@dataslice yes.. I did install the packages and a folder was created storing those files as well

[@josephjohns4626]

@Dataslice, I got the following error message when attempting to do the exact same functions: "> year = page %>% html_nodes(".text-muted.unbold")%>% html_text() Error in UseMethod("xml_find_all") : no applicable method for 'xml_find_all' applied to an object of class "function""

[@jerryzhang2693]

Why I have trouble with doing that, the data shows that 0 objective and 1 variable. "No data available in table"

[@ahmedfaraz9813]

My question...when i wrote file to CSV, I did not get the synopsis in Excel file...why is that

[@papaorgen4224]

could you please do a video with scraping off a website with ? rvest doesn't seem to help

[@onsfarhat1042]

Great video! Thanks :)

[@previncoin8592]

I got 50 titles & 38 ratings which returned an error, so had to remove rating column to run it. How can missing values be replaced with for instance, N/A?

[@suryamadduri1353]

Thanks for your videos. How to extract movie review from IMBD in R?. Please suggest

[@dimplenain0692]

What if there are any missing value for any variable like ratings? How to handle these missing values?

[@elizulkatri6758]

I can save the data in csv format, but when I opened, the data still not organized and was not in table form. what should I do?

[@logic0057]

Awesome!

[@sophiej4605]

How can I solve this error? When I run the "movies=data.frame(name,~~)", an error message shows up like "arguments imply differing number of rows: 100,91,1"

[@boon8472]

in the csv file the synopsis is blank cause there is commas in it. is there a way to fix it?

[@previncoin8592]

Very powerful stuff.

[@aleksandrawiacek1892]

what if theselection consists in two phrases from Selector Gadget? e.g. .altrow td:nth-child(1) , .row td:nth-child(1)

[@GnarTank]

Some of the information that I've tried this on is coming out as double in length. I'm trying to practice this more using data from one of my friends league of legends games. Using leagueofgraphs to get the data. For some reason when I try to get the .gameMode information, data seems to double itself. And when I try to get the outcome of the game, Victory/Defeat, it returns the information as either all Victories with 5 blanks or all defeats with 5 blanks. Does any one have any advice how to fix this problem?

[@oneone8017]

This method works for instagram? example: instagram comments?

[@goyanks08]

Any suggestions on how to do this on a website with a login and pw?

[@maazafridi2090]

really awesome

[@yimeilong5518]

Hi, thank you so much for your videos. I have a problem when doing so. I use View() to check the output, all columns look great, but when I use write.csv() to export the output, open it, I found some parts are missing, do you know what's the problem? Thank you so much.

[@dataslice]

That’s odd. Are you sure they’re completely missing? There may be a new line character before the data and maybe your CSV viewer isn’t being displayed? Or maybe try cleaning the text in R (removing all special characters from your data)?

[@yimeilong5518]

@@dataslice Thank you so much. My fault, they are not empty, there is space at the beginning, that made them look like they are empty. LOL

[@michelepaleologo6310]

That’s awesome