UiPath - How to Data Scrape from a web page and save to Excel - Full Tutorial

[@andersjensenorg]

More Data Scraping from web? Check my 20 min long tutorial: kzbin.info/www/bejne/rHqke6mhadaop6M

[@Jsual]

Hi Anders, Big fan of your videos. I'm hoping you could help me with a problem or create a tutorial for using multiple data scrappers within a for each statement. Using a list of origin and destination airport codes (ORD,CPH) I'm goin to the website matrix.itasoftware.com/ to extract the cheapest flight for the month. The problem I'm running into is when outputting the data. For some reason my airfare outputs the header along with the price. EX: Lowest Cost $59. I believe it's an issue with building the table. Thank you in advance.

[@jaelyn689]

When i try to change the number to 1000 so i can get more results,but they just give me 24.Did i do something wrong?

[@gavp1336]

The best RPA channel on KZbin. I actually was wanting to learn but gave up a while back after going through so many long and unclear videos. I had a task to extract many lines of data from a website and stumbled upon your channel. Thanks so much!

[@andersjensenorg]

Hey Gav P. What a nice message to receive and sorry for the a bit late reply. I'm glad it worked - are you continueing your UiPath/RPA "journey"? Kind regards, Anders

[@kaiwei8934]

Hello im from singapore, can u teach me how to do multiple scrapping and append the data

[@andersjensenorg]

Hey kai Wei Thanks for writing. I'm getting more than 50 questions/comments every day and having difficulty answering everyone (I try my best). I've created my UiPath Groups, where we all can post and solve problems or hang out around our favorite tool, UiPath. So you're very welcome. Get the addresses here: kzbin.info/www/bejne/eWfbnnqcerFjbas Have a great day. Kind regards, Anders

[@simenandreasknudsen9272]

Thanks Anders! Just a question, what if I want to go into each item to scrape additional info that I can't just by looking at the overview page. What would be the best way to do that? Thank you!

[@andersjensenorg]

Hey Simen. Many thanks 😊 Yes, you can use a Get Attribute with the CSS attribute you want to scrape like 'href' Kind regards, Anders

[@simenandreasknudsen9272]

@@andersjensenorg nice! I’ll look further into that, thanks!

[@laurasarriafrancodesarabia9685]

Hi Anders! Thanks for your video! It was very helpful! Nevertheless I'm facing a little trouble. My code only extracts the data from the first page. It goes through all the pages correctly but it does not extract the data. Do you have any idea about how to solve that? Thanks again!

[@andersjensenorg]

Hey Laura. Thank you very much 🙂 What site is it? I can try to solve it then. Kind regards, Anders

[@laurasarriafrancodesarabia9685]

@@andersjensenorg Thanks a lot for your answer! It's Amazon too! Instead of "thermometers" I'm extracting data of "Engineering Manuals " (title, url, author, price).

[@andersjensenorg]

@@laurasarriafrancodesarabia9685 Hey Laura. It works from here. I have created a simple workflow for you, just navigate to Amazon and do the search for Engineering Manuals (you can ofc automate this too) and then run the workflow. gitlab.com/uipathqa/20200824lauraamazon Let me know, if that works 😊 Kind regards, Anders

[@laurasarriafrancodesarabia9685]

Hey Anders, thank you very much! Unfortunately, the same thing happens, it only collects the data from the first page even if it goes through all the pages. Maybe it's a problem with my UiPath program...?

[@andersjensenorg]

Hey Laura. That is strange, since it indeed collects data from all the pages here. I am not sure why. In the properties of the Data Scraping, there is some setting called delay... (I am afk now) - can you try changing that to more than 0, e.g. 500 ms? Then run it again. Maybe just start at page 70, so you dont have to take 75 pages 🙂

[@mrmavit]

Hi Anders, thanks again for the wonderfull tuturials, I have expanded this tutorial you did here, so i have made a way of reading the document saved above and extracting the urls and automatically saving the documents in the URL using the downloadFIleActivity, however the download file activity alows for saving for a specific format, do you know a way for making it dynamic?

[@andersjensenorg]

Hey Mr Mavit Thanks for writing. I'm getting more than 40 questions/comments every day and having difficulty answering everyone (I try my best). I've created my UiPath Groups, where we all can post and solve problems or hang out around our favorite tool, UiPath. So you're very welcome. Get the addresses here: kzbin.info/www/bejne/eWfbnnqcerFjbas Have a great day. Kind regards, Anders

[@vishnupradhaban2189]

Uipath keeps on updating its feature. The present Uipath community Edition has a modification in Data Scraping.

[@andersjensenorg]

Hey Vishnu Pradhaban Thanks for writing. I'm getting more than 50 questions/comments every day and having difficulty answering everyone (I try my best). I've created my UiPath Groups, where we all can post and solve problems or hang out around our favorite tool, UiPath. So you're very welcome. Get the addresses here: kzbin.info/www/bejne/eWfbnnqcerFjbas Have a great day. Kind regards, Anders

[@azzamabdullah3469]

Magic just got real. And free too.

[@andersjensenorg]

Hey Azzam. Yeah, it's awesome and free - my favorite rpa tool 😊 Kind regards, Anders

[@Suraj_Khote]

Question: if we are scraping large data and if bot fails in middle how can we save the data which is scraped till then..?

[@andersjensenorg]

Hey SURAJ KHOTE Thanks for writing. I'm getting more than 50 questions/comments every day and having difficulty answering everyone (I try my best). I've created my UiPath Groups, where we all can post and solve problems or hang out around our favorite tool, UiPath. So you're very welcome. Get the addresses here: kzbin.info/www/bejne/eWfbnnqcerFjbas Have a great day. Kind regards, Anders

[@el-olamfellowship7978]

Thank you so much it worked for me...

[@andersjensenorg]

Hey Jency, Nice! And thanks for writing 🙂 Kind regards, Anders

[@BrindhaBerry]

Hi Jensen, wish you a happy new year .your videos are helpful. During data scraping to move to the next page we are clicking on next button. Eg.i want to extract all the google contact but only few names are coming how to extract the data without having next button , i want to extract all the names from a to z

[@andersjensenorg]

Hey Brindha. Thanks for the question. At the moment, I’m on paternity leave, meaning I don’t produce videos and solve problems until the 23rd of August. Feel free to post the problem at my Discord, where many bright automation minds are gathered: kzbin.info/door/Pdtz4gd_iYebJFYq9N8pWAcommunity?lb=UgxD1Qe9lWKL1I7HyAJ4AaABCQ Kind regards, Anders

[@mariamahmed7556]

thank you so much that was very helpful

[@andersjensenorg]

Hey Mariam. Thanks for letting me know - have a great weekend 😊 Kind regards, Anders

[@automationbachir8547]

Hi Thanks for the this tutorial. I tried to reproduce the same sequence as you, I get I get the following error : “Excel Application Scope_1: Error opening workbook. Make sure Excel is installed.” This message is incomprehensible because Excel is installed and work perfectly, I use it daily. So please what is the problem? I use Uipath Studio Pro 2020.10.0-beta-149

[@andersjensenorg]

Skip the Excel Application Scope and the read range. Chose the read range from workbook activities. That will solve it 😊 Kind regards, Anders

[@automationbachir8547]

@@andersjensenorg Thank a lot, it works perfectly with workbook activities. You are so kind.

[@andersjensenorg]

Nice and thank you 😍

[@aksh2897]

Hi Anders, I have a table of nearly 79000 rows in a websites. But i am not able to scrap the whole table. When i open the page and scrape after like 5 minutes, i can scrape 10,000 rows. Similarly if open the website and scrape after an hour then i can scrape the whole 79000 rows. Can you please help me to find a solution?

[@andersjensenorg]

Hey Aksh28 Thanks for writing. I'm getting more than 40 questions/comments every day and having difficulty answering everyone (I try my best). I've created my UiPath Groups, where we all can post and solve problems or simply just hang out around our favorite tool, UiPath. So you're very welcome. Get the addresses here: kzbin.info/www/bejne/eWfbnnqcerFjbas Have a great day. Kind regards, Anders

[@uipathbachir7610]

Hi Anders, I have uipath v0.10 and there are no more "datas scraping" or "screen scraping". where are they with this new version?

[@andersjensenorg]

Weird. I have 2020.10.2 and both of them are there?? Kind regards, Anders

[@shithijkumar3370]

hi mine only moves to 2nd page and stops there it does not continue to go to page 3 4 5 how do i fix this

[@andersjensenorg]

Thanks for writing 😊 I'm getting more than 50 messages daily. While I read all of them, I can’t reply to everyone 😔 But I’ve created an RPA/Automation community where we’re 9000+ RPA Developers helping each other with solutions and our careers. Here’s the video on how to join (the invitation link is in the video description): kzbin.info/www/bejne/roipq2CJbpuLfdE Kind regards, Anders

[@isabelchau9974]

how to save to Gsuite google sheet?

[@andersjensenorg]

Thanks for writing 😊 I'm getting more than 50 messages daily. While I read all of them, I can’t reply to everyone 😔 But I’ve created an RPA/Automation community where we’re 5800+ RPA Developers helping each other with solutions and our careers. Here’s the video on how to join (the invitation link is in the video description): kzbin.info/www/bejne/roipq2CJbpuLfdE Kind regards, Anders

[@abhiramieringikal6501]

Sir, can u plz tell me.. How to scrap company details from getting into each & multiple linkedIn profile?

[@andersjensenorg]

Hey Abhirami. We need to make a loop and I doubt we can scrape all profiles on LinkedIn. Usually you want to make a search and then scrape the results. Kind regards, Anders

[@abhiramieringikal6501]

@@andersjensenorg thank you for your kind reply 🤗🤗

[@sunnynagar1345]

When I am scraping data through data scraping, then what should I do if my robot is stopping again and again? And the data has 1000 pages.

[@andersjensenorg]

Hey Sunny Nagar Thanks for writing. I'm getting more than 40 questions/comments every day and having difficulty answering everyone (I try my best). I've created my UiPath Groups, where we all can post and solve problems or hang out around our favorite tool, UiPath. So you're very welcome. Get the addresses here: kzbin.info/www/bejne/eWfbnnqcerFjbas Have a great day. Kind regards, Anders

[@wecare30333333]

how to open excel using same work flow after extracting the data ?

[@andersjensenorg]

Hey Aditya Sen Thanks for writing. I'm getting more than 40 questions/comments every day and having difficulty answering everyone (I try my best). I've created my UiPath Groups, where we all can post and solve problems or hang out around our favorite tool, UiPath. So you're very welcome. Get the addresses here: kzbin.info/www/bejne/eWfbnnqcerFjbas Have a great day. Kind regards, Anders

[@wecare30333333]

@@andersjensenorg sure, thanks, joining the group now

[@el-olamfellowship7978]

I have a doubt, can website owners identify and deny the action of Bot.

[@andersjensenorg]

Hey Jency. In theory yes, but then they would have to know, that it's a bot. We can just mimic the actions of a human. Unless they implement captcha, then it will get hard. Kind regards, Anders

[@el-olamfellowship7978]

@@andersjensenorg thank you so much for clarity

[@adriangoh2016]

Anders, I'm facing two problems: I couldn't find ExtractDataTable in the "Write Range" Tab, what should I do? For the website, it says compiler error encountered processing expression. Appreciate if you can help me with this!

[@andersjensenorg]

Hey Adrian. The first one you can solve by click the data scrape activity, go down go variables panel and change the scope to the outer sequence. Did that help? Regarding the second one, I am not sure, what do you mean? Kind regards, Anders

[@adriangoh2016]

Thank you for helping me out! First problem fixed perfectly. The 2nd issue im facing when running the sequence, it appears something like this: Error ERROR Validation Error Compiler error(s) encountered processing expression

[@andersjensenorg]

Awesome. Please try another browser 🙂 It is some kind of security setting in your browser. Kind regards, Anders

[@adriangoh2016]

@@andersjensenorg Thank you so much!

[@butcherbaby1714]

Hi Anders, I'm facing an issue with scraping, uipath is only scraping the first row of data not all rows (search items are 7 spanning in 3 rows and 3 columns ) do u know why it's I'm doing exactly the way you're doing except I'm opening in incognito

[@andersjensenorg]

Hey Butcher. Can you share the page to be scraped with me or is it private? Kind regards, Anders

[@tobiewaldeck7105]

Hi Anders. You have done a video where the next button doesn't work for data scraping. I'm dealing with a website checkatrade.com where the last next button is gray and uipath keeps clicking it because the 'element exists' is always true. What do I do in that case to get out of that infinite while loop.

[@andersjensenorg]

Hey Tobie Waldeck Thanks for writing. I'm getting more than 40 questions/comments every day and having difficulty answering everyone (I try my best). I've created my UiPath Groups, where we all can post and solve problems or hang out around our favorite tool, UiPath. So you're very welcome. Get the addresses here: kzbin.info/www/bejne/eWfbnnqcerFjbas Have a great day. Kind regards, Anders

[@jam05eun]

Hi Anders, I have a list of 6500 names listed alphabetically on 33 pages, is there a way for the bot to easily find particular name on those pages? My list has no search function. Do I have to go through each pages everytime I need to look for a name?

[@andersjensenorg]

Hey jam05eun. We need to iterate the sheets, which can be a bit of a process, as you say. Is the list in xlsx format? Kind regards, Anders

[@achuuuccc4576]

Hi Anders, can you help me to sort this same excel according price low to high

[@andersjensenorg]

Hey Achuuu, sure you can sort the DataTable (before you write to Excel) with the Sort Data Table activity. In Sorting column choose Name and order :) Kind regards, Anders

[@roipatrickflorentino817]

I tried extracting multiple pages but upon inserting it to excel it only writes up first page result of scrapping. HELP.

[@andersjensenorg]

Hey Roi Patrick Florentino Thanks for writing. I'm getting more than 50 questions/comments every day and having difficulty answering everyone (I try my best). I've created my UiPath Groups, where we all can post and solve problems or hang out around our favorite tool, UiPath. So you're very welcome. Get the addresses here: kzbin.info/www/bejne/eWfbnnqcerFjbas Have a great day. Kind regards, Anders

[@mousamroy6865]

My excel can't save the file

[@andersjensenorg]

Sure, whats the exception? Can you send it in the Discord? Link is at my channel page 🙂

[@shine2rust]

Thanks, really useful.

[@andersjensenorg]

Thank you 😊

[@vanquangchannel7054]

Thank you so much

[@andersjensenorg]

Thank you, Hoa 🙂

[@katszyj]

260 / 5000 Wyniki tłumaczenia Hi Anders :) I like your movies very much, but I have one problem here. How can I get results for all prices. I have to do this example on eBuy and just like you I don't get all the prices. I only need to save 30 examples in Excel. Regards Kate

[@andersjensenorg]

Hey Katarzyna! Thanks for writing. I'm getting more than 40 questions/comments every day and having difficulty answering everyone (I try my best). I've created my UiPath Groups, where we all can post and solve problems or hang out around our favorite tool, UiPath. So you're very welcome. Get the addresses here: kzbin.info/www/bejne/eWfbnnqcerFjbas Have a great day. Kind regards, Anders

[@uipathbachir7610]

Hi Anders, I found, now its’ called “Table Extraction”.

[@andersjensenorg]

Very weird :/

[@afgaiel]

5 *****

[@andersjensenorg]

Thanks a lot, Atef! 🤩