RPA Quick Tips #4: Headless Web Scraping | Automation Anywhere

[@juniorpraise]

Thanks for sharing this video. I'm facing a situation that I need to extract a several information from web based on a specific shipment. As example, there are lof of information such as ETA, Shipping number, Vessel name and etc. How to create a bot able to extract it and put in the specifics rows in the excel file?

[@MicahDanielSmith]

Hey @Junior - we have a tutorial that shows interacting with a web application here. The primary goal of this tutorial is inserting data into a webpage, but you could do the same approach for reading from it too (just use get property instead of click/insert text). kzbin.info/www/bejne/fJnEhoWsa9-tmqs - let me know if you run into any other hurdles.

[@juniorpraise]

@@MicahDanielSmith Hi @micahman! Finally I could create the bot. Everything working as expected.

[@katarinaelenkov2930]

Can we do scraping of whole page without WEB services?

[@bagia1000]

what if i want to scrape pattern based text (scrape multiple text in one webpage, with multiple pages) by using specific technology ? (other than the autodetect) I can recursively scrape the text using autodetect method, specify specific DOMXpath , and creating variable so that the text can be scraped recursively. But some websites should be scraped using specific technology other than the autodetect one and it does not contain the domxpath field.

[@MicahDanielSmith]

Its 100% going to depend on the type of application you're trying to scrape from. Some pages load dynamically, which would make the approach shown above less than ideal. In those situations, you could use Automation Anywhere's recorder, Browser Run JavaScript, or Web Automation Packages to extract data/interface with the applications in different ways.

[@sandeepuniyal6817]

what if the webpage is dynamic

[@Automationanywhere]

Hey Sandeep - if the webpage is truly dynamic (in that it has new object ID's each time the page loads or there is a variable amount of data) - you'd likely be best off using the built-in recorder package, the Browser: Run JavaScript action, or the Web Automation Package from Automation Anywhere's Bot Store (dependent on if there is a requirement for headless web scraping or not)

[@hal9058]

Nifty And much easier with the new AA IDE

[@MicahDanielSmith]

Thanks Hal!