Web Scraping - find CHEAP flights with Playwright

In this video, I show you how to build a Web Scraper using Node.js
Web scraping is used to extract data from any website. We will be scraping cheap flight tickets using Playwright. Playwright is Puppeteer alternative. You can find more information & documentation about Playwright here playwright.dev/
💻Project code on Github: github.com/Cod...

[@nickstanovic]

Thank you! I'm just learning web scraping with NodeJS and Playwright. Your explanation and demonstration is so good that I don't need to watch any other video!

[@codewithosvaldas]

Thank you so much!! I really appreciate your comment!

[@JaKi052]

Very cool! I am new to dev and I always wondered how sites found cheap flights... Scotts Cheap flights and secrety flying for example. I figured it cant be all done manually. Will need some practise before learning this. Cheers!

[@pascallermit3200]

Thanks a lot for this video Osvaldas ! Really informative, starting Playwright I'm trying to go further in this example of flight ticket, do you have any discord to discuss about code, ideas etc... ?

[@codewithosvaldas]

Thanks! feel free to leave comments here on youtube video or create an issue at github.com/CodeWithOsvaldas/webscraping-playwright-flight-tickets/issues

[@Fodela]

Thanks for an amazing video. 🙏

[@olegkolesnik7710]

Hey, buddy! I'm amazed at how quickly you're able to scrape! You managed to complete eight cycles in just 10 seconds of waiting time. That's pretty impressive! However, I'd like to point out that true web scraping involves more than just completing one task. It usually involves scraping multiple web pages or performing repetitive tasks. Keep up the good work, though!

[@jsogman]

hello, Im brand new to Javascript and doing anything with the web and not sure: why is the JSON module process is necessary?

[@Malathsam1994]

Can you please help me, I need to login into a page and do something similar what you did. How can I store the cookies every time because I get session timeout when I try to consume the http request manually using Insomnia

[@codewithosvaldas]

You are using the playwright for the login, right? If that is the case, you can try to retrieve the cookies by await page.context().cookies() and set them with addCookies(). More documentation: playwright.dev/docs/api/class-browsercontext#browser-context-add-cookies playwright.dev/docs/api/class-browsercontext#browser-context-cookies

[@idcemail123]

I scraped a site with playwright inside my express js app.get() function. How would i deploy this scraped data to a website such as vercel? It works locally but when i try to host it on heroku i just get error 500 ?

[@soyebpatel2994]

How are you getting the helptext to appear. For example min 1.21, the timeout and url info when you filling the info out.

[@codewithosvaldas]

Hey, could you clarify the question? What do you mean by helptext? :)

[@mauriciobavaresco2055]

Good content, would you know if it is possible to use playwright inspector for scrapy, without test ?

[@codewithosvaldas]

Not familiar with scrapy yet :)

[@matiasb4662]

Thank you for your video. How can you bypass a site that blocks you?

[@codewithosvaldas]

That is tricky, so it depends :) first thing I would reach to is rotating proxies

[@AD-ey1gt]

hey, my pngs are not downloading. do I have to specify the path for that?

[@codewithosvaldas]

Hey, are you referring to the screenshots? Yes, you have to give it a path where to save the png. For example, await cheapestFlightCardLocator.screenshot({ path: `${from} ${to}(${city})-${actualPrice}.png` }); or simply await cheapestFlightCardLocator.screenshot({ path: 'my-screenshot.png' }); Are you getting any errors? It can be that your locator is empty? You can double check that by consoling out the inner html of your locator like so: const innerHtml = await myLocator.innerHTML(); console.log(innerHtml)

[@indirarath1092]

Hi, Thank you for the video. However, I am getting the following 'Microsoft.Playwright.PlaywrightException HResult=0x80131500 Message=Error: strict mode violation: Locator("div[data-test='ResultCardWrapper']") resolved to 3. Can you help resolve this ?

[@codewithosvaldas]

Hey, sure! The Locator API is strict, meaning if it finds more than one element with your selector and you try to call a locator, like .click(), it will throw an error. To avoid that you could loop through all 3 elements or just add .first() for e.x. const cheapestFlightCardLocator = page.locator('[data-test=ResultCardWrapper]').first(); You can check out the code here github.com/CodeWithOsvaldas/webscraping-playwright-flight-tickets

[@indirarath1092]

@@codewithosvaldas Thank you for the response. I had figured that out eventually and resolved the issue.

[@codewithosvaldas]

Nice! So did you managed to scrape some tickets?

[@aksharagupta9699]

Hi ,which software are u using ?

[@codewithosvaldas]

Hi, for what?