Dynamic Javascript Scraping - Web scraping with Beautiful Soup 4 p.4

[@7208044878]

I was banging my head with all those headless browser methods to run javascript. This is so much simpler. Thanks man! appreciated!

[@elahehosseini3933]

You can't imagine how your tutorials are useful to me. I'm really thankful and hope to you continue making videos like this

[@elahehosseini3933]

Never and ever sign up for fliixzone site, this is just bad trike!

[@EndersupremE]

I was just searching for a problem with this and BAM, u have an entire series on webscraping. I think its the 5th time this happens. Just sayin realy appreciate ur channel.

[@HarshPatel-ly3dh]

its WOW... i spent whole lot of time scraping dynamic content but couldn't. This was a very good idea.

[@sajjadhossan7972]

If it is possible I would like give this video thousands of likes

[@sertormi]

Thank you Harrison. I'm a fan of your python tutorials, I love python. Could you please make some tutorials about web scraping using Selenium to login in forms and scrap dynamic data?

[@georgitanev-w4b]

I like your videos. One of many, who fit my way of learning.

[@Londonwebfactory]

Great Tutorial Chum! Many thanks.

[@leandrowitzke6405]

Nice Sentdex. I was thinking of using PhantomJS for javascript, but is still more slower. I hope more videos like these. Thanks

[@Apfelloch]

QtWebKit is not supported anymore in PyQt5 *32*-bit. You have to install it by explicitly using a *64*-bit version of Python, e.g. py -3.7-64 -m pip install PyQt5 (the "-64" is important) then you should use: from PyQt5.QtWebEngineWidgets import QWebEnginePage from PyQt5.QtWidgets import QApplication from PyQt5.QtCore import QUrl

[@choudhurysudip666]

Thank you so much for putting out the answer! :-)

[@humayunkabir7925]

from PyQt5.QtWebEngineWidgets import QWebEnginePage NOT WORKING -_- using Python 3.6

[@finfan7]

Ooh, been wanting a look at python multiprocessing recently. I look forward to it.

[@hamzajibran5402]

You my friend are my Guru

[@chemhong]

Great~~~ Thanks Sentdex, I have been looking for this a really really long time~~ thank you so much~~~~~~!!!!!!!!!!!

[@idobenamram3743]

cant wait for the oop series plz hurry!!

[@satishpatil115]

Works fine with PyQt5, Thanks for tutor

[@tomaszbonarkiewicz6368]

Could u give some advice how? I have errors like "Importerror"

[@geneensor2326]

Yes, please post your code to help the rest of us. Many thanks!!!

[@mubeen44us]

You are a good teacher! Cheers

[@sentdex]

Thank you!

[@noelcovarrubias7490]

Could you please make an update video of this? PyQt has had a few updates or there is other modules to use. I'm trying to do it using selenium because I feel like it is the best for what I want but I just can't pass the "verify your identity" bs since webdriver doesn't take headers, and I haven't found a different way to do it. Thank you!!!

[@lennon4044]

Great turorial ! but I use PyQt5 so may I know the code for the Qt5 sir ?

[@datahat642]

Could you please explain the reason for using PyQT and not anything else here? Also state the alternatives. Thank You.

[@sumitdubey4386]

Hi Harrison, hope you are doing good. I am trying to fetch the Data through PyQt5. The webpage has button "Show More". I am able to control the "On Click" event through Python, still not getting the full list. Can you please make a video on extracting data for such events like 'Onclick"

[@Victor_Marius]

I've done something similar yesterday with PyQt5. I've combined html, javascript and python into one app (and some css goodies)

[@11hamma]

Can you provide the code? thanks

[@chrisharrel8837]

I usually use Selenium for scraping dynamic web pages but I really dislike it because of how slow it is. I'll try using this for a couple scrapers and see if it's any faster. Thanks for the info.

[@sentdex]

The only reason to be using selenium is if you're trying to fully mimic being a real user. Doing things like clicking/interacting with the website. If you don't need that, yeah you wouldn't want to be running that whole driver.

[@chrisharrel8837]

I don't usually need to interact with the webpages, but I often scrape pages which require additional server request to be made. For example, many ecommerce sites deliver the base page in the initial request, and then subsequent server requests are made to fetch JSON or other HTML which is inserted into the page. Prices are a great example of this. Will your method here evaluate the Javascript and make those additional calls? I am not referring to AJAX or sites that load more as you scroll.

[@sentdex]

Those requests are almost always done in the form of a link. When you click a button, or choose a drop down that dynamically changes a page, it's almost always a request made to the server. If it's a button you press, right click that button and copy the link. Chances are, that link is making a request to some sort of API, on or off site, which will have params in the URL, and that url will return a json of data. To do all of that, you definitely don't need selenium, you just have to handle the json data yourself.

[@sentdex]

Here's an example of some jquery that updates the page live from my flask tutorial series: $(function() { $('a#process_input').bind('click', function() { $.getJSON('/background_process', { proglang: $('input[name="proglang"]').val(), }, function(data) { $("#result").text(data.result); }); return false; }); }); Notice the part $.getJSON('/background_process' That's going to query www.thewebsiteyoureon.com/background_process ...and supply parms for proglang so the URL literally would just look like: www.thewebsiteyoureon.com/background_process?proglang=something Then that response will be a json, and you can handle that with Python's json module. In the case above, the result that populates the #result id data would be under the key of "result"

[@ishanksharma5999]

0akistà LP nodràma Chris Harrel

[@PKrishnamaNaidu]

Hi, I have been working a lot lately on web scraping tasks and I was using selenium as it required interaction with the web page. My question is there a generic or more common way to extract any web page content instead of navigating and identifying tags which has required information. If not, why? Also looking for how to control sending multiple requests to a server at a time while trying to fetch the data so that it would not stop taking my requests.

[@hardikajmani5088]

great series! went through it.. I wanted to know that how we can enter data in an input box in a form on the web page and scrape the results (complete process from python)

[@chrisgrippo371]

I'm getting an error "AttributeError: 'Client' object has no attribute 'mainFrame'" any thoughts on how to fix this? I'm using Python 3 and PyQt5. For PyQt5 I used: from PyQt5.QtWidgets import QApplication from PyQt5.QtCore import QUrl from PyQt5.QtWebKitWidgets import QWebEnginePage I can't figure out what's causing that.

[@firstnamelastnamesons6830]

take a look at 'stackoverflow.com/questions/42147601/pyqt4-to-pyqt5-mainframe-deprecated-need-fix-to-load-web-pages'

[@yergali]

Delete mainFrame on Client class: class Client(QWebEnginePage): def __init__(self,url): self.app = QApplication(sys.argv) QWebEnginePage.__init__(self) self.loadFinished.connect(self.on_page_load) self.load(QUrl(url)) self.app.exec_()

[@iwanhanjoyo107]

thanx for the tutorial. It helps a lot

[@mdsarwar5273]

2:20 when i run the code showing this error: Traceback (most recent call last): File "C:\Users\username\Desktop\a.py", line 9, in print(js_test.text) AttributeError: 'NoneType' object has no attribute 'text'

[@mdsarwar5273]

code: import bs4 as bs import urllib.request source = urllib.request.urlopen('pythonprogramming.net/parsememcparseface/') soup = bs.BeautifulSoup(source,'lxml') js_test = soup.find('p', class_='jstest') print(js_test)

[@shyambutani8618]

Hey sentdex.. please help me. In my case the html is generated dynamically through ajax call. With this code, I am not able to scrape the required data. Is there any way through which I can wait till the ajax call is made? i have tried qWait but it did not worked.

[@Azariven]

Oh sentdex thank you so much again for making me level up in programming grind. What makes you keep going with all the programming? Too much coding often drives me nuts.

[@theglobalconflict6904]

but, this is'nt working with pyqt5 and I'm unable to install pyqt4. What's the solution???

[@schwazroda7882]

can you please do a series on splash and scrapy? I can't find anything on it

[@huanwang4926]

Hi sentdex, thank you very much for sharing your Python programming experience. May I ask a question? Is it possible to extract the information "Look at you shinin!" between the tag without mimicking the browser?

[@OBPagan]

in 2021 I am unable to install PyQt4 on the latest version of Python 3.9. I use PyCharm under Windows 10 and just can't figure out how to get it to install. Any ideas would be greatly appreciated.

[@Yawgmoth1806]

Hi, I've just seen your video and it helped me understanding the principle behind scraping dynamic pages. I tried the code on your page and it worked fine, but I ran into a problem: I tried it on another website and after like 15 minutes the line: "client_response = Client(url)" is still being executed. Does scraping like this takes an eternity for bigger sites? Or is something wrong with code? I am using pythin 3.6 and 4.11 pyqt. Regards

[@naimurrahman2229]

is there any way to use it in a py 'Qt designer' Gui app? as QApplication(sys.argv) is called twice then and so new event loop is created and function fails to execute.. any solution? :/

[@jasangm4552]

AttributeError: module 'PyQt5' has no attribute 'QtWidgets' It seems like these modules have been deprecated now, I haven't found how to import QApplication to do this tutorial

[@anastasialee8083]

Hello! Thank you for these lessons! What is wrong i did?[ Traceback (most recent call last): File "C:\Python\parse1.py", line 2, in from PyQt4.QtGui import QApplication ImportError: DLL load failed: no found this module]

[@mahmoudtalebi7034]

hi, can we install pyQt4 on centos 6. or on the other hand i wana develop webapp and upload in VPS host for extracing data. PhantomJS makes so many problem in cgi-bin therefor I thought qtwebkit could be better.

[@knotratulshorts]

@sentdex Bro, I've been watching your tutes of a long time and its helped me loads!

[@farshidbalaneji1271]

Hey, Thank you for your great dedication in sharing your knowledge which was a great help to me. I am wondering how to scrape websites with infinite scrolling. I read that beautiful soup is not capable for doing so and another option would be Selenium. I want to scrape an infinite scrolling container in a page that includes three different containers. I was told to send request to scroll the box but I couldn't find any pattern in Ajax request. Any help would be appreciated.

[@Victor_Marius]

You could make a browser and load whatever web page you want and when finished loading you can execute javascript from PyQt5 and even return some data back to PyQt5 from your javascript code. As javascript code probably you will use document.querySelectorAll, scrollIntoView, or just set the scrollTop property. And if you don't want to see the browser window, you can set full transparency on the entire app and transparency for mouse inputs (clicks, so you could not interact with your app). The app transparency can be set with QMainWindow().setWindowOpacity(0), QMainWindow.setWindowFlags(Qt.WindowTransparentForInput | Qt.WindowStaysOnBottomHint)

[@dieuhuyen0812]

Why can't you just parse the script tag instead of the p tag?

[@jasonjeong3541]

Thank you and I solved my problem, I just tried to use selenium or mechanicsoup..

[@minurapunchihewa4592]

I tried the PyQt5 equivalent to this, but I am not getting the expected results. The dynamic content still cannot be extracted. Any suggestions?

[@erica.7008]

kind of the same here. Sometimes it loads and sometimes it doesn't as if I was only using BS4. Did you manage to find a solution?

[@dataaholic]

Is it possible to scrape the pinned location from an Embedded google map which loads all its data using the jaavscript? The problem is that the location and data that i want to fetch is only load up for that location when we click on the particular location Thanks in advance

[@subhrajitmohanty7511]

I want to scrap from a website containing reviews comments load on click of read more. Could you please suggest me what I have to do? I am new to web scraping.

[@shepard267]

Can i write this code on a django site? I'm thinking to build a web scraping web app. Or perhaps can you recommend a better way?

[@greatsea]

Hi, question. I was able to scrape and write all Latin words into CSV file from UT Austin Latin glossary but not their Old Norse glossary. I get this error: Traceback (most recent call last): File "C:/Users/JohnP/PycharmProjects/FirstProgram/main.py", line 39, in thewriter.writerow([name.get_text()]) File "C:\Python36\lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode character '\u0301' in position 1: character maps to Here is the code snippet that works for Latin but not Old Norse. What am I missing? with open('some.csv', 'w', newline='') as f: thewriter = csv.writer(f) for name in nameList: thewriter.writerow([name.get_text()])

[@Hitman1978]

looks like your program can't encode some of the characters in your glossary. I would try to find a codec that has all the characters in your glossary.

[@SiliconAddictTV]

This is great, however in my situation, the website is adding content every 1 minute, how do I loop and reload just the page without reloading the PyQt client every loop?

[@HustonPetty94]

Same issue here

[@SiliconAddictTV]

@@HustonPetty94 I found a solution using selenium. I'm using headless browser with pyVirtualDisplay. Google about it, might help your case too :)

[@kevinl.9657]

@@SiliconAddictTV pyVirtualDisplay helped me. Thanks for this.

[@JJ-gu3wq]

Thank you for the video! How would you loop the code for PYQT5? whenever I try to loop it, it crashes python! Thanks!

[@tiehouafantasilue385]

Thank you for this tutorial i want To know if it is possible To scrape comments from website which use disqus api with beautifulsoup

[@직장인-r5o]

I have a question~! How can I make a new window in matplotlib? When I run plt.show(), it just shows its graph in ipyton console instead of making a new window. I use anaconda Spyder python IDE. Please... tell me how to open a new window~!

[@adamaleksander5226]

Great vid sentdex! And how do you scrape the new Yahoo website with React code?

[@aeroplaneman747]

Thanks a lot of this great tutorial! It works really nicely for scraping a single page, but when looping through multiple pages it retrieves all the html but throws this error at the end: QObject::connect: Cannot connect (null)::configurationAdded(QNetworkConfiguration) to QNetworkConfigurationManager::configurationAdded(QNetworkConfiguration) QObject::connect: Cannot connect (null)::configurationRemoved(QNetworkConfiguration) to QNetworkConfigurationManager::configurationRemoved(QNetworkConfiguration) QObject::connect: Cannot connect (null)::configurationChanged(QNetworkConfiguration) to QNetworkConfigurationManager::configurationChanged(QNetworkConfiguration) QObject::connect: Cannot connect (null)::onlineStateChanged(bool) to QNetworkConfigurationManager::onlineStateChanged(bool) QObject::connect: Cannot connect (null)::configurationUpdateComplete() to QNetworkConfigurationManager::updateCompleted() Any idea on how to fix this?

[@simonchan2394]

Hello Harrison. Did you eventually make that tutorial on multi-processing / mutlithreading with PyQt?

[@sentdex]

I do with bs4, you can adapt this example to it if you wanted to use pyqt as well: pythonprogramming.net/multiprocessing-spider-intermediate-python-tutorial/

[@mrbee781]

Thank you for such wonderful tutorial, learned a lot. have you uploaded multi_processing tutorial??

[@SoraAmm]

how do I link that code to html tag? so when the user paste a link it scrape and display the data on html?

[@Tsetse2fly]

Have you tried requests instead of urllib? What do you think of it?

[@sentdex]

I haven't, since urllib has always served me just fine, but lots of people are suggesting it, so I'll poke around with it.

[@krutpatel8168]

Please do work on that! :)

[@rishabhrai8404]

download requests module import requests page = request.get('url here') soup = Beautifulsoup(page.content,'html.parser/xml/lxml') '''this will work super fine'''

Thanks a lot, you're great!

[@sentdex]

Thanks!

[@ЕркінАбдукаримов]

Hello, i want parsing one website, which information update(add new) when you scroll down(info in table),and how i can parse all 'td.text'

[@abbii1661]

thanks, but can you update your video to work with PyQt5

[@ytdejvid]

Hi what python do You use i cant import urllib.request at any point. Using Python 2.7.14. Shall i update python to newer version? As far as i know python 3.5 its a way different than 2.7 but by any chance 2.7+ would have this module imported? Or i have no other choice than going to python 3

[@Hitman1978]

the urllib module underwent major changes when it was upgraded for python 3. In your case, the python 2.7 version of urllib.request.urlopen() is urllib2.urlopen()

[@leandrowitzke6405]

Sentdex, i have an error in the line of: soup = bs.BeautifulSoup(source, 'lxml'). Throw the next error: TypeError: tuple indices must be integers, not QString It may be for use a python 2.7?

[@med0msakni]

i had the same error i try with 3.0 it did't work

[@SeaAnswer]

I cannot download the PyQt4, the Binary installers for Windows are no longer provided.

[@atulanand2118]

Hi Sentdex, thanks for great explanation, but I am not able to import PyQt4.I tried in both OS: Windows as well as Linux. It seems now PyQt5 is also available. I am able to install these but I am not able to import the same. Request you to please create a lecture video to install and import PyQt4

[@ttaqinmu707]

Thanks dude ! Awesome (y)

[@sayak1997]

Hey, how do I pass session to PyQt? I've used requests.session() to generate a session as variable. so how do I pass the session variable to PyQt so that PyQt can open the webpage, else it can't open the webpage.

[@宏杰李]

you should try selenium. it's less type and user_friendly. and it's more acceptable for beginner.

[@fredykapustin1789]

I have a problem with the page, you could do an example with javascript with the event onclick (), thanks

[@ericroque7968]

Hey. Is there any way I can use Beautiful Soup to fill out forms, click a button, then scrape information off of a page? I want to create a web scraper/crawler that will scrape textbook information off of an online textbook store. To search for the textbook, I need to fill out a form and pick several options (department, term, course, section, etc), click a submit button, and wait for the page to load. Any ideas? Thanks.

[@prodtokegod6315]

use selenium webdriver

[@ewatson9875]

@@prodtokegod6315 selenium is not portable and also needs browsers installed , using selenium for a while now still looking for something better

[@himanshupoddar1395]

Sir,Can you please make the video for mimicking the browser using PyQt5,Please

[@perezroy6623]

You are the best

[@sangitasable6919]

I have seen your all videos. Sir I wanted to identify the computer subject sites only. I wanted to build such tool which can recognise only computer subject sites.

[@tuobraun]

I installed PyQt5 for Python 3.7 (x64) but getting this error in VS Code: that "No module named 'PyQt5.QtWebKit'". Could you please suggest any solution?

[@tuobraun]

Managed to solve the issue. You need additionally install QtWebEngine: pip install PyQtWebEngine or pip install PyQtWebEngine --user

[@nickklaushartin8122]

@@tuobraun Excellent thankyou

[@mitchellwoodin6686]

Is there any way to scrape comments from html to be able to manipulate that text? I can't seem to use soup.find_all('

[@nextMovieClip]

i am crapping a page that required login, i have login with my code but i can't scrap the data from the table beacause it is dynamic how can i do that with pyqt with the login?

[@giorgikakulashvili2665]

Can we get 'inspect element' instead of 'source code' of html by python?

[@webapplicationguide3798]

Thanks for the playlist..

[@raghavkumar7779]

How can we scrape websites that require login? Are requests.Session() a good way or can Qt be more helpful?

[@SivaKumar-sy2rr]

i'm getting an error 'TypeError: QWebPage(parent: QObject = None): argument 1 has unexpected type 'str''. plz help me sentdex....

[@shahmi98]

Is there any tutorial on how to web scrape from drop down menu?

[@choudhurysudip666]

Hey guys.. please read the problem here: I use usually Selenium to scrape data, but now I'm facing a website that identifies Selenium and blocks its JavaScript functionality so as to not reveal the data I need. Like for the first 10 times, it gives the proper data, then it just blacklists any approach with Selenium and gives no proper response. Hence, I used the BS4 module and the approach discussed here (with PyQt5 though), and the Website worked only ONCE! And then again it just gives the 'source' HTML without any dynamic data. How is it possible??? Do websites recognize PyQt calls etc.??? What do I do?? Please help guys (especially sentdex if you are still getting this!)

[@lakshyanegi668]

How do I scrape content of pseudo elements like ::before and ::after?

[@KhalilYasser]

Thanks a lot. I have encountered this error (I am using pycharm) ModuleNotFoundError: No module named 'PyQt5.QtWebEngineWidgets' Any ideas?

[@samirsaci6723]

If you got the error for js_test.text : be sure to have urllib.request.urlopen(link) and not urllib.request.urlopen(link).read()

[@qianli8866]

Can you make a tutorial of explaining how to import from a website that contains a list of links, and each link points to a different dataset. I wonder how to import those datasets from the links in the same webpage and combine them in a dataframe. Thaaaaaanksssss......

[@tonytoms9858]

Hey. I am running the exact same code. But the program gets hangs at the line : self.app.exec_() Its stuck there and I am force closing it. Could someone help he ?

[@westjr5085]

would this work with data generated from react.js??

[@ericckw]

Hi, thanks for making this tutorial. Can you also provide the codes for PyQt5? I've tried installing PyQt4 but i just couldn't get it to install. I have no other choice but to work with PyQt5 that comes with Python 3.6.

[@simonchan2394]

stackoverflow.com/questions/42147601/pyqt4-to-pyqt5-mainframe-deprecated-need-fix-to-load-web-pages

[@SeaAnswer]

I get this typeError: 'Qstring' does not have the buffer interface. Anyone got this problem? I tried google but it not working.

[@huongluu2632]

Hi there, I want to get all URLs from a domain, but I don't know how to do.... can you suggest me something? Thanks for reading!!!

[@huongluu2632]

I had just got this answer :D :P

[@charimuvilla8693]

It's amazing how everytime i have a problem in python i run into one of tutorials and solve it XD. Just thank you. But i still have a question: To make the program lighter in case there are several scripts can you somehow onl run one of them? Thanks again for the tutorials :p

[@sentdex]

+chari Muvilla not that i know of. Theres probably a way if you know the scripts beforehand and just block them like an adblocker, but I dont know how I'd implement that.

[@charimuvilla8693]

ok I'll try that

[@darkstria]

Hello, could you plz show the same for pyqt5.7 and its QWebEngineView?

[@josuecano4205]

is possible to do this without classes?

[@shyambutani8618]

You are GOD.. thank you

[@shyambutani8618]

But Still not able to get it right.. actually target webpage is getting data via AJAX call.. can you please help me with this?

By the way - what about websites that require logging in, for example Facebook? Can we scrape them?

[@sentdex]

Yes. Usually you need something more again, something like mechanize or selenium is what you'd want to look into for that.

I'll check it out, but if I wanted to use your method, would it be possible to include a cookie to the request? I think it would make a great part 5 of webscraping series btw :D

[@kvzound]

How does QWebPage work behind a proxy?

[@fredericjuge9762]

How can I get the Source Code showed in this video ? It could be faster than retype all :) Thanks

[@abhishekkwatra1426]

I've installed pyqt5 and these statements aren't working for me: from PyQt5.QtWebKitWidgets import QWebPage from PyQt5.QtWebKitWidgets import QWebView from PyQt5.QtWidgets import QApplication from PyQt5.QtCore import QUrl is there any solution to it?

[@jacobkasner7492]

from PyQt5.QtWidgets import QApplication from PyQt5.QtCore import QUrl (not sure why this didn't work for you) from PyQt5.QtWebEngineWidgets import QWebEnginePage

[@shelaraarti6082]

How to resolve content security error , I'm scrapping LinkedIn page

[@GlennMascarenhas]

Selenium seems like a better option for scraping dynamic webpages