Twitter Scraping Tutorial in Python

[@KothaEd]

Hi Folks, Some times Instances might not be available to fetch tweets. In that case, try after sometime.

[@클래식클래식-x1r]

Thanks. mode = 'user' works well. But mode = 'hashtag' has error fetching.

[@avataraang3334]

@@클래식클래식-x1r same

[@mr.contentless5027]

i am getting index error . it was working fine two weeks ago but it not working sir !! please help me

[@MagnoFireSoftwareSolutions]

Thanku... It's very rare package... So much useful for academic research projects

[@KothaEd]

I am glad it’s helpful

[@mohammedbaig9569]

Legend bro, I needed this for an end of year project, bless you 🙏

[@corfy2813]

i need to any idea how to start or somethin i tried my own code but gets banned acc

[@OfficialLegend]

by any chance was it an A level NEA?

[@santhoshj985]

Thank you very much bhai! been scratching my head for week

[@uzairmughal9383]

excellent video, man! it's of great use since that old way of twitter scraping stopped working after elon musk implemented some changes to twitter API policy.

[@KothaEd]

Thank you 🤩

[@michealarun5482]

Thanks you. You gave me clear idea to implement on hedge fund strategy.

[@muradbaghirli]

Do you know any methods to scrape the comments from a certain tweet? I would be so grateful if you provide info on this, thanks.

[@akindefisayo3267]

Hello Sir. Great content, very helpful and rare tutorial. It will help in my research. However, i will like to download tweets from a particular region in the USA, i mean geo tagged tweets, along with certain hashtags. Please, how do i achieve this using this package and the function

[@MCS_T-2025]

How can I fetch images also, can I have more than 10 M records if I do it as batches?

[@tojo_MOV]

Thanks for the video, how can i use this to scrape data for a specific time period? Say from 2019-2021 specifically?

[@_Curiosity_]

Thanks bro! Really helpful content

[@tejaspednekar1412]

Can this be used for stock sentiment analysis project???

[@meetkambli1219]

can i fetch the locations from where the tweets have been posted too by this method?

[@MaytheLordbewithyou2]

Very helpful

[@KothaEd]

Glad it helped!

[@vegitasandy]

getting this if i run your code "No module named 'ntscraper'", why is that ?

[@Diego-Bravo]

Broo... thank you soooo much!! It helped me a lot

[@KothaEd]

Glad it helped!

[@AlphaGodz]

Thanks for this Video Brother.

[@KothaEd]

Thanks 🤩

[@VivianaIsabellaPenaPena]

Thank you for your help! i have a question: can you scrap tweet replies? thanks again

[@marianogarcia8130]

same question

[@randyluong6275]

can we get tweets from multiple users on News feed, or we can only get tweets from particular user account at a time?

[@KothaEd]

Yes tweets can be scraped from multiple users in newsfeed.

[@fadwabenromdhane2861]

@@KothaEd How please?

[@sukritgarg3175]

THANK U SOO MUCH BROOO LOTS OF LOVE

[@KothaEd]

Thank you 🤩

[@darshjain3133]

This Package isn't Working anymore i.e ntscraper cannot create more instances Do you have any other alternative??

[@nastaran1010]

Thanks.. I have two questions. 1) Are we only going to have access to the public pages?, 2) when I choose more than 5 tweets , I faced the error. How much data can be easily obtained ?

[@KothaEd]

By default it fetches 5 tweets, it can scrape more than that without any issues

[@huzaifa.arshad]

One question. Do we have to login first to out twitter to use this package?

[@KothaEd]

Not required

[@ginosawji]

Why do I only get 60 tweets? after that number it says that the page is empty.

[@cass-h4c]

I seem to be having the problem of all instances are being unreachable, do we have a solution for this ? thanks

[@KothaEd]

Sometimes instances might not be available. Please do try after sometime

[@Vantin33]

Hello, the mode "hashtag" works fine but the mode user give me Instances are unreachable. Check your request and try again. Can you help me ?

[@debatradas1597]

thank you so much sir

[@QuanquLeng]

Cool video! I want to ask that whether the number of tweets scraping by this method is limited or infinite? Because I need a lot of data to conduct my research. Thanks!

[@ginosawji]

Hey, did you resolve this? :) I need the same

[@yael2233ram]

thanks so much!! I want to scrape all tweets with a certain term, not by a specific user, how do I do that? please help!

[@KothaEd]

You can do that with term

[@yael2233ram]

Thank you, so should I put nothing before the mode?@@KothaEd

[@aadhyashreepandya]

Can i get a tweet detail from tweet id ? if yes, then what should we need to write in a code? please help me on this for my final year proejct.

[@HIMANSHUMISHRA-yf5bm]

Same doubt

[@dhruvkaul9072]

Thanks for this tool. I was also wondering if it was possible to also retrieve any images or videos associated with a tweet.

[@KothaEd]

Yes you can from tweet content

[@muhammadamir6499]

sir the ntscraper not fetching the tweets anymore is there any issue in it or please refer any other library that can be successfully scrap the tweeter.

[@KothaEd]

Sometimes instances might not be available. Please do after sometime.

[@nananastricia2937]

hello, were you able to get it to work? it doesn't work for me either

[@akanksha1323]

It is coming up with 'empty page on'... Were you able to get the tweets?

[@atulyakr.2003]

i am getting module not found error

[@martinrehbein7564]

Muchas gracias por su video me ayudaste mucho

[@gustavoeduardoberdejogaray2032]

Hi, thx 4 the video. Question: How do I use the 'near' param? Bc I only need tweets from Peru

[@KothaEd]

At current moment, there is functionality in Ntscraper python library to restrict tweets based on a location

[@ogr2spel]

Thank you very much! Is it possible to select a specific period of time?

[@KothaEd]

Just scrape all the tweets from Python Library. Later apply filter on tweet time to get tweets from a specific time.

[@AltafKhan-ka1zer]

@@KothaEd like applying it within code I don't get it. can you please explain?

[@KothaEd]

@@AltafKhan-ka1zer After extracting tweets from a specific keyword, you would get result set as tweets. On the result set apply filter on tweet datetime column.

[@3-fg4wp]

Thank you sir, is it possible to scrape quote of the tweet？

[@shahar-nx5wf]

Hi KothaEd, Thanks for a great explanation! is there any way to pull replies of a user with this package?

[@uzairmughal9383]

can we add multiple users in a single code? like if I want to extract tweets from more than one users?

[@KothaEd]

Yes you can. Add list of users as array and pass it

[@nameless3248]

How can i get the comments of the tweet? The text of the comments.

[@veenakumari5986]

Is there any way to fetch user location, just the country?

[@PelStrike]

Hi, are you purchasing the API for this to work?

[@KothaEd]

No it’s completely free

[@navyaangela1765]

Hi, i want to scrape Twitter data based on spesific keyword but to do SNA analysis i need column "in_reply_to_screen_name". Can it scrape that column?

[@nameless3248]

I wanna know too

[@justsimple2223]

You're doing a great job! I'm so OVERWHELMED trying to complete my dream app. But your video saved me like a lifeline :D My Question: I'm creating a Python program to find users similar to my bio, but I don't have an idea how I can retrieve. So, any advice from you made it happen.

[@KothaEd]

Try to get bio of all users and match relevant users based on similar words in bio

[@justsimple2223]

@@KothaEd okay, now it seems tangible. Here's the breakdown: - Fetch 50 random users. - Extract their bio (words). - Match their bios with the target user's bio using the same words. - In the end, return 10 people with similar words in their bios.

[@memesbyanony]

@@KothaEd hello sir how can i scrape all pages of that user on twitter ?

[@mstifler2023]

Hi, this was great tutorial, thankyou. Could you please do a video on how to create an api that can be used on front end? Also what is the best way to store the returned data in a database? Is it advisable to let the api handle that or should it be handled by front end code? I'm new to development/coding. Thanks.

[@KothaEd]

It depends on requirements. How frequently data is read by downstream systems.

[@stanleychukwuemeka8841]

Is scraping millions of tweets still possible because i would like to build an AI model using the text data. Please if it's possible how can I go about it?

[@Rrav1993]

Have you found a solution?

[@stanleychukwuemeka8841]

@@Rrav1993 no not yet

[@FaizaanHussain]

thanks a lot it is working fine for me, i really needed this for my research project. can i scrape top tweets or tweets during a time period instead of latest tweets?

[@KothaEd]

Thanks!!! If you need tweets from a specific period, just apply filter on tweet date field to get tweets from a specific period

[@bastiab5933]

@@KothaEd How exactly do you do that?

[@zakaria6419]

@@KothaEdi want also comments is it possible?

[@kenzakhelkhal7931]

Thanks a lot

[@KothaEd]

Happy to help

[@JeromeClarke-g1p]

what ide are you running this code on ?

[@KothaEd]

its Jupyter lab.

[@syedabdullah-jk6oe]

Appreciated your hard work. Is that really possible to fetch replies section on X profile and convert it into dataframe.csv? Kindly acknowledge.

[@KothaEd]

Yea that’s what shown in video

[@unaeantunestupinambas7121]

Hi, first of all thanks for the great video! Q: My code is returning too many fetching errors, is there a better way to do it instead the line below? Why does this occur? scraper = Nitter(0) I'm student, working on a scientific research, and I need a good amount of data. Thanks!

[@KothaEd]

The Nitter class object is going to initiate instances which are actually fetch tweets from twitter. Yes it causes some errors as instances might not be available at the moment. Try running the same code after sometime.

[@_daryyl]

Is it possible to fetch tweets from quote tweets?

[@safsataproje]

thank you so much

[@RicardaBedei]

Hey Thank you for this nice tutorial :) Is it possible to include the location from the different tweets in the file?

[@iSekarRaja]

IT showing - No module named 'tqdm' it seems. Kindly help

[@aizddnzhfri]

Help I have the same issues too @KothaEd PLEASEEEEE URGENTTT

[@KothaEd]

Some times instances may not be available. pls try after sometime.

[@raghadfahad211]

thank you sir

[@HIMANSHUMISHRA-yf5bm]

Did it work?

[@raghadfahad211]

@@HIMANSHUMISHRA-yf5bm It didn't

[@biobot-d3q]

I got some problem while using it. here it is: New working instances: 17-Nov-23 23:21:20 - All instances are unreachable. Check your request and try again.

[@KothaEd]

Sometimes instances might not be available. Try after sometime

[@arnavkumar9343]

same problem

[@클래식클래식-x1r]

same problem22

[@aomo5293]

I have tested and it works;

[@hemantrawat414]

same problem i need solution fast please help me

[@okanaltunbascom]

Thanks bro! How can we get follower and following list ?

[@ben-kn8bt]

is there a max amount of tweets we could pull using this model? thank you for such an informative video!

[@KothaEd]

Have tried for 100 as maximum and it fetched 100 tweets for me

[@DavidDiaz-sb4pc]

Hello, how are you, is it still working? It gives me an error that does not allow me to connect to the instances and does not allow me to scrape any tweets? Might you help me ?

[@KothaEd]

I just checked now. It’s working fine. Please try now

[@_daryyl]

how do you fetch tweets with certain keywords on it from a specific time frame? awesome vid btw!

[@KothaEd]

Just scrape all the tweets from Python Library. Later apply filter on tweet time to get tweets from a specific time.

[@王家幸-y2k]

thx a lot bro, can this package scrape Chinese hashtags or terms? I've tried it on my own, it seems the URL in the Nitter doesn't match....

[@KothaEd]

I tried to scrape Chinese Hashtags and Terms but Nitter is returning with Empty results.

[@YTDiscovery]

so this means this package can not scrape chinese?@@KothaEd

[@busracelik1165]

As far as i know elon musk doesn't give permission to extract data from twitter is this the new way of getting limitless data ?

[@rraviteja]

Hi bro thanks for this amazing video. Can you please tell is it possible to get list of latest 10 hashtags running

[@Giggaoggy]

Hello.. Your video is great!❤️... I have a question... How can I analyse/classify comments of any post of twitter just by pasting the link of that post?... Can you help me??

[@ogr2spel]

Is it still working, please? I've tried today, but there is an error:(

[@nathanlms5441]

Hi, does this still work? I'm currently trying to use get_tweets and it works when I don't specify the mode, but if I add mode ='user' it says "Error fetching" Thank you for the tutoriel !

[@ruslanakhmetzyanov5503]

I have "error fetching" too. Have you solved this issue?

[@KothaEd]

I am able to fetch tweets with mode = 'user'. Try checking now

[@nathanlms5441]

@@KothaEd I still have the same issue right now... but thanks !

[@alyaalsalamah2151]

Hi nathan, you have the basic package ? because I tried to not use web scrapping and it doesn’t work it says I have limited access while I am on API V2

[@nathanlms5441]

@@alyaalsalamah2151 I don't know, I have the version 0.3.5, is it the good one ?

[@AnushriDeshmukh-h9t]

I want to extract tweets of #obesity of last two years for my research paper, so how to do that?

[@nayelizapataalvarez84]

Can I limit the country where the hastag comes from? What is the tweet limit?

[@KothaEd]

I highly doubt restricting tweets based on country but you can restrict tweets limit by passing number argument

[@nsaiteja94]

it is limited to 20 tweets how to fetch more tweets?

[@3ux1n3]

i need to scrape followers list is there a way ?

[@KothaEd]

This feature is not yet available

[@Caca-wx1yd]

Hai KothaEd thank u for the vid, i need to scraping data from replies (text, retweet, like, & quote) on a someone thread, can u make a tutorial for that? i cannt find anywhere, its for my academic thesis sentiment analysis.Tthanks before, have a good day!

[@大乐在行动]

Thanks for sharing! Why the creation of Nitter object requires testing instances please?

[@KothaEd]

Instances are the actual scrapers to get tweets from twitter

[@poppyspuzzledlife]

ntscrapperr is not working

[@imk9633]

Pip install

[@wilgow1733]

Hello bro, cool project. I have thesis project to make crime map from geolocation twitter. Can i scrapping geolocation (longitude latitude) from twitter ???

[@KothaEd]

Ntscraper python doesn’t have functionality to filter tweets based on geolocation

[@wilgow1733]

ahh i see, do you know any method to get geolocation??@@KothaEd

[@fatmahalafari4212]

the same goes for me, I need to fetch the location coordinates of the tweets. Have you been able to figure out how?

[@mern_stack_dev]

Is it possible to fetch all the tweets of user

[@KothaEd]

Yes it is possible

[@justsimple2223]

As for my project, I need to find users by the intersection of hashtags like Python, writer, AI, and study. But idk how to get the same results. Please reply as soon as possible if you know something.

[@Lyacera]

Hello bro, can i get long lat scrapping from twitter ?

[@KothaEd]

Sorry long lat ??

[@Lyacera]

longitude and latitude@@KothaEd

[@asmajguirim6407]

it aint giving me large dataset maximun 30

[@dad_joke_depot]

how can i get the trending hashtags by this ??

[@KothaEd]

At the moment, trending hashtags feature is not available.

[@arnavkumar9343]

Is it possible to scrape tweets with multiple terms?

[@KothaEd]

Yes it is possible. Add all search terms in to an array and pass it in place of search term

[@justsimple2223]

@KothaEd, It only works for 2 search terms. like - scraper.get_tweets(["python", "writer"])

[@ywueeee]

how to keep waiting for a new tweet from a particular user and print in real-time?

[@KothaEd]

No scraper will provide live feed. You have to check for new tweets by a specific time frame

[@twitchzone3786]

Is there anyway to scraper twitter profile followers?

[@KothaEd]

The mentioned python library helps you to fetch profile data like No of followers and No of tweets. Are you looking for list of followers ?

[@twitchzone3786]

@@KothaEd yes list of followers

[@MaiYoulian]

is there a way I can scrape only posts with more than 1000 likes? I can choose to scrape all and then filter but this would be super ineffecient as there will be alot of redundant tweets from bots. Thank you!

[@KothaEd]

I’m working on it

[@avataraang3334]

mine are returning empty 😞

[@rubenagurcia906]

it isn't posible to use nitter anymore

[@hemantrawat414]

Another vedio for insta and Facebook account

[@KothaEd]

Sure

[@ruslanakhmetzyanov5503]

Does this thing still work?

[@KothaEd]

Yes it is still working.

[@ShekharMeena-e8x]

once i execute it , it obtains the latest tweets for a user but when i again re execute the code for scraping the for same user, it doesn;t show the latest tweets, it shows same previous tweets

[@memesbyanony]

thank u sir, can u show how to do same with restriction of specific country's tweet data ?

[@KothaEd]

At current moment, there is functionality in Ntscraper python library to restrict tweets based on a location

[@khaledhamdy9944]

how scrap number of views and retweet ... thanks

[@memesbyanony]

how can i get location of tweets

[@TheBazig]

Hi sir kindly tell how to scrap fast food chain like kfc and Macdonald's customer review data from Twitter and facebook using python for educational purpose. I shall be so thankful to you. Regards

[@KothaEd]

Ok Sure

[@FLASHGAMINGBGMI]

its not working bro

[@VenkataSatyaSrivamsiChillarige]

It's saying high retries detected, I tried many times

[@KothaEd]

It happens. Please do try it after sometime

[@MichealBeejNetwork]

only user is working upto 10 tweets its working beyond that it is not working.. outdated.

[@anushawaseemkhan5016]

how can i fetch tweets only in urdu?

[@BrainwaveSafari]

can you provide that file?

[@KothaEd]

Code link updated in Description. pls check

[@pedropablocalderon5012]

@@KothaEd I can't find it

[@KothaEd]

Here it is - drive.google.com/file/d/1a2bI9_PcYF8oN_hoDxOh1JCx1j1Tu-v9/view?usp=sharing

[@NanthaGopal-vj2wq]

ntscraper is not working lol

[@Varadi6207]

Hello bro, plz help in scraping email id