How to Extract Data from Twitter using Python [Tweepy API]

[@analyticswithadam]

Here are 4k tweets I pulled last year on the world cup refereeing if you need something for sentiment analysis now the API is shut down. There are also labelled. github.com/analyticswithadam/Python/blob/main/REF%20TWEETS.csv

[@danielmartin3153]

Hi Adam, this tutorial was very helpful, however, after making the API call, i got an error stating that i do not have the right level of access. This is after succesfully paying for twitter basic API and changing the access keys. Can you advice on how to correctly set up the API after making the subcription,

[@raulgarridogarcia512]

@@danielmartin3153 hey I had the same issue, were you able to figure it out?

[@analyticswithadam]

Hi @@danielmartin3153, I think someone else in comments had the same issue. There is a guide on how to use the official Twitter API V2 here developer.twitter.com/en/docs/twitter-api/tweets/search/introduction. With a few options to get started, Postman, Python, etc. See if you get it sorted from there. I haven't paid so haven't queried in a good while

[@njooda6253]

@@danielmartin3153 hi did you solved this issue I got the same.

[@droptimistic7419]

@@analyticswithadam can I use the same method now in 2024? and is your account the free one or the Basic paid ? if you can reply I will really appreciate your help. I'm stuck

[@ThanasisIoann]

You are explaining everything very well. And in combination with your simple, yet effective code I found what I was looking for. Thanks!

[@analyticswithadam]

Glad it helped!

[@analyticswithadam]

Hi All, Twitter and Reddit APIs has been cut off unless you want to pay a lot. KZbin API is still good, check out this video on how to pull comments from that kzbin.info/www/bejne/iXrQY4pvidmtiJI

[@letconsultingpartner]

Yeah :/ Thanks for your effort! Is there a way we can extract data from Threads?

[@analyticswithadam]

@@letconsultingpartner Cant even get access to threads in Ireland 😵

[@letconsultingpartner]

@@analyticswithadam 🙉Lol It was worth asking😵‍💫

[@gulabjamun247]

The search_tweets attribute /method of tweepy library works only if you have full API v2 and not free as it require high access level, So you have to pay to Twitter for high level API key.

[@mitchsammy4235]

Is there another way to pull tweets from twitter without paying?

[@iiicurry30]

Is it possible to pull tweets if I only subscribe the basic version of v2?

[@gulabjamun247]

@@iiicurry30 No, you need to buy.

[@migueljaramillorosado7285]

Is it there any way to search for tweets with the free API tier?

[@gulabjamun247]

@@migueljaramillorosado7285 You can try using a different library called as Selenium instead of Tweepy

[@mattfromblock]

Amazing video mate, earned yourself a new sub - such a shame about elon screwing us with the API, subbed with notifications on for the next one!

[@DominicNgugi-h7w]

love it!!, great work over here🙌🙌🙌🙌

[@adrianramos9947]

el nivel de acceso actual de la API free de twitter no da acceso a extraer la informacion de los tweets de la manera que has propuesto

[@RAHULYADAV-po1kc]

getting error- Status Failed On, 403 Forbidden 453 - You currently have access to a subset of Twitter API v2 endpoints and limited v1.1 endpoints (e.g. media post, oauth) only. If you need access to this endpoint, you may need a different access level.

[@MR_manti765]

did you fix it? i am getting the same error

[@ВадимПапуша-г2щ]

did u fix?

[@charlsiii2349]

I think thats because they changed the api when elon took over… if anyone has new solutions please let me know

[@zakariaelaaliji7747]

I have tried this code, but it seems that I'm using the free access level, which doesn't provide the access to the search tweets feature, otherwise, I have to move to the basic access level to apply this code, right ?

[@lamis_18]

same question here !

[@CherryAimz]

It seems like this code is not working anymore. I found that only way to do 'search query' by using Twitter API v2 and you have to access basic levels or higher that mean you have to pay at least $100 / mo.

[@zakariaelaaliji7747]

@@CherryAimzYess I think that's the point, otherwise, Reddit api is better when it comes to retrieve data from posts...etc

[@aparnagarg6390]

​@@zakariaelaaliji7747 can u please provide the reddit api

[@nehagaryali6122]

Can you post the updated video, if any? and share the video link? When can you do so? 453

[@dawnthomas6769]

The API is not allowing to pull the tweets now. I'm on the free plan. Any other options?

[@A-Jaradat-Z]

i have the 100$ sub and cant pull tweets !!!!!!

[@kwamenadadson]

@@A-Jaradat-Z same man

[@hibaammari80]

any updates ?

[@digitaldftns]

@@A-Jaradat-ZDid you solve the rpoblem?

[@larrybrown7632]

@jessehuang3046 Yeah, $100 for just playing around and testing, or just seeing what the API can do, seems like a lot of money out of pocket, to just see what it can do. Good Job ELON... Moron.

[@saratayyab4674]

I am getting this error: “Status Failed On, 401 Unauthorized 32 - could not authenticate you.” What can I do to solve this error

[@koya-_-]

did u find a solution to it? stuck rn :')

[@ksong5589]

same issue

[@lhit6795]

i am getting this: Status Failed On, 403 Forbidden 453 - You currently have access to a subset of Twitter API v2 endpoints and limited v1.1 endpoints (e.g. media post, oauth) only. If you need access to this endpoint, you may need a different access level.

[@gabrielmmenz]

same. I guess that's due to the API authentication key that he tries to use at the code... trying to find an answer within the documentation rn :P

[@supersuper-ir2ph]

@@gabrielmmenz any luck?

[@kabahaou1019]

Any update on this problem??@@gabrielmmenz

[@RAHULYADAV-po1kc]

@@gabrielmmenz I'm getting the same error, any luck with you ?

[@luckykittybro]

I also meet this issue. I guess Basic Version has no access for that?

[@wildansdalh6477]

Failed on, 403 Forbidden 453 - you currently have acces to a subset of Twitter API V2 Endpoint and limited V1.1 endpoints Should I apply for not free acces ?

[@codydrolet5595]

Getting the same error has anyone figured out the work around for this? I saw something about using a client instead of an Auth but getting no luck.

[@raghavendraniteeshg6198]

I think twitter changed there v1.1 access level, even I am facing the same issue

[@SHAOBOCHEN-yb5je]

@@raghavendraniteeshg6198 have you solve this problem? my problem is "401 Unauthorized"。 i have brought Basic level i also working my data analyst project。 could we exchange some idea about Twitter API?

[@soofishafiya72]

facing the same issue, any other way to extract tweets?

[@Fenix473]

Really waiting for this video remaster. Error 453))

[@Matt-lf1ik]

Same, any fix?

[@vision674]

Is this only for pulling the tweets from the time you made the video or I can extract latest tweets like real time tweets?

[@analyticswithadam]

It’s still possible but there is a charge now

[@DDissanayake]

very helpful, how do you get the user location? (to another column) , searching for a similar extraction is for my MSc project. many thanks

[@analyticswithadam]

If you see the code in github ive just pulled in 5 fields from the many possible you just pull another attribute tweet.location it did look like a lot of the locations were blank though from the full tweets.

[@johnnytripto5630]

hi, how did you authorize your account? it shows me an error that my account is unauthorized.

[@ragikrishnar7426]

Me too. Is there any other option to do this?

[@brumpmk]

Thanks Adam, really useful. I'm getting this error auth = tweepy.OAuthlUserHandler( AttributeError: module 'tweepy' has no attribute 'OAuthlUserHandler'. Did you mean: 'OAuthHandler'?

[@kshitizagarwal5012]

it's OAuth1UserHandler (1 instead of l) you can copy paste the below code auth = tweepy.OAuth1UserHandler(consumer_key, consumer_secret, access_token, access_token_secret) api = tweepy.API(auth)

[@dangerzone636]

it gives the error like ,can you update the code Status Failed On, 403 Forbidden 453 - You currently have access to a subset of Twitter API v2 endpoints and limited v1.1 endpoints (e.g. media post, oauth) only. If you need access to this endpoint, you may need a different access level.

[@TeraBaap-m6j]

Us i was writing my own code and it throwed this error i came on yt for help and you also हडबडी the same problem 😂

[@TeraBaap-m6j]

Us i was writing my own code and it throwed this error i came on yt for help and you also हडबडी the same problem 😂

[@markjsheehan9952]

Hi, newbie question here but does it matter what text editor you use to do this?

[@analyticswithadam]

Yes, you need to use one specifically for Python, easiest to get started is collab which is on Google drive as there is no setup..

[@noumanahmad308]

Amazing and thank you so much for sharing!

[@yasmineady2737]

Thank you for this video. I will try the Basic subscribe to see if this works. I saw a few in the comment sections with issues. I can try to find out why the error occurs.

[@analyticswithadam]

🤞

[@alatheaposey5858]

Let me know if this works for you/you figure out the issue please! I'm waiting for a grant from my school to get the basic plan, hoping not to run into anything too troubling

[@ogspeace]

it really was amazing when it lasted. thanks still.

[@TyrantJnrCODM]

wait so you cant use twitter API anymore?

[@hassansyed2552]

I bought a basic subscription and I am still not able to fetch tweets via tweepy, please help. Please help if we still need to use the four keys or just the bearer token key is enough.

[@analyticswithadam]

There is the Twitter API V2... I reckon you might need to go directly through that here is a code example.. Looks like the bearer token is enough.. github.com/twitterdev/Twitter-API-v2-sample-code/blob/main/Tweet-Lookup/get_tweets_with_bearer_token.py

[@larrybrown7632]

@@analyticswithadam But you still can't do anything on the Free subscription, right?

[@HitAndMissLab]

I created a developers account for myself, but those keys that you are showing stay hidden. I can not copy them or access any of them except bearer's key. Any suggestions?

[@emirozer9280]

hey, is this tutorial up to date? Because twitter API page says that free tier can only post not read twitts.

[@analyticswithadam]

The functionality can been pulled since for free tier

[@harish77720]

@@analyticswithadam Hi Adam, may I request an updated video on getting tweet data please? ? I believe with the free tier for X API v2, we get to read 1,500 Posts/month.

[@analyticswithadam]

@@harish77720 From what I see here we still cant pull tweets on the free tier developer.twitter.com/en/docs/twitter-api/getting-started/about-twitter-api

[@larrybrown7632]

@@harish77720 Yes please do another video only using the Free Tier, then I'll subscribe, not before.

[@sdljfgsdfg]

do you have the Basic tier or is this with the free tier?

[@analyticswithadam]

This was recorded before the tiers

[@ww1593]

Musk screwed everyone over by making this basic thing $100 per month

[@anshsinghal2679]

@@analyticswithadam do you know what kind of queries work with tweepy now on the free tier}?

[@TyrantJnrCODM]

@@analyticswithadam I'm having problems with the new tiers. i keep on getting this error: 453 - You currently have access to a subset of Twitter API v2 endpoints and limited v1.1 endpoints (e.g. media post, oauth) only. If you need access to this endpoint, you may need a different access level. I can't seem to find a solution

[@daniellepompeunoronhaponte8749]

hi Adam , tks by the video. I need have acces to twitters users data like follower number, location etc.. do you know if i get this in the new twitter api in the basic plan?

[@analyticswithadam]

Looks like it is in V2... Here is the documentation page. developer.twitter.com/en/docs/twitter-api/users/lookup/introduction

[@KishoreChandraShekharKandukuri]

So this will not be considerd as scraping right as we extract the data from api. Cause I have seen in several websites that it's an crime.

[@analyticswithadam]

Data through an API is not scraping . It’s only when you circumvent controls on a site

[@benjamincortez8146]

Is this method still relevant for 'X'? I'm trying to do sentiment analysis for a graduate-level research project and basically need to teach myself how to gather data from various social media in order to appropriately complete my thesis

[@analyticswithadam]

Everything was changed about the api with x . It introduce a need to pay.

[@benjamincortez8146]

@@analyticswithadam​​⁠are there any programs/tools you’d recommend for completing such a task? Preferably free, but if I need to pay or learn some python that’s fine I suppose. I just need to be able to input a date-range, # of posts, content-topic (ex. Natural disasters) and platform (Facebook, X, Instagram) and essentially do a “mass” sentiment analysis: Produced in a way I could potentially chart on excel or at the very least be able to gauge sentiment for that range and the parameters If there aren’t any programs available, what places do you recommend I look to at least attempt to learn how to come up with a way of making this easier rather than analyzing one by one. Idk if that explanation makes sense. Also thank you for letting me know the API has changed!

[@analyticswithadam]

@@benjamincortez8146 KZbin Comments are free, I have a few videos on how to pull those in Python or have one on trust pilot review scraping in Python.

[@maximilianh.1660]

​@@benjamincortez8146 Hey there! Interesting to find someone with the same problem. :D I am at the same point and looking for ways to do a "mass" sentiment analysis on social media posts. But with another topic at focus (crypto currency markets). Maybe we can have a talk about our findings so far. Wish you the best of luck for your thesis!

[@spacemat1329]

hello, i have problem with error: Status Failed On, 404 Not Found 34 - Sorry, that page does not exist is it depending on using vs code instead jupyter?

[@analyticswithadam]

I don't think so, Twitter changed the rules recently so possibly the code is out of date.

[@iannjagi5349]

hey Adam! Trying to use the twitter API for this an analysis but each time i put in my Api credentials my intellisense marks it as an unknown. I am using vs code. What should I do? or what could be the issue

[@analyticswithadam]

Try to copy and paste the code in to Google Colab (Free editor on Google Drive) and see if that works. Not to familiar with Python in VS Code. I normally just go Jupyter Lab or Colab

[@lazytv363]

code does not work I added my key and name 'consumer_key' is not defined

[@analyticswithadam]

Sorry about that it’s an old video they restructured the API

[@colombian92317]

hi adam, ive search endlessly on youtube on how I can set up an api to access twitter ads data. I signed up for a X developer account and i have been granted access to the twitter ads api account. I am not sure how I can connect the 2? so I can use python to extract analytics from twitter ads api. Can you please help. this is for my data analyst project.

[@SHAOBOCHEN-yb5je]

have you solve this problem? my problem is "401 Unauthorized"。 i have brought Basic level i also working my data analyst project。 could we exchange some idea about Twitter API?

[@thanvimannem2866]

I'm getting forbidden error

[@elijahekpenyong911]

Same here...wat to do?

[@khairulumam3953]

is there a documentation about how to write the query? I'm looking for it on tweepy docs but I can't found it until now

[@sandeepprajapati6894]

thank you sir, please sir make video retrieve likes count and retweet count use post id

[@parisaemkani5730]

hi Adam. I need to get tweeter API for sentiment analysis for ONE MONTH to track tech companies' stock market. but as I know API needs to charge around 100 $. Does this tutorial work for that or not?

[@icemixpl]

same problem :/ can't start my master thesis :(

[@icemixpl]

perhaps selenium and scraping would be better?

[@BlackIMW3IStorm]

Have you found another way of using twitter data? ​@@icemixpl also need some tool to start my master thesis

[@larrybrown7632]

@@icemixpl That's a pain in the a s s. Elon is really screwing over the actual people that can help him. F-Him. $100 per month to play with his api. F-that.!

[@paulomacasaet4248]

up to how many tweets can you set for the query to still work? can it handle more than 1000?

[@filipecs20]

im not finding a solution foi the "403 Forbidden" error when i try to perform the query with the api....

[@vokeejenavi3780]

same with me

[@asdzxc1506]

ya no se puede extraer con la cuenta grauita 😥

[@jiaweimo-dc8oh]

hello adam, please, how to set the starting date and the end date of the tweet searched?

[@libo8318]

hey Adam, seemed you were not using VScode to code here. You mind sharing what tool you are using to code here. Looks pretty efficient . Sorry very new to all this 😂

[@analyticswithadam]

I use google colab for python, its on google drive so easy to share, you can grab the notebook from the description

[@markjsheehan9952]

I am trying to run this but I am getting an error NameError: name 'consumer_key' is not defined

[@nandusraj7801]

Under the Projects& Apps link in the left pane, click on the project that you are using , click on the Keys and Tokens Tab, next to Settings tab, generate the API Key and Secret under the Consumer Keys section.

[@Waifu72Hunter]

Can I get the location of the user from each tweet using the attributes? If yes how so.

[@analyticswithadam]

There is a location available, you just need to mention' location in the field you are looking for. API ref here: developer.twitter.com/en/docs/twitter-api/tweets/lookup/api-reference/get-tweets

[@Waifu72Hunter]

@@analyticswithadam Thanks.

[@kushlendrpalahirwar6886]

how to display top Twitter trends in PHP or Python, can you make a video on this? Also, mention whether is free or not?

[@amostarspirit]

Pls can you tell me can i recover my tweets that i accidently deleted yesterday?

[@SiddharthGoyal-iit]

how is this better than web scrapping using selenium? wont this will lead to extracting sponsored tweets as well?

[@analyticswithadam]

I apologise for making this video

[@siddharthgoyal30258]

Uhh? Is that a sarcastic reply?

[@guguskendalimutu]

I got this error. Please help: NameError Traceback (most recent call last) in () 9 #Pass in our twitter API authentication key 10 auth = tweepy.OAuth1UserHandler( ---> 11 consumer_key, consumer_secret, 12 access_token, access_token_secret 13 ) NameError: name 'consumer_key' is not defined

[@dantedt3931]

Did anyone manage to pull any tweets?I have the $100 option(Basic) but i cannot get any tweets. Error:"Status Failed On, 401 Unauthorized 32 - Could not authenticate you." I have the keys.

[@ragikrishnar7426]

I got the same error. 😢

[@sandeepsanyal9382]

your access tokens may have changed. check again.

[@saivishalkannan3136]

@@sandeepsanyal9382 same error and I checked my access tokens, and they are fine. Did you solve it? If yes, can you please help me?

[@akiramahilynaguirrelugo8000]

I get that error Status Failed On, 401 Unauthorized 32 - Could not authenticate you.

[@abdullahalmaruf3973]

I also face same problem , can you solve this problem ?

[@gaurivenkatachalapathi6834]

Hi I am doing a sentiment analysis only on twitter tweets for my dissertation and the recent Twitter updates, it returns an error. I need close to around more than a few thousand tweets and I am not sure what I can do now to complete it successfully.

[@analyticswithadam]

Looking at the recent updates its 100 a month now for 10,000 tweets on the basic package developer.twitter.com/en/portal/products/basic.. So the option is subscribe for one month or find another platform ... I recently did a video on pulling KZbin comments which you can use for sentiment analysis and im going to be posting videos on scraping from other platforms in the next few weeks. If you don't have a specific topic I did a project last year on world cup tweet sentiment analysis and can give you those tweets. There are 4k in total here is a link to the csv github.com/analyticswithadam/Python/blob/main/REF%20TWEETS.csv

[@A-Jaradat-Z]

@@analyticswithadam I have the 100$ sub and can't pull a single tweet !!!!!!

[@analyticswithadam]

@@A-Jaradat-Z See if it works with this method developer.twitter.com/en/docs/twitter-api/tweets/lookup/quick-start

[@chrisw9597]

@@A-Jaradat-ZI’ve been seeing people buy the basic, pro and enterprise plan and they couldn’t grab tweets as well. So unfortunate

[@dantedt3931]

Did you manage to pull any tweets?I have the $100 option but i cannot get any tweets.

[@navidnouroz2853]

How to do it if I want to pull not a specific number but from a specific date to another?

[@metumoshifidi6607]

Is yours working , I paid for basic but I can't get the tweets

[@nada-rw6iy]

@@metumoshifidi6607same here, did it work with you?

[@lamaalrashed3395]

Amazing video, i want to know how can I possibly get the replies of an only one tweet ? How can i write the query

[@analyticswithadam]

It’s possible but Unfortunately you need to pay to pull tweets now

[@KMABRARAHSAN]

i am getting error 401. what to do?

[@zatilismahsyahirah346]

i srsly have no idea where to reveal my API key. It only shows 6 last characters

[@asmajguirim6407]

theres an error NameError: name 'tweets_df' is not defined

[@douglas20011]

how to extract the geographic location information fo each tweets？

[@kostasgoggakhs617]

this is for Free version API?

[@dogachatun8715]

Can you make a video of the reddit version since we can't use twitter version for free

[@analyticswithadam]

REDDIT is paid too now… I made a KZbin one

[@dogachatun8715]

@@analyticswithadam damn, i have a project about financial app and i need to collect financial reviews and make summary of them. Do you have any suggestions for me to use?

[@analyticswithadam]

@@dogachatun8715 what is the app? Thinking web scraping but depends on where the reviews are held

[@analyticswithadam]

If the app is on trust pilot. I found the code to extract the reviews, just change these three things in the code. Go to their page. See how many pages of reviews and the company and change the three bits below. from_page = 1 to_page = 60 company = 'eshakti.com' github.com/analyticswithadam/Python/blob/main/Web_Scraping_from_TrustPilot.ipynb

[@dogachatun8715]

@@analyticswithadam Thank you so much! We are making an app for our senior project so we didn't decide the ap yet

[@hemoo0]

can I scrap the tweet images?

[@analyticswithadam]

Can’t scrape anything anymore unless you pay.. thanks elon

[@GamerzShit]

I'm getting error 403, anyone know how to deal with this? Do I need to pay?

[@analyticswithadam]

The free tweets have been turned off unfortunately.

[@SHAOBOCHEN-yb5je]

have you solve this problem? my problem is "401 Unauthorized"。 i have brought Basic level

[@oraclesql]

Hi Adam, it doesn't work unfortunately. Returns a 403 forbidden error from the API

[@analyticswithadam]

Hi Steve, This video was made before Elon cut the functionality off. Im going to redo it with an API thats still open.

[@oraclesql]

@@analyticswithadam that's great! Thank you

[@petrovsk]

​@@analyticswithadam i was using it before stupid elon cut it off, goddamn!!!

[@vivekpatidar1750]

@@analyticswithadam Then wtf you have uploaded this video now

[@kushkumar6467]

@@analyticswithadam Looks like 453 for me.

[@hayatt143]

I Belive is search api is no longer available in free account.

[@analyticswithadam]

You are right

[@KateAnampa]

AMAZING!!!! THANKS! 😢

[@oxt4778]

ツイートがされた期間の設定方法を教えてください

[@larrybrown7632]

He throws out this videos and doesn't even state that you have to pay for the v.2, and never replies to anyone that asks him questions. So here goes: Is there a way to do this on the FREE version? I'd rather try it for free and get it working, before I start paying $100 or more per month.?

[@analyticswithadam]

I made this video before Elon took over twitter and put the basic api behind a paywall. It’s possible to use a web scraping tool like beautiful soup but it’s not entirely legal

[@javierryly1593]

Excellent