Twitter Data Pipeline using Airflow for Beginners | Data Engineering Project

[@DarshilParmar]

EDIT: Elon Musk ruined the API part, you can use some other APIs Here's one of the subscriber built the same project using other APIs - chenmeiqiao.notion.site/How-I-Reach-Out-to-KZbinrs-I-Like-As-A-Data-Engineer-c37bdddefde54c3789229ffa5a789432 or you can use a static dataset from Kaggle and then use Airflow to process it www.kaggle.com/datasets/mmmarchetti/tweets-dataset FAQ: 1. Twitter removed free access: It still has free access but with limits of requests you can make - developer.twitter.com/en/docs/twitter-api/getting-started/about-twitter-api 2. You need to request for V2 access: You will get an error but just by reading the doc you should be able to access Elevated Access Worked very hard for this project🤞 Don’t forget to hit the like button and if you want to support my work you can join channel membership, it's only 59rs per month so that I can keep these content for free

[@mohammadtanvirhossain5576]

already did !!! appriciate your efforts brother !!

[@akshunjain968]

Hey! Are you using Elevated access level because Essential doesn't work with the code you are using

[@nishantagarwal8016]

Amazing Darshil!! I was able to replicate the same in GCP

[@DarshilParmar]

@@nishantagarwal8016 Awesome! Create linkedin post and tag me

[@DarshilParmar]

@@Soulfulreader786 There is a free version also

[@sumantaghosh9299]

this is what is called content and actual do something for community ..rather than just doing faltu things...thank u so much Darshil ❤️

[@DarshilParmar]

Thank you Sumanta

[@abhishek.chakraborty]

_"faltu things"_ 😅

[@krishnachaitanya8088]

Faulty as taking other people interviews 😂😂

[@Vilayat_Khan]

​@@DarshilParmarredo the project with Twitter api!!!!!! now!!!!!!

[@ANKITKUMAR-nv8ur]

In data engineering community you are the GEM. Thanks from the bottom of my heart Darshil. Keep growing keep inspiring.

[@AshisRaj]

Mr. Darshil you have saved a lot of time for us by this video. This video has more content than others talkative videos of other youtubers. Thanku so much

[@hirenparkar7480]

Attention ❗❗❗❗❗❗❗ Anyone trying to do this project please check this imp note Twitter has made changes to its API access levels, and some v1.1 endpoints are no longer freely available. so that's why I wasn't able to do this project. Thankyou Darshil bro for this wonderful project, even if I wasn't able to execute this project. I was able to understand about the project. Thanks again for helping all of us in this Big Data Engineering journey

[@DarshilParmar]

Yes, I have added this in pinned comment and you can use other dataset for this

[@subhashkamale386]

So I have some doubt since the apis from Twitter were no longer available to use freely , in order to do similar kind of project which application we can use to get apis..

[@ThankGod143]

I came here seeing your LinkedIn post with comments,Being a fresher I was struggling for a project. This is Absolutely what I was looking for. Just Awsome. Thank you so much for your time and efforts.

[@kanduladivya5804]

This video is worth more than paid content on many websites.Thanks for such content.

[@swapneil5549]

Twitter stopped free access to their APIs :|

[@venkatah9847]

My sincere thanks to your efforts for contributing these kind of real time scenario based use case examples. 🙏

[@faramarzzrn]

one of the best tutorials I've ever seen on KZbin, a real-world example that was really interesting

[@Lady.2411]

Darshillllllll this is Gold. I literally love you rn 😭

[@dele_data]

I find it worthy to join a youtube channel membership for the first time. I hope with smart and harwork from myself and your help, I am able to land a Data Engineering job soon.

[@rohitpandey9920]

Bro I am unable to get access to Twitter API v2. Could you please help me out with that part?

[@akashsoni7925]

Those who are stuck in the api part can look for some other apis and perform ETL on it too and make your own project. Darshil helped us to understand the concepts therefore its not mandatory to use only Twitter api. Thanks a lot Darshil :)

[@snehalsylasmalladi9320]

What other APIs are a good place to start other than the Twitter/X API used in the video?

[@akashsoni7925]

@@snehalsylasmalladi9320 you can search for rapid api or search for open source apis

[@lylabata6813]

I have been following u for a long time now I don't even look at others videos......u r just great ....keep up the great work

[@DarshilParmar]

Thank you, I will

[@hritikapal683]

Hey thanks man I've been eagerly waiting for this. Would now explore and learn!

[@DarshilParmar]

Thank you Hritika for always supporting

[@tassamoo]

Thanks sir, I learn a lot following this project guide. I understand more about how to use tweepy, AWS services and airflow by following this tutorial. I hope you always have a nice day.

[@DarshilParmar]

You are welcome, check out more projects available on channel

[@shubairabbas5480]

Hi darshil you are doing a great job,have to request you to create some projects around Databricks and synapse in Azure. We all are waiting with desperation. Likewise AWS,Azure is also hitting popularity and companies ask more for Azure Data Engineers. I know you are very busy creating other playlists. But it's a humble request to create only 2-3 Data pipeline projects in Azure. Those who wants #Azure#Data Engineering. Please hit like so it came at top and Darshil got to know. Super excited for that series🌟⭐

[@prabhatgupta6415]

Yes I see azure in demand.

[@shubairabbas5480]

@@prabhatgupta6415 Yes prabhat in early level of carrer from 1-5 Yrs of experience, Azure is famous.

[@nomadicengineer22]

Please keep on bringing content like this it will be really helpful for freshers who want to understand, gain knowledge and shape their career in Data engineering🙏.

[@DarshilParmar]

I will try my best

[@ramgurung7752]

Thanks!

[@DarshilParmar]

Thank you for supporting my work

[@sujeetnair1748]

i m elder than u.. but u r an inspiration bro... i m new to data engineering..

[@aaravkumarsingh4018]

Thanku darshil for making this project, please make this type of project on regular basis so that it will help us to enhance our skill

[@DarshilParmar]

Thank you Aarav, I’ll try my best

[@chandrakantsingh9568]

Thanks for creating this project, with the help of this i schedule my data extraction task ,currently i am using solcast API to fetch weather data .

[@sagarnevase6923]

Really good video to understand how airflow, AWS and python use in ETL process. Nice man.

[@aritra1414]

Please take out time and create deeper conceptual projects like this. If possible please provide real time scenario based projects. We will be really interested to get further depth. This one is awesome!!

[@laminann1958]

This video really help me to build my first data pipeline with a different API, it is really helpful. Thank you Darshil.

[@owennowakowski7558]

This was a very helpful video for me. I have spent the last year learning web development, and I am expanding my skills to include ETL. This video gave me a great overview of the process.

[@abegpatel5300]

Thank you👍 for the awesome content We want more data engineering real world end to end project with real time streaming to deal with real world problem and situation.

[@DarshilParmar]

I will try my best

[@abdulrazakshaik1128]

Really Appreciate your efforts to the Data Community you are doing a great job. Thank you ❤️

[@DarshilParmar]

You are welcome

[@PrincessNaveena]

Wowww., Wonderful explanation.., never before ever after... True master for Big Data ,🙂Darshil

[@DarshilParmar]

Thank you so much 😀

[@anshujaiswal5622]

Really like your approach towards Learning Data Engineering concepts by doing project stuff. A good short smart project to learn basic working of a Data-pipeline using Python, AWS services and Beginner usage of Airflow. Thanks Darshil :)

[@payday_1150]

Please keep making more videos like this!

[@DarshilParmar]

I will try my best

[@anandahs6078]

Wish i found you earlier. i am learning a lot from you 🙂, recommended your channel to all my colleagues that are in data field

[@DarshilParmar]

Awesome, thank you!

[@apoorvpandkar]

Real content is finally here ❤ Loved the tutorial.

[@rohitpandey9920]

Bro I am unable to get access to Twitter API v2. Could you please help me out with that part?

[@kanui3618]

this is awsome ! can't wait for the next data engineering projects, darshil🔥 greetings from indonesia

[@moeketsekuenane324]

Darshil is just amazing 😊 . Good content! Ah Whatever is happening to the twitter API doesn’t really matter 😂

[@TehWaver]

Hey Darshil, thank you so much for this. Of course I got errors but as you said in the beginning, those are part of the journey. Definitely going to follow other tutorials from you as well regarding Airflow and AWS!

[@raghavaraosunkanapally7419]

At first, I tried on AWS Ubuntu t2.micro to run airflow - Gave error(Didnot execute Airflow Standalone) But Later, I tried t3.medium, It worked well. Thanks for the video!

[@DarshilParmar]

Awesome, congratulations

[@raghavaraosunkanapally7419]

@@DarshilParmar Can u please also create a detail project video that includes Kinesis stream, EMR (PySpark) and Lake Formation or DynomoDB, Please!

[@marchanselthomas]

I've been breaking my head over it. T2.micro is not supporting right?

[@Neji7373]

How much did it cost you when you used t3.medium?

[@way2prosperity]

I guess, it is because airflow requires min 4GB memory to run, t2.micro is not sufficient

[@shahriarentertainer4749]

This is so impessive. Keep up your good work. I wish for a true success in your life

[@gabrielfds3461]

Darshil, amazing! I do not have words to say thank yoU!

[@kundan8618]

This video is very helpful. Keep up the good work Darshil.

[@aaosoto]

Keep up the good work! Your project is the best. Greetings from Chile 😁

[@turquoisetravels]

Great videos, Darshil! Also a side note, I often watch KZbin videos at 1.5. Yours feel faster even at a normal pace. :)

[@DarshilParmar]

My natural talking pace is faster, many people complained about it but I can’t help it

[@Cosmos1987]

Now with the arrival of ChatGPT, even though one can generate CODE without analyzing all these, it is always better to understand the logic, nuance and intricacies of CODING. This vid helps a lot in deconstructing this step-by-step.👌

[@Duniya_Daari]

I watch u everyday ✨ and working on my skills to became a Data engineer 🙂

[@harshalrana1336]

This is really Amazing Darshil. I would also like to see Architecture level videos and how all tools all integrated into the cluster.

[@mrcrblr850]

Amazing Video Darshil, i really appreciate your video, you are a great professor!! Thank you very much keep doing this great content!!

[@bilalhassankhan221]

This is what exactly whay I'm looking for! Thanks brother.

[@marchanselthomas]

enjoyed the process! took me more than 12 hours to do this lol with all the trouble shooting and error fixing!

[@avni_103]

can u suggest me how error on " from airflow import dag" error was solved on vscode ??????

[@manuprakash1989]

Great! Got broader understanding of api, airflow and aws. Thank you!

[@yhsreddy]

Awesome demonstration, truly appreciated l, waiting to see lot more soon

[@amirazad1762]

Great and useful video. Thanks Darshil 🙏

[@avshekraj]

Amazing...got to know something intresting. Thanks for the detail explanation.

[@ogissgi7441]

Thank you! thank you! thank you!

[@raimanisha5197]

Hi Darshil, Thank you for this .It was a great learning experience and it was fun too ! 😛.I am eagerly waiting for more such videos on airflow

[@jashanbansal2613]

Great Video Darshil. Keep up the Good Work :)

[@keyursolanki]

Love the way you have explained things. Loads of success to you😇

[@_strollingaroundtheworld_]

Appreciate the content. Cleared the basics by watching it.

[@DarshilParmar]

What about execution?

[@segnimahmoud5898]

@@DarshilParmar Not working

[@eternal2187]

You always continue to be great asset in data engineering content, thanks again!

[@gopi10aug]

I really appreciate the effort and simplicity..! I was looking for actual content like this ... Thank you Darshil 😇

[@DarshilParmar]

You are welcome

[@partabparmar5537]

THANK YOU SO MUCH FOR SUCHA QUALITY CONTENT ......GOD BLESS YOU

[@gimmestonks5333]

Thanks for teaching basics of air flow and Dags..

[@abhijeetpradhan143]

Best content ever..👍 So nice of you.. Really want more such Projects..

[@DarshilParmar]

I will try my best

[@shaikusman536]

Awsome brother...Loved the way to teach...Hoping for detailed projects.....thanks a ton brother.....

[@mirli33]

As usual very nice content and appreciate your effort. If you could make a video with OOPs approach and include some test cases that will be really great.

[@namvu607]

Super practical, valuable tutorial. Thank you for sharing, sir.

[@reypaulobae4895]

Thank you so much! I encountered some issues along the way, but was able to resolve them!

[@rohitpandey9920]

Bro I am unable to get access to Twitter API v2. Could you please help me out with that part?

[@reypaulobae4895]

@@rohitpandey9920 Yeah, just delete the default and create a new one. Worked after

[@rembautimes8808]

Thanks - very comprehensive tutorial

[@mohsinrajpoot3780]

great work you helping lot of people.

[@ashishveer4591]

Perfect Explanation .. keep it up .. thank you 🙌

[@SaiyedOwaisAhmed]

Mentor of many data engineer ❤

[@tejasborkar5917]

Great darshil this is really helpful video Thank you.. keep uploading content like this please

[@DarshilParmar]

Thank you Tejas for always supporting

[@AjaySharma-jv6qn]

thanks for sharing info - A step by step guide

[@realSrvBhtngr]

great content Thanks Darshil bhai, very cool

[@KK-rh6cd]

Very informative and practical video. Thank you for making a video 😊. Waiting for advanced level video on airflow.

[@shravanikadam6700]

were you able to extract data in csv??

[@Laikacc]

Very crystal clear 🔮 explanation 🎉

[@rohitpandey9920]

Bro I am unable to get access to Twitter API v2. Could you please help me out with that part?

[@iamdare]

Good job mate. I thoroughly enjoy the video. Thank you.

[@jitendraparihar56]

Eagerly waiting for this project. Thank you darshil for such amazing projects...

[@DarshilParmar]

You are welcome

[@lokeswarnaidu2398]

Please do a complete project based on streaming data and transform the data using spark and orchestrate it using airflow and then store it cloud or any database.

[@chaiseee1642]

Very good tutorial, excellent 😊

[@jeeteshkumar11]

Great tutorial, appreciate it, keep up the good work brother. :)

[@mikitaarabei]

Thanks for this content, Darshil :)

[@karunakaranr2473]

Thank you for your time and effort to make these videos. Really helpful.

[@santoshchavan1929]

Awesome sir...... That's what I actually want...

[@mangeshgawali7683]

Great explanation. Thanks, Darshil! 😊

[@DarshilParmar]

Glad you liked it!

[@bharathsankaranarayanan]

Did something change with the API? I get a 403 forbidden error when I try to run the tweepy.user_timeline function.

[@kirankv1956]

Really love the content darshil. Very precise explanation.

[@sufiyankhan726]

Seriously thanks dudeee❤

[@africarising360]

Your video was really helpful. Thank you very much

[@csgs13]

Very clean and understandable 🎉

[@pradeepdhanola1327]

You are best bhai ❤️

[@CaptainRabbit-babii]

i had an issue Forbidden: 403 Forbidden 453 - You currently have access to a subset of Twitter API v2 endpoints and limited v1.1 endpoints (e.g. media post, oauth) only. If you need access to this endpoint, you may need a different access level. You can learn more here: when I tried to print(tweets) at 14:27 of this video I used a free account, so I just had 1 environment, do you know if this is the reason for the error? thanks

[@ahmedsaied8373]

The same happened with me yestarday, I guess the Twitter team changed the API plan privileges where we can't pull/get tweets data on the basic/free plan

[@gaganaulakh41]

Even I am getting the same error. Was it resolved for you?

[@karangupta_DE]

this is perfect, thank you Darshil

[@DarshilParmar]

Thank you Karan

[@khushahmed97]

Great explanation! Will try to replicate in GCP

[@shubhampatil-xx5qq]

Thank you soo much darshil sir❤

[@mehmetkaya4330]

Thank you for another great tutorial!!

[@avnishgupta1903]

Awesome Work !!!

[@dhruvingandhi1114]

Great efforts!!!

[@vidyanishok7735]

Thanks and keep doing more... Please also make python part 2 ..

[@ini_affiah]

Thank you for the video. It was helpful.

[@rolinejohnaguilar5272]

cool man, just what I needed