Set up & use PubSub with Python

[@suvratrai]

Beautiful & crisp explanation with parallel demo. Thanks for making this video Ryan ! This is really helpful.

[@d-i-ry]

Hey Suvrat, thanks for that! I'm really glad you liked it!

[@Sane4ka7777]

Haha, you are such a positive person, I believe the people around you are very lucky because they always see your smile and it is energizing. You are truly an example of a happy life. Thanks for the smile

[@d-i-ry]

I've read and re-read this comment more than any other comment on any video I've ever made. I can't thank you enough for reaching out and being so kind. The only reason I haven't replied to it is because KZbin only shows me a list of unread ones, and I wanted to see this note over and over again Thanks, Ryan

[@teddyfulk]

Literally exactly what I was looking for! Thank you so much. You got yourself a subscriber.

[@d-i-ry]

thanks a lot! that's great to hear

[@techgik]

Excellent tutorial, straight to the point. You made my day. Thank you so much!

[@d-i-ry]

I’m really glad to hear

[@sebastiancalderon9559]

Beautiful explanation, thank you so much !!!!

[@d-i-ry]

I’m really glad it helped!

[@RivenOmg]

Hi Ryan, this is so helpful! Thank you!

[@d-i-ry]

Hi Jane, I'm really glad to hear that! Thanks

[@micropythonarduino9653]

Great, concise tutorial, thanks a lot!

[@d-i-ry]

Glad it was helpful!

[@olajuwonadebayo5162]

Thanks for this wonderful tutorial. Concise, clear, and easy to follow. Delivered in the calmest fashion.

[@d-i-ry]

that was really kind, thank you :)

[@ohlai]

best tutorial on yt!

[@d-i-ry]

Wow! Thanks!!

[@ElCanalMeu]

Thanks for the video Ryan, super easy to follow.

[@d-i-ry]

Welcome, and I’m glad you liked it

[@timkao2545]

Concise and easy-understanding! Nice work!

[@d-i-ry]

Thanks a lot, Tim!

[@Niyander9]

Thanks, lots of love.

[@d-i-ry]

You're welcome! I'm glad it helped :)

[@jamunv]

Hey Ryan, I was looking a simple explanation and It seemed me excellent. Thanks for the video!

[@d-i-ry]

Hi Javier, I’m glad it helped!

[@oussamaladhari550]

great explanation! thank you very much

[@d-i-ry]

Thank you!

[@thiagobarrichelo]

Awesome man thanks for sharing. This is gold!

[@d-i-ry]

Glad you enjoyed it!

[@Andres_Alba]

Thank you very much, you helped me a lot, greetings from Colombia.

[@d-i-ry]

I'm glad it helped you! And thanks, greetings from California :)

[@Victor-hc2gq]

thanks a lot, you've helped a brazilian!

[@d-i-ry]

Awesome, I’m glad! Is that more or less than a million?😹

[@jaytian341]

Crystal clear! Thanks for the video. Nice Made!~

[@d-i-ry]

Thanks a lot, Jay!

[@FabianoRomanBeraldifrb]

Great post, clear and straightforward! Thanks for sharing!

[@d-i-ry]

Glad you enjoyed it!

[@chonyy3533]

Quick and clear... Rly great tutorial!

[@d-i-ry]

Thanks Chonyy!

[@deepshah2305]

Well explained!!

[@d-i-ry]

Thanks!!

[@fabiocoutinho3328]

Thanks for this awesome tutorial.

[@d-i-ry]

Welcome :)

[@turtleyoda7703]

Thanks! Clear and easy to follow

[@d-i-ry]

Welcome :)

[@Idontreallycareatall]

Just perfect. Thank you!

[@d-i-ry]

That's really great to hear :)

[@Idontreallycareatall]

@@d-i-ry You do have really nice way of presenting things (:

[@NaziyaM-vz8yp]

thank you it’s very clear

[@d-i-ry]

Thanks, that’s really nice to hear

[@olegpopov3180]

Wow, such a clear explanation! Thank you! And the question is: how to make that stuff multithreaded?

[@d-i-ry]

Hey Oleg, sorry to respond so late! You'll need to use `futures`. The answer on this page has good code stackoverflow.com/questions/69307567/how-to-publish-multiple-messages-in-google-pub-sub-fast

[@minc33]

This helped a ton, thank you!

[@d-i-ry]

Glad it helped :)

[@jaredfiacco1107]

Thank you for this very helpful video!!!

[@d-i-ry]

Welcome, glad you liked it!

[@michelkluger]

this is an awesome tutorial, thank you!

[@d-i-ry]

Thanks! That really makes me happy to hear

[@HansCastroJimenez]

This video really help me to understand things :) thanks

[@d-i-ry]

Awesome! I’m glad it helped

[@ankitmishra5566]

Thank you man!

[@d-i-ry]

No problem!

[@takeshikovacs1081]

good one ! as for the json data, I guess you could also serialize it with json.dumps on the publisher side to convert JSON to a string and then json.loads on the subscriber?

[@d-i-ry]

Yeah, that’s probably be an easier thing to do!

[@gadeichhorn]

Awesome! thanks.

[@d-i-ry]

Glad it helped!

[@user-rb7nc6tq9v]

amazing, well explained!

[@d-i-ry]

Thanks!

[@pirucynpiru]

great!!! thank you so much

[@d-i-ry]

I’m glad it helped :)

[@srikika]

Thanks a ton! This really helped me out!

[@d-i-ry]

You're welcome :)

[@eriksm7]

Excellent explanation

[@d-i-ry]

Thanks Erik!

[@stanleyogadachinedu]

Thank you sir 🫡

[@d-i-ry]

Glad it helped!

[@anuiyer2601]

Hi Ryan Thanks for the video, this is very informative. I have a question and wondering if you could help with this. How do we set up email notifications to multiple users who subscribe to a topic? This is fairly simple on AWS, we just choose email as protocol for the Subscription and add the user's email address. How do we set this up in GCP Pub/Sub? Thanks in advance

[@d-i-ry]

Hi Anu, I found the walkthrough for you. I knew that Google Cloud doesn’t natively have a mail sender, but this link has the code for you cloud.google.com/security-command-center/docs/how-to-enable-real-time-notifications also, I’ll add this to my list of future videos, so thanks for the idea!

[@anuiyer2601]

@@d-i-ry Ryan thanks for your help and quick response

[@sabianmedia9498]

hey man! I am working on this django microservice project and I wanted to use pub/sub for my inter service communication but since my services are running on cloud run I can not use pull subscription since I can not run both celery and django in one cloud run so I was thinking it would be better to use push subscription can you please point me to some direction pls?

[@d-i-ry]

Hummmmm.. we’ll I just finished using Apache Kafka which is like pubsub on steroids. But really, if you need to push, I’d use Cloud Functions in Python (I’ve got a video on it, and it’s powerful, but you have to build everything from scratch)

[@deepGanguly98]

Thanks a lot Ryan! Very nice presentation could you also show how to work with pubsub locally using the Google SDK the tutorials. Cheers!

[@d-i-ry]

Hey Deep, thank you!! And I really like your idea - I’ve wanted to do another PubSub video, so I’ll try making that soon :)

[@santiagoruaperez7394]

Do you know how to use pub/sub using future ? Is there any way to use a subscriber without blocking the code? I mean start doing stuff and just ask when there is an actual message in the topic?

[@d-i-ry]

Hey Santiago, that's an interesting and useful question. Here's the code adding concurrency and futures :) scroll to the bottom levelup.gitconnected.com/how-to-use-google-pub-sub-to-build-an-asynchronous-messaging-system-in-python-3b43094627dc

[@iloveyou143639]

great video

[@d-i-ry]

Glad you enjoyed it

[@luisvillar8320]

thank you! everything worked

[@d-i-ry]

You're welcome!

[@harrytjokroaminoto9212]

Great tutorial .. Thanks

[@eladed7434]

best video on this topic. does the subscriber receive also RTDNs ? or is that a whole different topic ?

[@d-i-ry]

Thank you! The subscriber just gets the topic notification, no other dev notifications

[@gaidem11]

you safe my life, thanks

[@Junesumm]

HI Ryan , thanks to teach us how to use the pubsub and im success to run by using pycharm and raspberry pi but i cant success while i using cloud run

[@d-i-ry]

Hi Lee Jun Ming, I'm glad you got it running on your own machine! I haven't played with Cloud Run I actually misread your question, and wrote a long solution for Cloud Functions instead of Cloud Run. But here's something that might help you... cloud.google.com/run/docs/testing/local When I need to build a Cloud Function, I test locally first. I hope that can be of help when you build a Cloud Run environment, too. Good luck!

[@cloudlover9186]

Sir, beautiful explanation, sir can i have pub sub to have message to trigger cloud function or do i call dataflow only , please advise.

[@d-i-ry]

Thanks! Yeah you can have Cloud Functions react to PubSub. Check my video list. I have a few videos that’ll help you there :)

[@cloudlover9186]

@@d-i-ry Thankyou sir, One more question can i have video to sink cloud logging with Big query dataset (table) updates and that will publish a message in a defined topic which is subcribed and triggers cloud function

[@d-i-ry]

hi @@cloudlover9186 this one sounds a little bit intricate. i can definitely help you with the steps, but I'm a little confused. what are you hoping to do?

[@SimKieu]

Thank you so much for the video. Do you think I can have a Python server that listens to the pubsub events? Should I use Django or Flask for that? I want to host a Python server that listens to GCP Pubsub events and handle the incoming events.

[@d-i-ry]

Hi Sim, I like Flask because of how many tutorials there are, and because of how easy it is to start up a small server. Your project sounds fun :) good luck :)

[@khaledbouazzi8024]

First like me Ryan! is Google cloud is different from the firebase?

[@d-i-ry]

Haha! You're my first ever "first"! :)

[@khaledbouazzi8024]

@@d-i-ry keep working Mr Ryan your content is great

[@d-i-ry]

Your question is surprisingly intricate. There's a "yes" part to it and a "no" part as well. "Yes" they are the same - it's all Google Cloud, but not 100% the same because you can access Firebase without needing to work in Google Cloud. Firebase is a GCP offering, and it relies on the Google Cloud for all of its own hardware. When we're billed for using Firebase, we're charged for Google Cloud storage, too. But Firebase was a separate company that Google bought, and they try to keep their identity a little bit distinct, too. But slowly, Google Cloud is incorporating it as if it was the faster, younger sibling of BigTable :)

[@d-i-ry]

@@khaledbouazzi8024 oh man!! that means a lot, you made my day!! thank you

[@khaledbouazzi8024]

@@d-i-ry so interesting informations, thx, that's the truth sir ✌️

[@tasneemara1893]

Hi Rayan, I have one doubt regarding authentication of service account. When I try to use gcloud auth activate-service-account account name keyfile=key.jason, am getting error as gcloud crashed

[@d-i-ry]

Hi Tasneem, based on your other question, I think your libraries need to be updated (don't forget to update pip, too!). Let me know if that works :)

[@gustavoag6]

Nice tutorial, Ryan. I only have one doubt, when I open two subscribers the message works as if it were distributed, instead of reaching the two subscribers at the same time. Do you know how to solve this?

[@d-i-ry]

Hey thanks Gustavo! That’s how they’ve made PubSub work, as if it’s distributed. If you need to sync actions, PubSub won’t be able to do that, but there are a bunch of other services that can. My favorite is Firebase Cloud Firestore. You can have any number of listeners, and they’ll all receive the update at the same time. You can even integrate PubSub with Firebase (I think I did 2 videos on that a while back)

[@gustavoag6]

@@d-i-ry thanks ⚡⚡

[@kgmuzungu]

are there libraries to connect to PubSub directly form an Arduino / esp32?

[@d-i-ry]

Oh wow, that sounds like a cool project. I’m sure there are for an esp32, I just haven’t used any

[@omarlx]

Thank youuuu!

[@d-i-ry]

Hey Andres, I’m glad it helped you!

[@karolinakr4734]

is it possible to use PubSub, to receive notifications when a new mail comes in and if it is possible I don't know if you could tell me how to do it.

[@d-i-ry]

Hi Karolina, that one’s not possible because Pubsub messages are very simple messages, and not full blown email messages

[@marcusstateofmind]

Thanks so much

[@d-i-ry]

Hi Dave, I'm glad it helped :)

[@MrDecepticon1993]

Hi Ryan, the subscriber type which you used in this video is Pull type. Could you please help me how to publish messages if the subscriber type is Push. If there is any python code to publish messages to push type subscription, that would be great.

[@d-i-ry]

Hi Rahul, check this article out cloud.google.com/pubsub/docs/push you want to keep your message senders and message receivers nice and simple… pull subscribers listen and receive, and push subscribers speak to the server and send messages :)

[@rrafaelpaz]

Hey mate, how can I get the pub sub private key to connect from an external service? Let's say that I wanna have the pub sub as my backend service and I wanna send message from my application to pub sub... Would I have to create a Service Account pointing to the Pub Sub topic and I would use the Service Account Key to access Pub Sub? Cheers for the help mate!

[@d-i-ry]

Hey Rafael, that’s the exact way I’d do it. I know that my BigQuery in Python video shows the steps to make that service account and link the JSON key (it’s a real short video)

[@rrafaelpaz]

@@d-i-ry You're legend mate! Thank you so much!

[@ivanzaqqa]

excuse me sir, can you explain how to subscribe and then save data to database?

[@d-i-ry]

Hi Ivan, I have a video that should do exactly what you need, kzbin.info/www/bejne/ioqspXiZnpdkbNk

[@SimKieu]

sir, could you please help with how to dockerize your code? I tried but it didn't seem to work the way I expected. Were you able to dockerize it?

[@d-i-ry]

Hi, try running it without a docker image first

[@SimKieu]

@@d-i-ry I did, it worked ok without docker

[@d-i-ry]

@@SimKieu hi, ok that means it’s the docker setup and not the code

[@hamids2065]

Nice video but screens changing very fast. thanks.

[@d-i-ry]

glad you liked. ill try to make the next vid not so jumpy

[@kamlakarkamble3488]

Thanks

[@d-i-ry]

Welcome!

[@MnemonicCarrier]

Is the "path" the same thing as a pub/sub "topic"?

[@d-i-ry]

Hi Scott, I rewatched the video and I didn’t see any reference to “path”. Did I update the code that way? There should be a topic_path for the publisher to write to and a subscription_path for the subscriber to listen on

[@byte__3076]

Does this work even for gmail messages? I'd like to receive just in time the message someone sent to me on gmail

[@d-i-ry]

Hi Byte, for any emails, you’re going to have to run a mail server - very different

[@poojabhardwaj7250]

I follow the same process and getting default credentials error... what to do

[@poojabhardwaj7250]

D-l-Ry

[@poojabhardwaj7250]

@d-l-ry ....

[@d-i-ry]

Hi Pooja, you need to create a full account instead of using a trial account

[@miltonvilleda3344]

nice video tnx!

[@d-i-ry]

Welcome!

[@meredithhoo2312]

really like your video :0) could you make a video to explain this for push subscriptions?

[@d-i-ry]

Hi Meredith, I'm really glad you liked it! When you say push subscriptions, could you link me to what you mean?

[@meredithhoo2312]

@@d-i-ry how to set up and use pubsub push messaging with python

[@d-i-ry]

@@meredithhoo2312 sorry, im still lost cause I thought this video shows that topic. can you explain an example of what you want to see with message pushing? btw, if I describe PubSub in another way, it's like talking in front of a tv camera. the publisher (the speaker) has no idea if anyone is watching. so if you're looking for a delivery guarantee --- like pushing directly to a specific computer --- that's not something PubSub is able to do

[@meredithhoo2312]

@@d-i-ry I thought this video is covering pull subscriptions not push

[@d-i-ry]

@@meredithhoo2312 AHHH! that's correct. PubSub is a passive system. There are only pulls, not pushes. But pulling and pushing can take on a few meanings.... sometimes people mean "passive" listening systems for "pull" and they mean "active" systems for "push". In PubSub, the message pusher is the message publisher. the message puller is the listener or message consumer. PubSub's a passive message system, which means that there's no guarantee the message being produced will ever be consumed. There could be no one listening to the message topic. All the code for the publisher (in my link in the description) is the code for message pushing. I hope that helps :)

[@FireCrowCl]

what do you think is better to communicate two cloud functions, a pubsub or a post request that activates it?

[@d-i-ry]

Huh…. I don’t know. I think there’s a few things to consider. The answer’s probably “no” on this first question, but just to ask, can the 2 functions be combined? One thing I wouldn’t want to do even though it’d be easy is to make 2 pubsub topics and write to both. 1 cloud function listens to 1 topic, the other listens to the 2nd topic. And the reason I don’t like that is because you can’t guarantee which is acted on in what order. Can you share some more details?

[@d-i-ry]

Oh I might have started off on the wrong foot. Do you want the 2 cloud functions to talk directly between each other? Like they need to share info and it’s not necessarily coming from a 3rd source they’re listening to?

[@FireCrowCl]

​@@d-i-ry yes 2 cloud functions to talk directly between each other, one unzip various file and the other insert data with the name of files

[@d-i-ry]

This might be a little over-engineered, but could you let them read and write to Firebase? I think I made a cloud-function-to-Firebase video that would help if that idea’s not tooooo much extra work The problem for me with cloud functions is they’re stateless, so they can’t retain info (or state), so my immediate thought is to let them read and write to a database

[@FireCrowCl]

@@d-i-ry ok thanks a lot

[@muhammadnajibuddinjodiansy2497]

what does it means "unpack the data" using ** ???

[@d-i-ry]

Hi Muhammad, parameter unpacking is a way of sending multiple parameters to a function, all wrapped in a list or dictionary. Suppose your function looks like this: def sum(a, b, c): return a + b + c and you normally call it like this: result = sum(1, 2, 3) but you can also send a "packed" list (or dictionary) of parameters my_values = [1, 2, 3] result = sum(*my_values) the star * will "unpack" the variables for you. it'll turn your 1 list into the 3 input parameters. Keep in mind that if your list has more values than your function is expecting, there'll be an error take a look at the answer from "Lorin Hochstein" (with about 753 votes right now) here stackoverflow.com/questions/36901/what-does-double-star-asterisk-and-star-asterisk-do-for-parameters Lists use 1 star *my_values Dictionaries use 2 stars **my_values, because the first * gets each key-value pair, and the second * gets the individual keys (you'll need to extract the values with the keys you've just been given) :)

[@muhammadnajibuddinjodiansy2497]

@@d-i-ry whoaa cool, thanks a lot pale

[@shadeslayer07]

can we have multiple pub?

[@d-i-ry]

Hi Varunmeda, yeah, you can have 1 program that makes any number of publishers

[@gurudattshet6914]

How to upgrade to python 3 from python 2.7? Please help me i am not able to understand it from 3 days

[@d-i-ry]

Hi Gurudatt, what problem are you having?

[@gurudattshet6914]

I'm getting error asa python 2.7 version is outdated.. Please upgrade while running *python **publisher.py*

[@d-i-ry]

@@gurudattshet6914 www.python.org/downloads/ download Python 3 from here :) you should really only use Python 2 instead of Python 3 these days if there's a specific reason (like "the project started in Python 2 and it would take too much time to rewrite it in Python 3" or "this library only works in Python 2 but not in 3"). And not to worry, I keep all the code simple to understand. If you're comfortable in Python 2, you can follow my project in Python 3 without any problems :)

[@gurudattshet6914]

Please provide me you insta account or any Facebook account so that i can share the screenshot. So that i can explain you me prblm clearly.

[@d-i-ry]

@@gurudattshet6914 i dont use FB, sorry

[@darkdeathoriginal]

how to not use ADC

[@d-i-ry]

Whatcha mean?

[@darkdeathoriginal]

@@d-i-ry application default credentials. I don't want deafout credentials I want pass my own

[@d-i-ry]

@@darkdeathoriginal download your service account's credentials in a JSON key file, then add this code: from google.oauth2 import service_account key_path = "/path/to/your/service-account-file.json" credentials = service_account.Credentials.from_service_account_file(key_path) publisher = pubsub_v1.PublisherClient(credentials=credentials) subscriber = pubsub_v1.SubscriberClient(credentials=credentials) :)

[@darkdeathoriginal]

@@d-i-ry Thank you. I was using my project credential all this time and getting error.