Pub/Sub tips and tricks

[@googlecloudtech]

Tune into our #AskGoogleCloud premiere on Friday, March 12 10AM PT for a chance to ask Google Cloud’s serverless experts any questions regarding serverless architectures → goo.gle/3evqXcU Get $300 and start running workloads for free → goo.gle/2Zvv4wE

[@MuhammadAmjad-qz1ik]

Help me

[@aaronbelchamber5007]

A great example used, very clear! Nice job!

[@AntonioCachuanAlipazaga]

Excellent tips! Please, a Dataflow tips and tricks would be aweosome and a video about real time analytics (Pub/Sub + Dataflow + Tensorflow)

[@tianzicai9964]

There's a streaming example that uses Pub/Sub + Dataflow + Tensorflow, which it is part of a longer tutorial that showcases batch processing. Try to search for "molecules-walkthrough" or "Machine Learning with Apache Beam and TensorFlow" in the GCP docs. Hope you find that tutorial useful, and thank you for watching and for your comment.

[@AntonioCachuanAlipazaga]

Thanks Tianzi! I found the example cloud.google.com/dataflow/docs/samples/molecules-walkthrough I'll try it.

[@laskdkmgful]

Ordering is important. Might try to share more in detail next time. Thx. 👍

[@Tenelia]

I also want to see ordering!

[@MartinOmander]

@@Tenelia More details about ordering -- got it! We are adding this to our list of new episode ideas.

[@JosephLust]

Great presentation, Martin & Wes!!

[@pinkanrou7167]

Excellent one!!! Thanks @Tianzi :) I tried the dead letter topic one but with mixed luck :( It worked when I tested without any subscriber . I pushed a message and checked the delivery attempt after 5 times it pushed the message to dead letter one. When I used dataflow as a subscriber and intentionally threw exception from subscriber code it did not work. The message is not getting acknowledged as I can see from Unacked message count ,but it never reached to dead letter topic. So the error count keeps on increasing in dataflow.I have not written any code inside subscriber to push the message to dead letter as I think It should be pushed automatically. Am I missing anything ?

[@tianzicai9964]

Thank you Pinkan. So glad to know that you gave it a try. And thank you for the question. Mind I ask you how you "intentionally threw exception from subscriber code" in your pipeline? `PubsubIO.readMessages()` returns `org.apache.beam.sdk.io.gcp.pubsub.PubsubMessage.PubsubMessage`, which is unlike `com.google.pubsub.v1.PubsubMessage` and does not have an `ack()` or `nack()` method. I think it's for this reason that all Pub/Sub messages arriving in the Beam/Dataflow will be considered acknowledged. "The Dataflow runner's implementation of PubsubIO automatically acknowledges messages once they have been successfully processed by the first fused stage (and side-effects of that processing have been written to persistent storage)." [1] Does this make sense? [1]: cloud.google.com/dataflow/docs/concepts/streaming-with-cloud-pubsub#integration-features

[@robinthomas7584]

Whats the behavior when publisher sets message ordering and publishes messages with an ordering key and there are multiple subscribers listening to the same topic and only one of them is interested in ordered messages.

[@tianzicai9964]

If you are using any of the official client libraries, messages of the same ordering key will be delivered to the same subscriber. Messages published with no ordering keys can be pulled by subscribers to a subscription with ordering enabled in any order. Thanks for watching and for your question!

[@ganeshsonawane4276]

Excellent idea. Got more insights about pub/sub through this.

[@amoslam-w3y]

for the message ordering, what if request a ride fails in dispatcher? since message ordering cant be supported via dead letter and if request ride + cancel ride is pulled together in the same pullResponse, the subscriber can process cancel ride as well even if request ride fails.

[@GabrielAmyot-q3e]

Thank for the tips. Love this format. Thanks

[@PatrickSteil]

I tested code running on Cloud Run to add 100 messages to a queue and it took .04s per message. That is 40ms per message add. The message was only like 32 bytes. Does this sound like it is too slow? Any way to speed this up?

[@TheMomander]

Have a look at "Publish with batching settings" in PubSub. That may help speed it up.

[@PatrickSteil]

@@TheMomander Thx, just ran into this in the docs a few minutes ago... :) Appreciate the reply

[@alexpalau5350]

Really nice tips! Ordering messages feature will help a lot of people and projects. What will happen if order is enabled and dead lettering also? Will good messages still be sended in order eventhough some of them are missing between the good ones?

[@tianzicai9964]

To answer your first question, if the subscription attached to the dead-letter topic has ordering enabled, then messages (sent with an ordering key) that you receive from that subscription will be ordered. To answer your second question, (good) messages not forwarded to the dead-letter topic will not be received by the subscription attached to the dead-letter topic. Thanks for watching and for your questions!

[@eyesopen6110]

What are the calls between the Dispatcher and Fleet manager (get-available_drivers, send-available-drivers)? Are these also pub/sub? How are these communicating?

[@raihankhan5184]

@Google Cloud Tech. When you guys show the demo on console, it doesn't benefit anyone. Most of the companies are doing it using IAS or using Python api. So please provide more tutorial using coding...

[@TheMomander]

You bring up a good point. We often use the console as it is more visual, easier to understand, and shows the concepts across programming languages, not just for one language. We'll see if we can include more code in the future!

[@castellesenminier6291]

Interesting intro

[@havardestensen1012]

Too bad dead-lettering doesn't work for event-driven functions since you cannot manually NACK.

[@tianzicai9964]

Hm, I'm not sure I understand the full context. A "NACK" in the case of event-driven functions means any response code that's not 102, 200, 201, 202, or 204. It's possible to write your function to return failure code where necessary. Could you tell me more about what's on your mind?

[@havardestensen1012]

explains it better than I can 😄

[@dheer211]

Great Video thanks

[@googlecloudtech]

Glad you enjoyed it

[@LittleRapGuy]

nice

[@nenora]

Great

[@stackdev-io]

good

[@MuhammadAmjad-qz1ik]

help me