Building a Webhooks System

Do you need to integrate with external systems or services? Leveraging an event driven architecture enables you to build a webhooks system that can be decoupled from your main application code. Enabling you to call external systems that have subscribed via webhooks in complete isolation from your application code.
🔗 EventStoreDB
eventsto.re/codeopinion
🔔 Subscribe: / @codeopinion
💥 Join this channel to get access to source code & demos!
/ @codeopinion
🔥 Don't have the JOIN button? Support me on Patreon!
/ codeopinion
📝 Blog: codeopinion.com
👋 Twitter: / codeopinion
✨ LinkedIn: / dcomartin
📧 Weekly Updates: mailchi.mp/63c7a0b3ff38/codeo...
0:00 Intro
0:44 Coupling
2:00 Publish/Subscribe
3:12 Demo
6:39 Isolation
#softwarearchitecture #softwaredesign #webhooks

[@kartikag01]

New to system design, great to learn.

[@sathyajithps013]

Great video. I might need to use it soon...😅

[@xance]

Azure durable functions can cover these scenario’s and one of the best fit’s

[@kevingaiman4464]

Very timely! I am currently writing a webhook processor for a major hotel company.

[@CodeOpinion]

Best of luck!

[@KeithBarrowsToday]

As an alternative to sending it back to RMQ, you could have a (simple) internal Event Bus.Your web hook would still have the Subscriber to get the message from RMQ, your function would then send an event signal to all the external subscribers via whatever mechanism is defined for each subscriber. Probably a splitting of hairs here but I tend to lean towards keeping my Brokers self contained and not back feeding the source in a round robin manner.

[@CodeOpinion]

If you have a means for handling the failures individually and in isolation, by all means, because that's the goal. Most messaging libraries and/or brokers support defining how to handle retries, (exponential) backoffs, fallbacks, DLQ, directly out of the box, hence why I'm suggesting it as you end up using the same execution path.

[@SnOrfus]

@@CodeOpinion for what it's worth, using a library like Polly makes a lot of those resilience patterns very straightforward to use. I've heard you recommend Brighter before, and it's what they're using under the hood to do the same.

[@CodeOpinion]

@@SnOrfus Yes, I like Polly. While it can work well in some scenarios, not so much in others. For example, if you're consuming a message and you use something like an exponential backoff, that entire consumer thread is blocked. Meaning when you're consuming message from a broker, you're only consuming so many at a given time. For example purposes, lets say you're only consuming 1 message at a time. While you're waiting for your exponential backoff with Polly, you won't be consuming or processing any other messages. This can kill throughput, even if you had multiple different threads/processes with the competing consumers pattern. I did a video about this, I may have to revisit it. kzbin.info/www/bejne/iZbWdox-fM2fgrM BTW, I'm not implying you don't know this, I'm just saying it incase others read it and find it helpful. Thanks again for the comment!

[@mohamedbeyremmakhlouf]

😊

[@ahmadjaber2940]

Is using both return value from stripe and updating my backend + using webhooks as a redundant a good solution?

[@EverydayRoadster]

Having a well designed topic structure allows you to skip the middle layer Webhook. The fine grained, backend system specific Webhooks would subscribe to the topic (and maybe partial data pattern) they are capable of dealing with.

[@CodeOpinion]

Not entirely if were on the same page. OrderStatusChangedToPaid event is getting published to a topic. The webhook system has the subscriptions from the 3rd party outside services. it's not until it actually consumes the event does it know who to send it to since that's a constantly change state of subscriptions. Meaning consuming OrderStatusChangedToPaid turns into multiple SendWebHook(data, webhookSubscriberCallbackURI) commands being sent to a queue.

[@thatoneuser8600]

Would you consider the cross-language ReactiveX API an implementation of pub-sub, or do you think that's just an observer?

[@CodeOpinion]

Observers subscribe to the subject. In more event-driven pub-sub, the subscriber doesn't subscribe to the publisher, it subscribes to the broker/topic/channel.

[@rey6253]

What i did in my app is that i create a simple mechanism to create a new task from main thread which will execute a handler within the same application but will have its own lifetime and everything, hence no need to await from main thread, since i dont want to add a rabbitmq instance for now and keep a simple monotholic approach. In case i have a lot of users, adding another node of the same app increases the processing power as well. If that will not work out then i can switch to rabbitmq easily because the handling logic is encapsulated properly

[@CodeOpinion]

Yes, you could use something like Hangfire or any type of background job worker. The difference with that approach is you're being explicit about the thing that needs to run in the background. That's not inherently bad, it's just more coupled. Publish/Subscribe, when you publish the event, you have absolutely no clue who the consumers are. One consumer could be a webhook system, another could be for sending out email.

[@jdang67]

Can we remove the WebHook project, and on the external projects, subscribe to the channels the system needs? What do we gain when using the Webhook over Pubsub?

[@CodeOpinion]

Sure, assuming you can expose it. Webhooks are just push mechanisms for untrusted or external consumers to subscribe to events. Ultimately it is pub/sub, it's just how they get notified is through an HTTP request callback to them.

[@shaunwhyte6686]

How would Auth be set up to external clients?

[@nilsgoovaerts4580]

The HttpClient in the WebHooks project handles the auth to the external apis, so this could be anything the external api supports, like just an api key in the URL, an OAuth token, ...

[@izorget]

I am so confused bro. Am a self taught developer. i need direct assistance integrating webhook events with my flutterflow app which should send alerts when a payment is made through the payment gateway i used: YOCO

[@oleksandrsova4803]

But what about the replies? Does this approach assume fire and forget semantics? And what about acknowledging? How will the Application find out that this particular hook is processed? Is there any standard way of doing this? So far those are just fancy words on top of what have been around for decades...

[@malikrumi1206]

Technically speaking, if you separate the webhook from the other processes, is it still a web "hook"? I've been told the name and definition comes from the fact that it "hooks *into*" some other process. If you separate it, it is no longer hooking into anything. Right?

[@mhDuke]

why do we need the extra webhook service layer?! it seems redundant!

[@CodeOpinion]

What's sending webhooks (http requests to external HTTP APIs) is decoupled from your application code that's generating the event. Sending an HTTP request to an external service is likely going to require fault tolerance (timeouts, unavailable, latency, etc) that you want to handle in isolation.

[@mhDuke]

@@CodeOpinion cheers. that's enlightening. thank you derek :)

[@isi1044]

huh

[@lightprogrammer]

you talk way too slow

[@CodeOpinion]

Thaaaaaaaank............ yooooooou........... for........................................the..... coooooment!