Why Idempotency is very critical in Backend Applications

In this video I explain how Idempteoncy is very important for backend applications.
* Problem Explain Order problem
* Define Idempotent requests
* Examples
* Show Problem with Multiple Requests that times out (client gives up microservices
* Kafka exactly once Exactly-once Semantics is Possible: Here's How Apache Kafka Does it
*
* Ways to make requests idempotent
* Client submits the id
* Server serializes params
🏭 Software Architecture Videos
• Software Architecture
💾 Database Engineering Videos
• Database Engineering
🛰 Network Engineering Videos
• Network Engineering
🏰 Load Balancing and Proxies Videos
• Proxies
🐘 Postgres Videos
• PostgresSQL
🚢Docker
• Docker
🧮 Programming Pattern Videos
• Programming Patterns
🛡 Web Security Videos
• Web Security
🦠 HTTP Videos
• HTTP
🐍 Python Videos
• Python by Example
🔆 Javascript Videos
• Javascript by Example
👾Discord Server / discord
Support me on PayPal
bit.ly/33ENps4
Become a Patreon
/ hnasr
Stay Awesome,
Hussein

[@hrishabhg]

Love you being a quick and simple solver.

[@upadajanki8447]

Explained nicely. Here is what I can think about your question. The API, which is used for shortening the URL, should first look into the DB (or wherever its stored) for the given input (assuming it would be URL) . in case URL exists in the DB, API should return the short URL for the same. if DB does't contain then only API should generate the new short URL and return. Assuming older data in DB is cleaned up or archived periodically.

[@yodude2493]

Love you Hussein thank you very much for contributing a lot to community we see how hard it is how much time it takes and we appreciate that, keep going man!

[@aaronrasing2710]

Greate video Hussein! I really love how you explained it. I've always had a small understanding of this topic and with your video, I get it completely now.

[@jacquesduplessis6175]

Good stuff. Your question: Validate url to be created against the db, if found=send that, else create new one (store in db) and send short url

[@gillfromamritsar]

Awesome video again, lot of these concepts I came to know cuz of your videos.

[@smoothbeak]

You have a really wonderful voice and way of speaking!

[@chaitanyaujeniya8601]

Thanks for all the content!! Super helpful👍👍

[@fartzy]

Another great video man!

[@neeravarora530]

All your videos are awesome.

[@aseel1024]

يعطيك العافية ممكن فيديوهات عن الفرونت اند والاخطاء الي يجب تجنبها مثل المذكورة في الفيديو وبلتحديد في اطر العمل مثل React js

[@zaheerkhan8097]

Very well explained !!

[@sbylk99]

Great video, thank you

[@joeygarcia7826]

I would hash the passed in url and store this hash and data in a database. The hash would be the primary key, so for every future call I can do a quick search and return what is already stored, otherwise run it through the url shortener and store it.

[@NikhilTripathy]

Hussein Nasser, you are a gem! May god bless you.

[@hnasr]

Nikhil Tripathy thanks Nikhil God bless ❤️ enjoy the content

[@litcode702]

for url shortner idempotency I would hash the payload and will take it as an idempotency key. this should be done because for a particular user requesting for shortening a URL with some properties i.e. short url being private / public etc will uniquely identify a short url request

[@ravitejavankam2977]

To deal with idempotent request you can maintain checkoutId in checkout page and in the order page you can pass that checkoutId and you can ensure if an order is placed or not with that checkoutId

[@TannerBarcelos]

We are facing this problem at work big time. We are trying to build a massively high available system and we need our apis / services to be idempotent but many sub systems we manage via APIs are inherently not idempotent so it’s a high challenge.

[@yadrielgonzalez1]

Great video!

[@namandubey20]

Hi Hussein, shouldn't we use put request instead of post in these cases to get idempotency ? That's why put requests are made, right ?

[@mohammedroshan5647]

Hi , Does anyone have any implementation suggestion to handle de-duplication at Kafka consumer side ? i.e to make idempotent consumer which will discard dupllicate messages from Kafka in the event of any broker or offset issues. One solution aware of is to store last read consumer offset number in a database. Store the offset number and msg values in single update call to ensure atomicity. Whenever consumer reads again, it check in the DB to ensure is not reading offset number less than already in the DB. Downside is if DB is huge , it will be perfomance impact. Another possible way is store topic name and last read offset number in a seperate table. Any other recommended practice ?

[@abhinavpandey2772]

URL Idempotent: Can we put URL Mapping in the Protobuf?

[@shivangchaturvedi237]

Crisp! 😀

[@alfredskaria]

Great video Hussein. However I have one doubt. In the video you explained that for making the request as idempotent, we can create a client id for a request. So , suppose if two POST requests are coming from two different devices ( not concurrently) , then there can be a case where client id becomes different. So it can end up in creating two records in the DB. But , my requirement is to update the record created by the 1st POST request. In such scenario, how do we handle idempotency? Or can we use client id approach?

[@amila325]

You use PUT requests to update and use POST for inserts. Therefore if you send a post request from two different clients it should enter two new records. But lets say you can’t do that. In that case you will have to create some sort of a hash key from fields which are not gonna get updated and unique to that record.

[@alfredskaria]

@@amila325 Thanks Ami, I was also thinking about a similar kind of solution.

[@marting1219]

HTTP POSTs are defined as NOT idempotent. Please use PUT when you define an idempotent endpoint to write to db.

[@satyamsinha5429]

Nice ! Followed you .

[@vivekacharya3110]

I have a partial resource update so I made that API as PATCH. And the operation and implementation is idempotent. But in REST PATCH is not idempotent. What's your thought on this? Is going with PUT a better design or PATCH with idempotent?

[@hnasr]

Think about it semantically. If you issue that resource update multiple times what well happen? If it is safe, use PUT else continue using patch or POST

[@Aleks-fp1kq]

But if you keep refreshing the page it would generate a new key so even if your POST indempotent, since the new key is generated, it would still create a new resource. What am I missing here?

[@hadeedtmgggg5837]

In case of url shortener we will save real link in db and when user send a request again we check if the link already existed in db we will send the shorten link of that link else we generate a new link if I t doesn't exist

[@mridul1161]

when you give example related to real world problems like the transaction one....video become much more interesting...........try make more relations with practial implementation of those please!!

[@hnasr]

Quarantine Coder thank you 🙏I did give an example in this video right? The ordering system ? Were you looking for something else specific let me know so i do better next time 😊

[@mridul1161]

@@hnasr yeah.. Gave one in this.. That.. Why I liked this one.. I mean to say your experience with solving real world problem .. That amplifys the video

[@Raaviolis]

Isn’t idempotency like atomic isolation “A” “I” transaction from ACID concurrency rules ?

[@hnasr]

Ravi Hindocha Hey Ravi, there are some similarities but also very big difference between the two properties. Check out my ACID video for details 😊

[@Raaviolis]

Hussein Nasser Appreciate , I have knowledge about acid(not in hippy way) just trying to mash up two worlds, and trying to find common principles, between http and sql world they are kind of doing same crud operations let’s say you order pizza a you will put the order and wait for transaction to complete at the check out window and you if update the order again, so basically aa = a , if that is translated in acid principles in a finite state machine once you insert the record to buy, isolate the transaction meaning give a uuid to it and make sure it’s atomic all or nothing meaning you ordered once exactly once, no updates or cancelation yet if they are it will be a separate finite transaction , same state is consistently preserved just once on same uuid transaction which is put or insert, so key part is isolation of the transaction in an atomic way and you can always roll back in case of failure, just thinking out loud let me know if you need clarification.

[@animatedzombie64]

i love that moment when Hussein laughs at some hole in the system.

[@shafeeqr2]

Your question is very interesting. Typically we want to allow multiple shortened URLs for the same input URL (for different reasons) but at the same time we want to avoid loss of idempotency. I would take a hash of the URL and store it in the database. I would also store the Datetime of said request in YYYYmmDD HH format. I would create a unique index in this table as the composite of the hash and datetime. The result is we will not allow the same URL to create multiple shortened URLs in the same hour. If user requests the same URL in the same hour, our endpoint can return the pre-existing value from the database.

[@rampandey191]

Hey can you explain the reason to create different shortened URLs for same URL? Because I think it should not be allowed

[@lonerider543]

@@rampandey191 Bit late to the party, but I'd assume for analytics (as an example) you'd want to allow multiple. Let's say I create a shortened URL that leads to a popular website and someone else does the same. We only want to see however how much traffic that shortened URL is getting from wherever we have shared the URL to. If both people get the same shortened URL, then the analytics would get mixed up together. By creating 2 shortened URLs for the same URL this wouldn't happen and each of the 2 users would have their own URL that they can share and keep track of how much traffic they are getting from it.

[@rampandey191]

@@lonerider543 makes sense thanks

[@sumarouno]

I just know about this from this video

[@hnasr]

It is an important concept, learned about it while researching Kafka around a year ago. Thanks for watching 🙏

[@footballhero9717]

استاذ حسين كنت عاوز اعرف منين انتا عارف كل هذي الاشياء انا اول مرة اعرفها لما دخلت على قناتك لو تدلني على مكان اقدر اعرفها زيك اكون شاكرلك بقية حياتي

[@hnasr]

Football Hero اهلا عزيزي، القراءه المستمره، استماع للبودكاست ، مشاهده فيديوهات. رما زلت اعتبر نفسي لا اعرف اي شي. وعاء العلم لا يمتلى بما فيه. بالتوفيق شاهد هذا الفيديو اتكلم بالتفصيل My Preferred Method of Learning Backend Engineering Technologies kzbin.info/www/bejne/an_WiKGKlJJ5pac

[@footballhero9717]

@@hnasr اشكرك استاذ ناصر على كرمك و مساعدتك لنا و انا اوعدك في يوم حردلك هذا الجميل 💙💙💙💙

[@Cdswjp]

How about.... I have an idea.... how about..... a video that shows how to do retries

[@adityasethi9794]

you can explain stuff in few minutes and that's an art. people make 30mins videos on this lol. just puts me off

[@meghaagarwal6159]

Just a suggestion- Its a knowledge video, it would be better if you clearly narrate the things instead of speaking in an unclear accent.