Why Idempotency is very critical in Backend Applications

[@gillfromamritsar]

Awesome video again, lot of these concepts I came to know cuz of your videos.

[@hrishabhg]

Love you being a quick and simple solver.

[@upadajanki8447]

Explained nicely. Here is what I can think about your question. The API, which is used for shortening the URL, should first look into the DB (or wherever its stored) for the given input (assuming it would be URL) . in case URL exists in the DB, API should return the short URL for the same. if DB does't contain then only API should generate the new short URL and return. Assuming older data in DB is cleaned up or archived periodically.

[@jacquesduplessis6175]

Good stuff. Your question: Validate url to be created against the db, if found=send that, else create new one (store in db) and send short url

[@litcode702]

for url shortner idempotency I would hash the payload and will take it as an idempotency key. this should be done because for a particular user requesting for shortening a URL with some properties i.e. short url being private / public etc will uniquely identify a short url request

[@TannerBarcelos]

We are facing this problem at work big time. We are trying to build a massively high available system and we need our apis / services to be idempotent but many sub systems we manage via APIs are inherently not idempotent so it’s a high challenge.

[@Aleks-fp1kq]

But if you keep refreshing the page it would generate a new key so even if your POST indempotent, since the new key is generated, it would still create a new resource. What am I missing here?

[@meghaagarwal6159]

Just a suggestion- Its a knowledge video, it would be better if you clearly narrate the things instead of speaking in an unclear accent.

[@hadeedtmgggg5837]

In case of url shortener we will save real link in db and when user send a request again we check if the link already existed in db we will send the shorten link of that link else we generate a new link if I t doesn't exist

[@marting1219]

HTTP POSTs are defined as NOT idempotent. Please use PUT when you define an idempotent endpoint to write to db.

[@ravitejavankam2977]

To deal with idempotent request you can maintain checkoutId in checkout page and in the order page you can pass that checkoutId and you can ensure if an order is placed or not with that checkoutId

[@vivekacharya3110]

I have a partial resource update so I made that API as PATCH. And the operation and implementation is idempotent. But in REST PATCH is not idempotent. What's your thought on this? Is going with PUT a better design or PATCH with idempotent?

[@hnasr]

Think about it semantically. If you issue that resource update multiple times what well happen? If it is safe, use PUT else continue using patch or POST

[@chaitanyaujeniya8601]

Thanks for all the content!! Super helpful👍👍

[@smoothbeak]

You have a really wonderful voice and way of speaking!

[@adityasethi9794]

you can explain stuff in few minutes and that's an art. people make 30mins videos on this lol. just puts me off

[@mridul1161]

when you give example related to real world problems like the transaction one....video become much more interesting...........try make more relations with practial implementation of those please!!

[@hnasr]

Quarantine Coder thank you 🙏I did give an example in this video right? The ordering system ? Were you looking for something else specific let me know so i do better next time 😊

[@mridul1161]

@@hnasr yeah.. Gave one in this.. That.. Why I liked this one.. I mean to say your experience with solving real world problem .. That amplifys the video

[@namandubey20]

Hi Hussein, shouldn't we use put request instead of post in these cases to get idempotency ? That's why put requests are made, right ?

[@mohammedroshan5647]

Hi , Does anyone have any implementation suggestion to handle de-duplication at Kafka consumer side ? i.e to make idempotent consumer which will discard dupllicate messages from Kafka in the event of any broker or offset issues. One solution aware of is to store last read consumer offset number in a database. Store the offset number and msg values in single update call to ensure atomicity. Whenever consumer reads again, it check in the DB to ensure is not reading offset number less than already in the DB. Downside is if DB is huge , it will be perfomance impact. Another possible way is store topic name and last read offset number in a seperate table. Any other recommended practice ?

[@fartzy]

Another great video man!

[@joeygarcia7826]

I would hash the passed in url and store this hash and data in a database. The hash would be the primary key, so for every future call I can do a quick search and return what is already stored, otherwise run it through the url shortener and store it.

[@sbylk99]

Great video, thank you

[@NikhilTripathy]

Hussein Nasser, you are a gem! May god bless you.

[@hnasr]

Nikhil Tripathy thanks Nikhil God bless ❤️ enjoy the content

[@yodude2493]

Love you Hussein thank you very much for contributing a lot to community we see how hard it is how much time it takes and we appreciate that, keep going man!

[@abhinavpandey2772]

URL Idempotent: Can we put URL Mapping in the Protobuf?

[@Cdswjp]

How about.... I have an idea.... how about..... a video that shows how to do retries

[@aseel1024]

يعطيك العافية ممكن فيديوهات عن الفرونت اند والاخطاء الي يجب تجنبها مثل المذكورة في الفيديو وبلتحديد في اطر العمل مثل React js

[@animatedzombie64]

i love that moment when Hussein laughs at some hole in the system.

[@aaronrasing2710]

Greate video Hussein! I really love how you explained it. I've always had a small understanding of this topic and with your video, I get it completely now.

[@sumarouno]

I just know about this from this video

[@hnasr]

It is an important concept, learned about it while researching Kafka around a year ago. Thanks for watching 🙏

[@zaheerkhan8097]

Very well explained !!

[@satyamsinha5429]

Nice ! Followed you .

[@Raaviolis]

Isn’t idempotency like atomic isolation “A” “I” transaction from ACID concurrency rules ?

[@hnasr]

Ravi Hindocha Hey Ravi, there are some similarities but also very big difference between the two properties. Check out my ACID video for details 😊

[@Raaviolis]

Hussein Nasser Appreciate , I have knowledge about acid(not in hippy way) just trying to mash up two worlds, and trying to find common principles, between http and sql world they are kind of doing same crud operations let’s say you order pizza a you will put the order and wait for transaction to complete at the check out window and you if update the order again, so basically aa = a , if that is translated in acid principles in a finite state machine once you insert the record to buy, isolate the transaction meaning give a uuid to it and make sure it’s atomic all or nothing meaning you ordered once exactly once, no updates or cancelation yet if they are it will be a separate finite transaction , same state is consistently preserved just once on same uuid transaction which is put or insert, so key part is isolation of the transaction in an atomic way and you can always roll back in case of failure, just thinking out loud let me know if you need clarification.

[@footballhero9717]

استاذ حسين كنت عاوز اعرف منين انتا عارف كل هذي الاشياء انا اول مرة اعرفها لما دخلت على قناتك لو تدلني على مكان اقدر اعرفها زيك اكون شاكرلك بقية حياتي

[@hnasr]

Football Hero اهلا عزيزي، القراءه المستمره، استماع للبودكاست ، مشاهده فيديوهات. رما زلت اعتبر نفسي لا اعرف اي شي. وعاء العلم لا يمتلى بما فيه. بالتوفيق شاهد هذا الفيديو اتكلم بالتفصيل My Preferred Method of Learning Backend Engineering Technologies kzbin.info/www/bejne/an_WiKGKlJJ5pac

[@footballhero9717]

@@hnasr اشكرك استاذ ناصر على كرمك و مساعدتك لنا و انا اوعدك في يوم حردلك هذا الجميل 💙💙💙💙

[@neeravarora530]

All your videos are awesome.

[@shivangchaturvedi237]

Crisp! 😀

[@alfredskaria]

Great video Hussein. However I have one doubt. In the video you explained that for making the request as idempotent, we can create a client id for a request. So , suppose if two POST requests are coming from two different devices ( not concurrently) , then there can be a case where client id becomes different. So it can end up in creating two records in the DB. But , my requirement is to update the record created by the 1st POST request. In such scenario, how do we handle idempotency? Or can we use client id approach?

[@amila325]

You use PUT requests to update and use POST for inserts. Therefore if you send a post request from two different clients it should enter two new records. But lets say you can’t do that. In that case you will have to create some sort of a hash key from fields which are not gonna get updated and unique to that record.

[@alfredskaria]

@@amila325 Thanks Ami, I was also thinking about a similar kind of solution.

[@yadrielgonzalez1]

Great video!

[@shafeeqr2]

Your question is very interesting. Typically we want to allow multiple shortened URLs for the same input URL (for different reasons) but at the same time we want to avoid loss of idempotency. I would take a hash of the URL and store it in the database. I would also store the Datetime of said request in YYYYmmDD HH format. I would create a unique index in this table as the composite of the hash and datetime. The result is we will not allow the same URL to create multiple shortened URLs in the same hour. If user requests the same URL in the same hour, our endpoint can return the pre-existing value from the database.

[@rampandey191]

Hey can you explain the reason to create different shortened URLs for same URL? Because I think it should not be allowed

[@lonerider543]

@@rampandey191 Bit late to the party, but I'd assume for analytics (as an example) you'd want to allow multiple. Let's say I create a shortened URL that leads to a popular website and someone else does the same. We only want to see however how much traffic that shortened URL is getting from wherever we have shared the URL to. If both people get the same shortened URL, then the analytics would get mixed up together. By creating 2 shortened URLs for the same URL this wouldn't happen and each of the 2 users would have their own URL that they can share and keep track of how much traffic they are getting from it.

[@rampandey191]

@@lonerider543 makes sense thanks