14. Design Idempotent POST API | System Design to Handle Duplicate Request by Idempotency Handler

Solution for System Design Interview Question - Design Idempotent API or Design Idempotency in Microservices or How to handle duplicate request using Idempotency
Today, i have taken a session on High Level Design (HLD) of "HOW TO MAKE POST API IDEMPOTENT"
Chapters:
00:00 - Introduction and Topic Brief
00:50 - Different between Idempotency and Concurrency
02:11 - Definition of Idempotency with Example
04:00 - Idempotency Nature of POST, PUT, GET, DELETE
08:31 - How Duplicate Request come to the Server
12:08 - Design Idempotent API
Similar interview question like:
- Design Idempotent API
- Design Idempotency Handler
- How to Make POST api Idempotent
- How to handle Idempotency in Payment API
- How to handle duplicate request
Connect with me 1:1 on topmate: topmate.io/concept_coding
Join this channel to get access to perks:
/ @conceptandcoding
#systemdesign #idempotency #idempotentapi #hld #interview #amazon #faang

[@ConceptandCoding]

Enjoy the complete LLD and HLD playlist (both BASICS to ADVANCED): Complete LLD Playlist: kzbin.info/aero/PL6W8uoQQ2c61X_9e6Net0WdYZidm7zooW Complete HLD Playlist: kzbin.info/aero/PL6W8uoQQ2c63W58rpNFDwdrBnq5G3EfT7

[@dhirendrajha6133]

Hi Shrayansh, I have one small doubt if IK is unique for each request than how it is found that the coming IK is duplicate? since its already creating unique UUID everytime? Please let me know

[@vishalsingh-nk8nt]

For parallel request if we are putting lock on code level using synchronise or mutex then only one thread can access that method at a time because of which our latency can increase for other requests. hence we should keep row level lock on DB side while creating a IK key then it will not block any request related to other IK key to reduce latency and we can release lock once status updated to created. Thanks for really nice video!!

[@nishankdas1895]

While I feel the idea is correct here I feel we can greatly benefit from not using a lock here instead we can go over and have some weaker isolation level here like serialized snapshot isolation on the database for parallel requests.

[@vmalhotra98]

if we are using a SQL DB eg Mysql I think we can use UNIQUE KEY CONSTRAINTS which make sure there is always one record inside db. and can do handling for exception. What are your views on this? @@nishankdas1895

[@brohit44]

Wow !! You just helped showing a way to handle one critical issue, I have been facing. Thank you so much for this topic-refresher video.

[@ConceptandCoding]

Thanks Rohit, pls do share it with your connections, if you liked the content

[@KonNishant]

first time i bought any course or joined a membership, now i m thinking that i could have joined your membership earlier, just loving your content.

[@ConceptandCoding]

Happy Learning

[@shahidkhan-ln8mz]

Thank you sir... Ur lectures are awesome 👍

[@ConceptandCoding]

Thanks 👍

[@thecloudbaba8668]

wonderful content on idempotency... you explained it in such a easy way.. thanks a lot :)

[@ConceptandCoding]

thanks

[@sumitbasu5146]

I am not hold myself to write "Thank you for this wonderful video"

[@ConceptandCoding]

Thanks buddy

[@bangbang86]

Nice video, good amount of cases covered One very important case not covered is if client sends a request R1with idempotency key I1, it gets successfuly executed. Now client again another request R2 but uses same idemopotency key I1, without any validation of incoming request the response returned would be 200 One possible solution is to store request checksum with idempotency key and validate it.

[@HA-ky5vd]

Idempotency Key is generated in such a way that it would be different for request R2, I think you will not control what Idempotency key to send

[@bangbang86]

@@HA-ky5vd point is that if somehow client duplicates idempotency key for different request then it could be handled by using request checksum

[@mahesh_kndpl]

Thanks for wonderful Video. I have one doubt, as there are duplicates request, as we are using uuid for IK. How can we make sure it will create same uuid for two requests in sequential? Even a second difference will change uuid.

[@yashbansal2526]

Learnt something new, thanks for the video 👏

[@ConceptandCoding]

Glad it was helpful! :)

[@AyushGupta-kb9iv]

Awesome Video Very Well Explained...

[@ConceptandCoding]

Thank you

[@anojk1]

Thanks. Awesome lecture

[@ConceptandCoding]

Thank 🙏

[@belalahmedkhan4619]

if you have one master and multiple read server then unique constraints helps. We had solved the problem by composite key and unique constraints

[@gurmeetchawla8362]

Although an excellent video again. Thanks for the effort. But I think it still needs some more details. Especially in the first case of serial execution and retry scenario when the entry was created in idempotent schema but not updated to consumed or claimed and another try came then you said the message will be it is being processed. Eventually how will it work , how will it be communicated to the client after the third or fourth retry. What happens if the retry does not come.?

[@Lucifer-xt7un]

Wooooooooow😍❤️

[@ConceptandCoding]

Thanks

[@vmalhotra98]

Hi @ConceptandCoding Great Content.. I have feq questions 1. Rather than having IKey cant we maintain a unique ID lets say I have customer database so my unique key will be customerId. Now inside post req implementation I can first fetch if any entry existed for this customerId if yes then it means we already have a record. 2. If I am using Mysql can we use UNIQUE KEY CONSTRAINST to solve problem of duplicate records .. no need of using mutual exclusion then 3.If a request marked as Created gave an exception then I think we should update IKey status in your provided solution else for a duplicate Req it will be like prev req is still processing but in actual prev req have already given exception..

[@ujjalroy1442]

Thanx sirrrr...

[@ConceptandCoding]

Welcome

[@anjumanislamiachhatarpur7093]

Nice explanation, I understood the concept of how to handle idempotency. However, I see it as vulnerable because if the client calculates a different key for the same payload in a duplicate request, in that case, our server will create a duplicate row in DB. so basically, my question is how the client generates the unique idempotent key for the same payload request.? could you please tell us in your spare time?

[@ConceptandCoding]

You are right, if client generate 2 key for same request, then 2 row will get persisted into the DB. Idempotency key generation logic differs from client to client. For ex: take one example: I can generate a Idemp Key based on UserID: API name: timestamp. But there can be many logic to generate this based on clients need

[@anjumanislamiachhatarpur7093]

@@ConceptandCoding Thank you so much, Understood🙏

[@rutvikdoshi1820]

Hi Shreyansh, @ 35:02 can we not use consistent hashing for delegation of api requests to a single server here? This way we ensure that requests go the same server and avoid cache?

[@Randomvideo2610]

Hi Shrayansh, How client will ensure for parallel request they will same IK?

[@vamsiprasadkumili1237]

@ConceptandCoding Suppose every table has updated_by timestamp that gets updated whenever a rows is updated, in case of put request multiple times, though it doesn't cause other data to change, but it makes the updated_at change for every request, how does PUT is idempotent in this case as the db state is changed for every put request w.r.t updated_at field??

[@vmalhotra98]

I guess use case of this field is to fetch last updated time.. and in put we are updating same record not creating new so its not violating idempotency

[@ManishChauhan1095cs]

HI , just one question Since the idemptency key is being generated by client(browser) ,is it not vulnerable , attacker my look at a request and figure out the key pattern, should we not have some kind of validation on the key itself?

[@r4ravi4u]

Suppose if original request failed due to any payment gateway failure or another backend service, then IK1 status should be "failed", neither created nor consumed, hence duplicate request must be handled and allows to make another request to gateway/another backend service. Whats your opinion ?

[@abishektvs3902]

Nice Explanation 👏 I am wondered how the idempontent key would be consistent, in sense, how client identifies that the idempotent request and create same key for both, atleast let say the single client manages it, but how parallel request from different client can create same key. Hoping to know about it. Thanks

[@ConceptandCoding]

No, client only retries it right. Even if there are 100 clients, only 1 client makes retry after timeout and it may go to any server machines

[@ziyamohammed635]

I think in parallel scenario, once the key goes to created, other threads should be unlocked in this way, you can send message saying "request is already in process" and thread can used for some other tasks. i have one question: if we use locks, then each request will go sequentially. wont this increase latency?

[@vishwanath501]

Hi Shrayansh, thank you for amazing explanation, really it is very helpful and expecting more videos 😊. I have one doubt if we have two server and one DB then how mutex or semaphore will work for parallel request with same IK...

[@ConceptandCoding]

Hi Vishwanath, there are many Distributed Mutual exclusion Algorithms present like Dekker Mutual exclusion algo,Dikshtra mutual exclusion algo, token based algo etc. With that these can be achieved, in video i did not go much deep into mutual exclusion as it's a big topic in itself but i think let me cover this in next video.

[@ADITYAKUMARking]

For mutual exclusion we can use transaction level “serialisable” in database(works in same region though). It will take all read and writes sequentially.

[@Sumit-yn7dn]

Hi Shreyansh, is this the last video of this HLD series? . I have gone through all the videos uploaded till today. They are simply top-class.

[@ConceptandCoding]

No Sumit, there are so many videos in line. Currently I am just covering imp topics and questions i will cover .

[@uditagrawal6603]

Why Idempotency key is generated by client, shouldn't it be generated by server, like in payments scenarios payment gateway like stripe generates the IK and provide it to client for each operation? Secondly we can put the IK that we got in cache like redis and store in db only with the transaction id. Thirdly we can maintain 3 states for IK like created (when generated and returned to client), processing (when a request has started processing with that IK), completed (when request is completed successfully) So in case of parallel request as well with the help of row level lock we should be able to achieve idempotency. Also marking of key status as completed and the operation should be atomic like a transaction. Do let me know if you think otherwise.

[@mandadapukinand2742]

if both requests are coming at the same time there are no rows present in the db on which row do we have to add the lock?

[@akshaygupta6785]

Great Video as always. Cherish your channel 😀Kudos to you 🥳 One doubt - Let's say we are adding IK and marking it as created and now something gets broken, things are not completed and hence the status of IK is not changed to Consumed. Now the repeated request will come and see it as created, so will treat it as it is still being processed and return 409. How should we handle this case ?

[@ConceptandCoding]

as far as i know, Every IK get inserted with expiry time. After a particular time those keys marked as invalid and after that same IK can be entertained again.

[@mayankb9]

Hi one doubt, how does client post requests know whether its new or duplicate ? How does it know whether to use the same IK or generate a new one ? Can someone please explain ? This thing only I am not able to understand .

[@raghav_1997]

What if after generating the Idempotency Key in the table as created, server fails to create the resource then how should we handle it? Because the client will be retrying with the same Idempotency Key and it will be getting 409 error as they key already exists. Shouldn't Retries have new IK?

[@theankurpathak]

Hi, You forgot to cover the case what will happen when resource creation failed or exception is thrown during resource creation

[@gauravraj2604]

Hi Shrayansh, thank you for amazing explanation. 1 doubt I had which is related to storage of Idempotency-key and its status. For happy flow, when request reaches server and IK is new, we store it in db with status as "CREATED". And once operation is success, it gets updated to "CONSUMED". storing key into db AND performing business logic for "add to cart" are in different transactional boundary? Can you please explain a bit from this perspective? Also, another case when a new request reaches server with new IK, this IK is stored in db with "CREATED" status but while processing business logic it gets failed with some server error. Now when client retries, what would be the flow? Is old IK still existing in db? case when a new request reaches server with new IK, this IK is stored in db with "CREATED" status and server is processing business logic however request gets timed-out. Now when client retries, what would be the flow? I understood that as per old IK status "CREATED" it will send 409 code. But how would it identify whether previous request has got failed or previous request was in processing state?

[@ConceptandCoding]

Good question Gaurav, generally in. Ase of failure of add cart, it also take care of either removing the IK from DB.

[@gauravraj2604]

@@ConceptandCoding So if I can conclude the approach 1. Request is sent to server with IK in header 2. Server checks if IK already exists and its status 3. If IK already exists with status "CREATED" then we will return 409 saying conflict 4. If IK already exists with status "CONSUMED" then we will return 201 saying resource already created 5. If IK does not exist then we will create IK with status "CREATED" 6. Process business logic for adding to cart 7. If processing fails due to any reason then we will delete IK and return some message to client 8. If processing succeeds then we will update IK status to "CONSUMED" So in case a request gets failed due to some server error, a retry request won't find IK in the table (as it would have been deleted in step 7) and this request would be treated as a new request. Creating IK (Step 5) and (step 6 to step 8) should be in different transactional boundaries.

[@kkiitian]

@ConceptandCoding - Can you please share your view on this approach - what would be the issue in having different transactional boundaries.

[@Seekplay883]

Hi Shreyanshu nice explanation, I have one doubt please find below. Here I understood IK is unique for every request even for retry and parallel also, then how the above solution will work for 'retry' scenario. I just want to know deeply about this, any insights please post here.

[@ConceptandCoding]

Hi Sree, for retry and parallel IK is not different , it would be same.

[@harishaseri]

Hi shreyansh Thank you for another good content I have a doubt . Let consider the retry scenario where client is making a duplicate request as time out has happen for original request. As duplicate request comes and find that someone is already doing this processing client will get 409 error which is fine . Now if some how original request has failed and server will send 500 error let say . then status of IK will be created only right . . Now in this case resource is not created itself and client will never know that resource is not created bcz for original request time has happen and for duplicate request we have send 409 . What do you think about this ? Feel free to correct me .

[@chineseboycott829]

Yeah, to handle this type of case we have to have some intermediate state as well, which will signify that the original request did not succeed, may be due to some error and the duplicate request has to go and do the job (e.g., adding item to the cart) and than updating the status to consumed.

[@gurupreetsingh8347]

Thanks, it's very good and clear but could you explain little bit more about cache solution that you told about video in case of totally distributed systems, how caches system will be able to handle the idempotent of post request? And if duplicate request is coming and meanwhile synchronisation is still need to be happened than what will happen?

[@ConceptandCoding]

Generally cache replication is very very fast, almost instant, so it's not an issue but let's say if cache synchronisation doesn't happen and DB sync up also not happed, then duplicate request will get processed. Hope this clarifies

[@gurupreetsingh8347]

@@ConceptandCoding yes , got it but processing duplicate request is the issue, in this case how we can make sure duplicate request should not be processed.. ? That was my concern actually...

[@ConceptandCoding]

@@gurupreetsingh8347 no system is 100% full proof and same also applied here.

[@gurupreetsingh8347]

@@ConceptandCoding yes, true but interviewer ask this kind of questions too,

[@learning7220]

Sir your DSA questions videos are no longer on the channel, can you please make them available? was really looking forward to watching those videos

[@ConceptandCoding]

I have removed those buddy, bcoz all DSA related things i will put on that DSA Practise channel. I will upload those questions on that Channel

[@learning7220]

@@ConceptandCoding please .. looking forward to that playlist

[@MindwithSoul]

Hi Shrayansh, I need some guidance here. I am new to system design, actually I am able to understand only 40-50% of the concept till this video. should I continue watching the entire playlist and revisit all again or should I start from the begining? Thanks in advance.

[@ConceptandCoding]

no, i would say, continue learning. And slowly pick some questions and try to solve it yourself. and in that question, whatever technology you see which you don't know, then go ahead and read again or cover it.

[@MindwithSoul]

@@ConceptandCoding Thanks, will do the same

[@aryankumar488]

If in Step 6 operation is not success and Status of IK is not changed to Completed. New request with same IK will keep returning HTTP 409, as it will read status of their IK as Created. How will we handle this?

[@ConceptandCoding]

There is a Retention time/ Expiry time added for each IK. worst case if IK status is not changed to Completed, after expiry time, another request with same IK will pass through.

[@tejasshaha6629]

In case of add cart , what if I added 1 item and my mind changes and I added 1 more item within few seconds so 2 different post calls and both are valid , right? So how can we handle this ? Thank you so much for beautiful video ❤

[@ConceptandCoding]

Both are totally diff request and both will have diff Idempotency key. So it should work normally. What is your doubt Tejas?

[@tejasshaha6629]

@@ConceptandCoding Thanks sir. Its cleared now. Just 1 more question- When you are saying client will generate UUID and send it in request header you mean client side languages will generate, right ?

[@dpynsnyl]

SJ, one question - how are we determining if two same requests by same users from two different clients (firefox and chrome) will have the same IK? the session will be different in both browsers right?

[@ConceptandCoding]

In the case of idempotent POST requests, the determination of the same IK (Idempotency Key) for two identical requests from different browsers relies on the generation of the IK by the client application itself, typically using a unique identifier or algorithm, rather than relying on browser-specific session data.

[@dpynsnyl]

@@ConceptandCoding got it. Thanks a lot. ⚡

[@shubhamgupta-bl1tr]

Great session..but how to achieve in coding

[@ConceptandCoding]

It's very simple, if you understood the flow, will try if i can code and share the GitHub link.

[@anjumanislamiachhatarpur7093]

Hi Shreyansh, I was revising my notes and got one doubt for the cache implementation in multiple server. If parallel requests come at the same time on two different server and they read the Idempotent key not present in Cache not in DB they both these requests will pass the check and create the resource. How this is being controlled through our design??

[@ConceptandCoding]

No only 1 will go. Let's say both are going parallel and now going to check in DB. We will be using Mutual Exclusion Locking, so that only one will get the lock and if IK is not present it will create the Row, now second will go and see that Row is in Created state with same IK, so it will not go the same flow

[@anjumanislamiachhatarpur7093]

@@ConceptandCoding does it mean that the locks are implemented on cache level since parallel requests are received on two different server?

[@ConceptandCoding]

@@anjumanislamiachhatarpur7093 even though it's on different servers, concurrency handling (locking) can be easily managed, know as distributed Concurrency handling. Whether it's Db or cache

[@anjumanislamiachhatarpur7093]

@@ConceptandCoding okay understood, I got you I will read more on Distributed concurrency handling..thank you 🙏

[@akshatpandey2609]

make a video on LLD of Online Coding Judge like Leetcode

[@ConceptandCoding]

Noted

[@thecoder7570]

My 2 cents - in this case we cant use uuid nor can combine with the timestamp - uuid will give you different string value for the app deployed in the same machine , if the app is deployed in multiple pods uuid may collide for different parallel requests so UUID is not 100% unique but why we need UUID ? . In this scenario no need to pass any unique ID in the header from client side , It is the Server responsible to define the unique Key based on the API contract that is DB composite primary key or composite unique key or concatenate multiple key fields store in one column as unique key and it should be Transactional operation while writing to DB in case of single/ distributed environment whether it is Dynamo DB , Redis or any RDBMS ( Snowflake , oracle , mysql , etc..) , if key is found no need to insert otherwise insert .

[@ConceptandCoding]

Sorry, but i am not inline with, it's server responsibility to generate IK. What if 2 duplicate req comes in and if server has the responsibility to generate IK, then how server will generate Id for it?

[@thecoder7570]

@@ConceptandCoding lets say as per the API contract we have { firstName , lastName , emailId } so we can make a unique id either from only email id or combination of all three columns . Once the sever has the uniqueId , then concurrency management comes into picture as we have to insert only once irrespective of the number of requests . lets say first time we dont have the key in our DB and you have multiple pods running for the same server application , two or more requests trying to write at the same time then DB unique index will help to create only one record and reject the rest . Once data is inserted the rest of the request will be rejected as part of earlier check , to make it bit faster the Key can be cached in a distributed cache ( Redis ) but it strictly depends on the nature of the project as the volume directly linked with the cost .

[@jaisamtani303]

@@thecoder7570 This solution for Idempotent Key is much better and safe than UUID. I would say creating a HASH of {req.body} as Idempotent Key can be more Unique. Incase of Duplicate POST request, body will be same so, HASH will be same. If we use req.header than I think if source is different than header will be different, so only hashing body can work! For distributed systems, if you have a load balancer, if you use Redis there and do this check and if IK exist, return back else pass on the request to any server, should work right?

[@thecoder7570]

@@jaisamtani303 As you know , Hash code can have collision so it depends , you can refer some of the external HASH algorithm Like HMAC-SHA512 / 256 which generates 64 / 32 bytes hash code , so rare to have a collision and 2nd thing we need to check how big the request size is , you need to consider how much time it takes to get the hash code as you need to convert your request to Byte array [] as an input to the hash function , So design the API contract in such a way that it is better to get the keys directly from the request having one or fewer attributes. On your 2nd point , load balancer is required if the application has multiple instances doesn't matter whether it is distributed or not ( Application Load balancer / Network load balancer / both ) .

[@jaisamtani303]

@@thecoder7570 usually for hashcode you can add epoch time due to which your collision can be avoided.