The CORRECT way to implement Retries in .NET

[@robertmckee9272]

I've been using Polly for a long time. One of the things you have to be careful of though is when making HttpRequests and getting a timeout. If your request should mutate data, and the request is not idempotent, then recalling the method on a timeout (that actually succeeded) during a retry could cause duplicate calls. Most of my API work unfortunately falls under that umbrella, so I don't set a global retry policy, and handle each call individually.

[@billy65bob]

If it's your own APIs that have this weakness, it's a good idea to do it within a Transactional block, so the work can get rolled back if it times out too early; CancellationTokens are your friend here. And if the call has a point-of-no-return (e.g. because it interacts with software outside the API), I've managed that by requiring unique message GUIDs for those requests. These get recorded in a database so I can't reprocess them by accident, and once processed, I also record the response. So if they time out and resend the request, I just return that stored response instead.

[@Denominus]

Correct, once you distribute your application over the network, you have to start caring about all those wonderful distributed computing problems and timeouts are the one that are often forgotten about (did it complete? did it not complete?). Idempotency and eventual consistency are then the solution but that often involves shifting to using queues/streams with retries rather than something as straightforward as just wrapping http calls with Polly.

[@NiamorH]

Yes it is always a blessing when using a new Api to see that it accepts some kind of idempotencyKey in their Post endpoints. Otherwise I wouldn't recommend at all retrying any kind of Post request.

[@sonicmouse3083]

I have my services split by "Command" and "Query" -- a light-style "CQRS" pattern. I can apply a retry policy on the "Query" services, while I leave the "Command" services to fail fast.

[@jessica_tee]

Love this channel. just started a new role where I am using Polly, had a rough idea of what was going on but not a full understanding, totally get it now, including Jitter which I seen being used but didn't really know what was going on! Thanks :)

[@alexp1445]

Please don't use goto in public. A lot of guys, especially newbies don't even know it exists and this is so good

[@davestorm6718]

Very good to know this info. I do approach things differently: I have a background service that polls external APIs at regular interval and caches the data locally. Then, my API will send cached data to my clients. My polling service would do the retries in the background and if, for some reason, fails, the client's experience is not affected as they get the last cached data. This is great for a couple of reasons: Client experience is not interrupted nor delayed. External calls are only made at predefined intervals - so light on the provider's end (no "punishments" for hitting their service too hard). External API usage costs based on hits/traffic can be reduced greatly, thus saving money: Say you had a 3rd party service that allowed 5000 hits per month for $100, and the next level was 50K hits for $500, and, without pre-polling/caching, you were calling ~30K hits per month. You would have your background poll every 10 minutes, or 4320hits/mo and save $400 every month - right off the bat. It would also save on bandwidth (while making it far more predictable), and with cloud services nickel-and-diming you to death on just about everything, you might save a few hundred more dollars every month on top of that! I love cloud tech, but it's still absurdly expensive and there's always those invoice shockers. Caching data is a good way to mitigate some cloud-sticker-shock.

[@si-fi]

+1 That's good info, but assumes you know in advance what the API call and its parameters will be. Obviously you could use a LRU cache or similar in your service if you don't know in advance.

[@davestorm6718]

I started proxying external services (that i've no control over) a decade ago for security purposes primarily and still do to this day as a matter of policy.

[@timthacker63]

Thanks!

[@DamianWalczak]

I came here to see polly implementation and that is excactly whats there, kudos

[@jonasgranlund4427]

Great content as usual, I remember we talked about the jitter a couple of years ago when you did a video on retries. Regarding status codes to retry on, I think 429 that for instance APIM deliver if you add the Throttling Policy can be good to retry on. Also, when I get the 429 from Cosmos I first retry a couple of time with the RetryAfterMs that the DocumentException property deliver and if I dont succeed I respond to caller with a Responseheader that have the RetryAfterMs value so the Client can retry after a "recommended" wait time. Same with 403 Forbidden, some APIs incuding google use 403 instead of 429 if a client call them to many times in a short period so that can also be good to retry on against some APIs, while against other APIs it can be a waste of resources... Happy Coding! :)

[@HomeSlize]

Polly is one of my favorite packages. I can't image life with out it.

[@int15h97]

Your content is always top notch! Can´t wait to watch your take on the Circuit Breaker. It was the first ¨resilency pattern¨ I learnt and I think it´s one of the hidden gems when talking about resilency.

[@jakesurrett3518]

Love this style of production-ready solution to common problems that are easy to mess up on your own. Same goes for the working with secrets video. Considering how many current and future software disasters you've enabled your viewers to avert. That's more than just good content - it's good karma.

[@dakotapearl0]

I'm really looking forward to see what other videos you come up with on the subject of resiliency. It's so often just retries, but so extra topics would be amazing!

[@andreybiryulin7207]

Really proper way for retries is to implement circuit breaker pattern. Jitter and exponential retries just smooth out load on downstream API, but it almost doesn't reduce number of requests which you will fire into it when it fails.

[@iSoldat]

I've used Polly in the past; simple and elegant.

[@nove1398]

Great insight, i was on the fence about using polly

[@dolaudz3285]

I like that the 0.69 and 420 were not left out. Nice.

[@brianflowers8417]

Me Too!😃

[@GregUzelac]

This is really helpful. I love your enthusiasm and knowledge. Thx.

[@__alexfox__]

Thank you! This is exactly what I need in my project right now!

[@pquesadacr]

I really loved this video... Thank you so much Nick!

[@ArnonDanon]

As usual, great content, easily explained, just wanted to say thanks for that 👏🏾

[@antonmartyniuk]

Nice one! I also like the circuit breaker in Polly package

[@nickchapsas]

It’s coming next week

[@jonathanmarler5808]

This feature will make if faster to write some C# code, but, increases what you need to learn to read that code later. As I get older I tend to prefer code that's easier to read even if it's more characters, at least for long-lived projects. For quick short-lived scripts these features are nice and taken to the extreme you end up with a "write-only language" like perl.

[@stratosmav7993]

Nice video... as always. I was also wondering whether you are planning to make a video about unsafe performance optimizations that can be done. Even though they might not be used regularly, they would still be very interesting.

[@nickchapsas]

Ohh that's a great idea!

[@jpsytaccount]

@@nickchapsas YES!

[@CarrigansGuitarClub]

Excellent video and attention to detail

[@dayv2005]

I think a good thing to mention here is idempotent design and when not to retry or how to make post endpoints idempotent.

[@snapsna3567]

Awesome solution! Keep up the good work! 👍

[@andrewshirley9240]

I'll say that this scenario probably isn't appropriate for a retry anyway. If your API simply proxies another API, and the other API is down, then in reality your service is down too. Delegate the retry logic to the client, because the client can determine what kind of UX it wants. Is it okay with a user waiting 16 seconds as it spins? Is it going to throw up an error message on immediate failure and passively retry in the background? Is it going to stop trying and prompt manual reattempts at some point? They're all fine, but the thing is the backend doesn't need to worry about it at all, it does its one-time proxy and if it fails, it fails. A circuit breaker of some sort may still be appropriate to avoid contributing to an accidental DDoS, because a client doesn't have to respect your retry policy anyway and can just spam requests until you DDoS anyway. There are situations where this kind of resiliency *is* more important, though, and the concepts you demonstrated here + the use of polly is great. Maybe in aggregation requests where your API is calling 4 or 5 APIs and you don't want a transient failure in one of them to kill the entire request is probably the best example. Another popular one is if you expect call chains across dozens of services waiting for your final return, but if that's what the company has built then then murder is probably a more appropriate solution than retry policies. Passive background processing services with no user on the other end are probably acceptable as well, but such a process is probably already on an infinite loop of some sort anyway so it really doesn't matter there.

[@michaelkhalsa]

Nice summary - in short it depends. on what is mission critical and what is not. If someone is trying to implement a transaction scope across 4 or 5 API calls, then at some point murphy's law is going to kick in. Most likely a failure is when you least want it. In that case, I would move that processing into a single API call which internally has its own rollback mechanisms, and simplifies consumption. Identify the mission critical failures and then separating those out. For example, if an order is being completed, and a payment token needs to be finalized as part of that, then 1) Saving of data in a pre-finalization state 2) Obtaining the token (and rollback above it user cancels or fails - if the rollback fails no damage is done except for orphaned data which is already flagged as such) 3) Charging the token 4) Finalizing the order, including notifications, inventory, etc. - The pre-finalization state you can roll back easily if needed, and have the customer try again.. The purpose of this is to lesson the workload in the finalization step, and those less things that can go wrong, as well as providing a final order id when getting the token. - If unable to obtain the token you can direct the customer to try again (and roll back above). - Charging the token is the critical point of failure - Finalizing the order you can recover from if using a queue that records the failure point, so you no the state, and this can have retries for extended period, and administrative solutions if needed. So in this case the mission critical step is charging the token. If a call is made, and it simply evaporates, then additional steps need to ascertain if the token was indeed charged or not. It does not matter if the user needs to wait 20 seconds in this rare case, because if it is not ascertained then it is a critical system error,. You do not want to roll back, because that could make things worse. _ For a weather app, the weather API provider would most likely have a cache, and can do a couple retries if cache does not have an entry yet. If that fails, then they return the appropriate message. Otherwise they could use stale cache until recovered. None of it is mission critical. Then if a client wants to add there own retry logic, it is unlikely to result in a DOS failure of the weather app. -- Just this second, a lighting bolt came, then followed by thunder, and power flashed off and back on (for real). Lets see if this reply makes it through.

[@davemasters]

Beautifully explained 👌

[@Denmaverick39]

Great upload nick was about to search this haha

[@dylanshapiro36]

Could you do videos on authentication and security ? you are the best love your videos!

[@JoeIrizarry88]

Great! Add circuit break and your golden 😉

[@abdulmagedkhaled9480]

Super useful video, thank you very much

[@andreisalagean1241]

Any plans to add a new course anytime soon ? Maybe something about microservices ? They are so popular nowadays and hard to find some good explanation about properly implementing them covering failure cases and scaling.

[@slowMoe2746]

try this : kzbin.info/www/bejne/epi5m3ilaLR6eKs

[@massimopaone91]

Approved!

[@dennycrane2938]

Lesson 1 of that course should be "get really flipping good at CICD" and all brands of automation"

[@MariosKimonos]

we never randomized with the power before and Jitter, we just randomized. I never knew this is called the Jitter even when you already know the feature you can learn something new!

[@Kingside88]

Would you implement the same policy for database stuff? Is it possible to register for example SqlConnection with policy?

[@novak-peter]

with the timeouts, should not we also capture TaskCanceledExceptions? my experience is that usually that's thrown during http call timeouts with httpclients, and HandleTransientHttpError is not covered by that (we also add SocketExceptions as well)... Or are we using the httpclient wrong? (although some of it comes from company-internal libs...)

[@avdain]

As always love your explanations!

[@italofernandes6402]

And what would you suggest if I have an azure function, and I need to await a larger delay between retries, like 2 minutes. I don't think keeping a serveless app awake up to 10 minutes (per request) is a good idea. 😅

[@LDdrums20]

Very cool! Thanks for sharing!

[@levmatta]

This is a very nice topic that does not get much love -- tks

[@voswouter87]

What about maintaining a status for the backing api that sets it to offline when there's a connection problem. You can then start polling until you get a response again and continue serving calls for all clients. Also, you specified internal server error, but that's can be specific to the call. Shouldn't there be an error specific to the api not being available?

[@MrWTFNETWORK]

Do you recommend using an interface to replace the direct dependency on Polly? How would you do it

[@dtownssqwe]

hi can DecorrelatedJitterBackoffV2 be used with AddResilienceHandler ?

[@marcs_matthew]

Can you be notified when a retry is taking place so that you can inform a user than an error is being elegantly handled? For instance, "now retrying 1/x ... 2/x ... 3/x" perhaps with a Cancel button?

[@nickchapsas]

Technically you can stream a response during retrying but the handling of that streamed response is very client specific.

[@martink.7497]

Definitely, depends on implementation. Maybe with hub? I was just logging timeout with logger to see if it is not persistent error in my app and worked like a charm. Cancel would be trickier. Just realized, why not using cancellation token ;)

[@marcs_matthew]

Hmm, an event or callback would be nice, as well as a cancelation token. 🤔

[@itsCalYo]

Quality content, cheers

[@illiaqwerty]

11:05 Why not 4.20 ?

[@nickchapsas]

God damnit

[@kunalr_ai]

Thanks for the video

[@richardaubin1951]

Hi Nick, your httpclient video link does not appear when you mention it should.

[@robertjooste4911]

Thanks for the great video. We have a service (s1) with a similar retry policy, calling another service (s2) that also has a retry policy. The second service is also calling a service (s3), which is the last service being called. For some reason s2 is retrying to call s3 and then s1 is starting to retry calling s2. We are seeing http cancellation exceptions, but can not figure out what the underlying issue is. Can retry policies in multiple services cause issues like this? Thanks

[@Exosia]

Maybe because of timeout between s1 et s2 while s2 retry to call X time s3?

[@mihaiprocopi5207]

Thank you for your videos! I have one question: Can I use this library when successfully request status is defined one response content level, not status code?

[@nickchapsas]

Absolutely. It supports reading the content of the http response body

[@MHashem]

Hii, thanl you for greate videos. I want to ask about dependency injection of my own services. it's pretty easy to use in contorllers. But what if i want to use it in other classes or for static variables ? Is it ok to get the ServiceProvider (app.services) and assign its value to a static variable ?? (eg. TestClass.Provider = app.Services)

[@futurexjam2]

also Polly is not restircited to be used with HttpClient, it can be used for any method call.

[@nickchapsas]

This is mentioned in the video

[@sigma_z]

Polly is awesome. Also been using this since forever. Love the fact you can reduce everything to just 2 lines. Can you now do it with just 1 character? 😛 Maybe with .NET 15?

[@nemanjamilosevic1594]

Great content as always. I'm wondering though, why didn't you use the presented approach together with the circuit breaker pattern? For the example you've shown it seems like a good idea.

[@ek6065]

You managed to smoothly sneak both 69 and 420 into the video. Nice job m8.

[@andregarrido4892]

Hello Nick, loved your advice! But doesn't that breaks the 'open closed' principle from S.O.L.I.D.?

[@nickchapsas]

It doesn't. Open-closed can also be achieved with composition, you don't need inheritance

[@ConnorZ]

Why not wait for 4.20?

[@abhishekshrivastava9856]

Could you please make some videos on Owin, OAuth and Saml Authentication

[@0shii]

Interesting - I have always skipped retries on 500 on the assumption that it could be my input that caused the error. Thinking back to it though, lots of the services I've worked with would indeed return 500 for transient or operational issues.

[@ClAddict]

Why do you have httpClientFactory.CreateClient() in the method and not in the constructor? I missed if the nuance as to why it moved was in your last video. For a high throughput application wouldn’t it be a lot of scoped allocation on that var client variable?

[@nickchapsas]

It's the same amount of allocations. The HttpClient is transient even in the constructor. It's the handlers behind the scenes that are pooled. Check my HttpClient video

[@bnotorious1]

Hello Nick. As always bringing the good content. I have been following you for a quite time and you´re a machine. I have been implementing multiple approaches from other videos in my company, I really wanted to say thank you in the first place. I have one question, would this approach be useful to implement a token expiration to an api call? For example if the token expires, implement the retry policy to get a new refresh token? Or the better one is to implement a delegatingHandler in HttpClientFactory?

[@justarandomcitizen210]

Can polly retry errors that occur while reading the response body? For instance, a connection error when downloading a large blob to a Stream. If not, how would you handle that?

[@qj0n]

it can retry enything, so any exception in a code block will result in calling it again

[@lucaciandrei]

"or maybe another time..." proceeds to silently write 0.69

[@DarraghJones]

It's interesting that the exact same code which is production grade by the end of the video is not allowed anywhere near production at 2:50. I'm curious where you think the responsibility belongs for implementing the retry policy: the author of the client library...or the consumer of the client? Also, I was expecting you to mention the circuit breaker policy which I think would help in the scenario you describe. Nonetheless enjoyed the video.

[@nickchapsas]

Circuit breaker video is coming next week

[@DongoBongKong]

that's a lot of patrons!

[@andriisnihyr6497]

Thank you! Is there something standard for 429 Too Many Requests handling?

[@MushokuThing]

The Retry method has the ability to provide a function to consume the context provided by Polly to determine the sleep duration. This is more flexible because some APIs wont use the standard retry header, and even more so some libraries like Azure Cosmos DB will wrap the recommended wait time into an exception.

[@Martin-kj1od]

In my case I need to get the error results from all the iterations. For example if I am retrying 2 times without success I need to return {errors:[ "timeout", "timeout","timeout"]} from my API. Can I achieve this with polly ?

[@nickchapsas]

Of course, you can get everything about the response body

[@Martin-kj1od]

@@nickchapsas I am not sure you understand my question. Sample scenario: I want to write to file and retry 2 times if it fails due to Exception. If all of my retries fails I want to get all 3 exceptions that occurs (during retries by polly). The problem is ExecuteAsync throws only the last Exception.

[@novak-peter]

@@Martin-kj1od if you just want to e.g. log it, there's a separate handler for that: WaitAndRetryAsync has an optional function which is called on every failure with the details. If you want to collect it for later use - like at the end throw an AggregateException - well, i am not sure about that, my guess is that you have to solve yourself

[@BL0NSKY]

What if my company puts the restriction to the nuget packages that I'm allowed to use. Is there any way to make it right without Polly?

[@jessecavada3015]

You should educate the persons in your organization that placed a restriction on Polly.

[@holger_p]

Checkout the source code, and include the source.

[@karthiksrivatsa8243]

@Nick, could you please make a video on reactive programming and how it is implemented (if it is) in C#?

[@OhhCrapGuy]

If you have clients accessing the API at random times, rather than synchronized, shouldn't that imbue the effective jitter necessary?

[@nickchapsas]

Yes

[@injenius21]

Awesome video as always! I'm building my first Blazor WASM application and wondering if thats a good use case for polly on the backend calls.

[@sadzio986]

Hi, can You make video about installing minimal API project on Linux?

[@prashantkumar15689]

Which IDE you are using?

[@neppe4047]

Rider

[@34823ehwe6w]

Good vid 🥰

[@alexpablo90]

Awesome vídeo Nick!! Love it. PS: in your opinion, what is the best way to test Store Procedures integration?

[@nickchapsas]

In my opinion you should not be using stored procedures

[@justingerber6832]

@@nickchapsas Why?

[@dayanim]

You could also implement the retry inside the HTTP pipeline by the custom handler

[@chrono0097]

Yes you can. The advantage of Polly is that, aside from beignt easy to implement, you can add a retry to basically anything, i use it for event sending, api calls, ftp file writing, almost everything that can fail for a transient error

[@egoegoone]

Hey Nick, Which CI tool do you use with Octopus Deploy? I remember that you use Rider (like me) so I figured it might be TeamCity, but I think TeamCity is maybe a bit expensive if you don't need a lot of fancy stuff. There are a lot of different tools so I figured you may have some good input. Cool video as usual!

[@nickchapsas]

I’ve been using TeamCity indeed

[@94keba]

I always enjoy your videos. But just an idea. Your face cam always hides input in the IDE whereas under the solution explorer is mostly empty space. Then your cam would not hide any content :)

[@mcland1055]

Can Polly be used for APIs rate limit?

[@nickchapsas]

It can! It has a dedicated Rate Limiting functionality

[@mil3761]

Great video, thank you. What pen and tablet do you use for the whiteboard app?

[@CharlesBurnsPrime]

Look at the application title at the top.

[@mil3761]

@@CharlesBurnsPrime or better yet, try reading my post properly. I didn't ask about the name of the application. I asked about the hardware (pen and tablet).

[@CharlesBurnsPrime]

@@mil3761 You are right, I did not read your post properly. No need to be rude about it

[@mil3761]

@@CharlesBurnsPrime that's funny cos "rude" is kinda how your initial reply came across.

[@CharlesBurnsPrime]

@@mil3761 I can see that.

[@marcelius8649]

Wow awesome, Polly evolved.

[@benbenwilde]

Curious no mention of 429!

[@fraysa]

Nick, why aren’t you suffixing your async methods names with “Async”?

[@ibrahimhussain3248]

Convention. Good practice. As soon as you see suffix you know to await the call

[@fraysa]

@@ibrahimhussain3248 I know. I asked WHY he’s not doing that.

[@ibrahimhussain3248]

@@fraysa oh! hehe. my bad

[@sachinkainth9508]

Am I the only person who finds it hard to get Visual Studio to run unit tests in the latest versions of .NET Core?

[@ali1d1]

awesome

[@Youcef8830]

you should 429 codes to the policy as well.

[@1Eagler]

Actually, studies show that a user will wait for 20 seconds before assuming something has gone wrong.

[@dayanim]

Nice

[@fotofoxes2255]

You don’t need that jittering randomization, as it will happen naturally during execution, as there are naturally happening network delays and thread scheduling jitters. Also, it is much better if you use prime numbers as delay intervals: 1, 3, 5, 7, 11, 13

[@MushokuThing]

Jittering is important when you're calling an API with a naturally small response time. Go read the documentation on Polly's pages, because they have done way more research than reasonable on why a smidge of random is important.

[@DarraghJones]

Why are prime numbers better for delay intervals ?

[@fotofoxes2255]

@@DarraghJones less chance for concurrency even if two clients hit the same thing on precisely synchronized clocks.

[@PelFox]

@11.12 missed a 4.20

[@aj.arunkumar]

cool..!!!

[@root317]

Fascinating...

[@michelvosje]

Why not let the server be responsible for deciding on whether to retry handling a request in case it failed? I expect that in most cases the client should not be responsible for that. Clients should not care about why the server failed. There's a response coming from the server and the client should handle that and continue. Timeouts are there to make sure that clients don't wait forever.

[@TechySpeaking]

420, nice.

[@krigrtrue]

Why do this in code? Isn't this something better suited for service Mesh

[@holger_p]

May depend on the type of error. Something like "server busy" could be solved in Mesh, Something like "access denied" couldn't. As more up you go in the level, as more you are able to change the call-parameters.

[@erkan2284]

Your videos are good but can you use less click-bait titles on banners please.

[@nickchapsas]

Sadly, this is how KZbin works. Sorry for that but it can't change

[@loicdurager5724]

complinted tNice tutorials guys voice (that I like as well).

[@MariaCharalambous1989]

0.69 420 😅

[@gargirlofthenight5037]

GTFO w the controllers bro is 2022 we on that minimal apiiiiiiiiiiiiiiiiiiii

[@anatolia23]

When will you stop reminding your API key is invalid?

[@nickchapsas]

Never

[@anatolia23]

@@nickchapsas So irritating. Be pragmatic.

[@nickchapsas]

@@anatolia23 Nah

[@OhhCrapGuy]

If you have clients accessing the API at random times, rather than synchronized, shouldn't that imbue the effective jitter necessary?

[@nickchapsas]

Yes