what is a deadletter queue? (intermediate) anthony explains

[@StaringRail]

This was an easily understandable explanation for Dead letter queues. Thank you so much!

[@optimiserlenergie1094]

Thanks ! That would be interesting to show how you put in place the queue? Rabbitmq ? How does it work ?

[@anthonywritescode]

depends on the queue implementation, for sqs there's a rest api (via boto3)

[@optimiserlenergie1094]

@@anthonywritescode what is sqs ?

[@anthonywritescode]

aws's simple queueing service

[@akshaymestry971]

Hello after a while... this was really an interesting and super helpful video (as always). How did you define the control flow for the items to move to the DL Queue? Is is something we can trigger on an exception? Does it makes any sense to push items off to this queue through the exceptions in some scenarios?

[@anthonywritescode]

for pre-commit.ci my redrive policy is currently: "timeout after N minutes (variable if the run needs to do additional work), after 5 failed attempts go to the deadletter queue". as for exceptions, I do have special handling for that as well -- if there's an unknown exception I change that queue item's delay to be 5 seconds so it will be retried more quickly instead of waiting for the full timeout

[@anthonywritescode]

but yeah, you could do all sorts of clever stuff there as well

[@akshaymestry971]

@@anthonywritescode Ah... clever as usual! You rule! 🤟🏻

[@lewisd4398]

Can you elaborate on your CI setup ; i am very curious as I don't fully picture it

[@pierredonias8940]

Interesting! Can a run have side effects, like writing something in a DB? If so, if the first run wrote something in the DB before it failed, how do you prevent the re-run from writing the same thing again in the DB?

[@anthonywritescode]

yep -- though in those cases you have to be careful to design your work as idempotent (or to realize when the work has already been performed and skip) -- this is also true for queues which offer "at least once" delivery

[@theendlessriver13]

Great video! I always confuse message and item are those the same?

[@anthonywritescode]

yeah in this case they are the same -- there is usually a "receipt handle" associated with the message/item which allows you to "finish" / "extend" / "cancel"

[@Gerald-iz7mv]

can you use a Dead letter queues for both retry and logging? or is the retry only done in the worker?

[@anthonywritescode]

the redrive policy usually drives the retrying

[@Gerald-iz7mv]

​@@anthonywritescode if the worker reads a message from the normal queue - it could also implement some retry with exponential back off? lets say the worker fails to process the message after x retries - it could put the message in the dead letter queue - so it can be retried later?

[@anthonywritescode]

it's certainly possible, though I haven't seen deadletter be used that way -- it's usually for postmortem debugging for failed things

[@kaushikrishi01]

wtf! You use ubuntu in a vm?

[@anthonywritescode]

haha yep --- here's more on that: kzbin.info/www/bejne/bnzHcqSieriEgNE