How to use Postgres as a simple task queue for Django 🐘

[@dojjy5798]

Amazing, freshly baked 2 months ago. Needed it for a company project where we run some automatic script in the background when new task arrived. Thank you! Also benefited a lot from your google login video.(Although I had a GSI_LOGGER error, I saw some commented on your original video which is still not solve, so I am using the old google login API at the moment).

[@tomdekan]

Awesome - great to hear 😊 On the Google login video, I updated the written guide 2 days ago to fix the GSI_LIGGER error (Google no longer allows redirect with Django). Check out the link in the Google video!

[@dojjy5798]

@@tomdekan OMG this is like christmas. Thank you.🔥

[@tomdekan]

Haha - you’re welcome :)

[@TarikCA]

simple and effective 👍 super convenient way to experiment with a new integration (webhooks, data syncing etc) until you reach performance boundaries.

[@tomdekan]

You summarised it well! Glad it was helpful Tarik 🙂

[@GBOAC]

Isn't this a bit like reinventing the wheel that Celery already offers?

[@tomdekan]

Nice question. In some ways, yes - this implements a simpler version of celery. The benefits are that it is much simpler, with no need for Redis or any other broker. Let me know if you've any more questions.

[@franciscoribeiro5759]

@@tomdekan I wouldn't describe Postgres, a complex dbms with 1.5M+ lines of code taking (together with its dependencies) about 1.5GB of software, much higher hardware requirements, etc as simple let alone simpler than celery. The fact that celery can be wired with different backends or brokers says more about its modular and interoperable architecture. In reality, Redis and celery can be loaded as a bundle to effectively work as one so you can think of redis as an optional dependency. Postgres has way more dependencies / possible configurations so you are really not making it any simpler. If you want more reliability and nice monitoring you can use it with RabbitMQ which happens to already be the default. I used to do this sort of thing, the only advantage came from the fact I was already using postgres for my data and so I wasn't really adding much new software/complexity to add task queueing.

[@tomdekan]

​@@franciscoribeiro5759 Thanks for your comment 🙂 In the video, I mention that you can actually use this technique for SQLite (which is even simpler) than both. But, besides that, regarding your main point, I understand your point. However, I view simplicity differently. For me, simplify is not largely based on dependencies, or possible combinations. A dependency in itself is not a bad thing (you could arguably see the python language itself as having many in-built dependencies in its in-built modules!). For me, there is much more complexity when you have multiple services running. E.g., Rabbit / Redis + Celery. In contrast, I view Postgres as very simple to setup, ultra-reliable, and generally easy to use without problems. In a positive sense, it's so reliable that it's boring 🙂 So, I view simplicity and complexity differently to you. But I understand your point and appreciate your comment.

[@Jakub1989YTb]

@@tomdekan Exactly. Nowadays, in the time of Docker, simplicity may be indeed measured in "containers running". I too no longer care, what is being run inside the container. Merely about it size and resources used. Thanks for the video.

[@tomdekan]

@@Jakub1989YTb Interesting point Jakub - and you're welcome!