django Architecture - Connection Management

[@phantomrex1348]

I feel like Hussain is one of the only KZbinrs that actually puts hours into his work. Keep up the good job

[@muhammadahsan5676]

I agree

[@MartinsTalbergs]

Feel ya, men must do work to be men

[@luuc]

'hours' is peanuts when it comes to content creation lol

[@fadious_padious2711]

I feel like he is also one of the few tech KZbinrs who knows his stuff inside and out. There are definitely others but the majority of tech channels are new grads reading snippets out of the documentation.

[@hxxzxtf]

🎯 Key Takeaways for quick navigation: 00:00 *🐍 Django simplifies database connections by using objects instead of writing SQL queries.* 01:22 *💻 HTTP requests to Django connect to the database to execute queries and build responses.* 02:56 *📊 Django establishes database connections upon receiving requests, minimizing connection overhead.* 05:37 *🔄 Persistent connections are favored in modern architectures to minimize connection overhead.* 06:19 *🛠️ Django allows configuring persistent connections with 'connection max age=none' for efficient handling.* 07:00 *📄 Django follows a one connection per thread model, limiting connections to the number of threads.* 09:35 *⛔ Django's single connection per thread model can lead to blocking when multiple clients request database connections simultaneously.* 11:57 *🔀 Scaling Django with multiple threads can improve performance but should align with CPU core recommendations to avoid context switching overhead.* Made with HARPA AI

[@samydriss5223]

Hussein, I love and appreciate all the content you put out. You truly are a rare breed of teacher. I can't express my gratitude enough as well as how much you have taught and have helped me throughout my career. I always learn something new when I watch your videos, take your courses, listen to your podcasts or read your articles. You have this great ability to go very deep into the technology stack without confusing people. Really, not all heroes wear capes. Also, are we seeing a teaser for a Udemy course here? 👀 Salam from 🇩🇿

[@imanmokwena1593]

Thanks to watching this guy last year, I now understand everything in this video lol. Last year, it felt so foreign. Even before he mentioned cores, i already knew where he was heading. Hussein is an excellent teacher.

[@aashayamballi]

Django is ❤ Thank you Hussein for talking about Django and covering about this topic...

[@isurucumaranathunga]

Hussain I don't hv words to say, these stuff are absolutely important and these are the actual engineering stuff. I love this content and really appreciate your effort on these videos. There stuff are really important for everyone specially for fresh graduates like me. Again thank you so much. Watching and noting down your videos is my new hobby. Thanks again.

[@icns01]

Only a few minutes in, I can tell this is a good instructor...Good clear explanations backed by nice diagrams to further illustrate the concepts...Subscribed..👍

[@abiiranathan]

I believe the problem is not with Django per say but python's threading model. Yes you guessed right, the GIL. Also to avoid data races, Django has to execute the queries synchronously. Even using async frameworks like django channel, you must wrap your async coroutines with synchronous helpers. Hello GIL

[@pleli]

I'm guessing multithearding in python deserve a video of its own. Django usually used behind a wsgi server (nginx, gunicorn etc.) In a way you will have multiple processes with single thread on each process.

[@SsaliJonathan]

Thanks for making this video Hussein.

[@MakeItStik]

Really informative video. Make more like these !! We love it.

[@matrixtoogood5601]

Thank you for this amazing video! I would love to see a video on frontend listening models.

[@LOLxUnique]

Great video from one Hussein to another

[@vfryhn2]

Hussein could you please explain how WSGI servers work or why are they necessary for production with the python frameworks and not with other languages like node js or go? I just can't understand why they need that intermediate layer between the python server and the clients

[@MaulikParmar210]

Your inbuilt http server isn't a process manager, what wsgi does is it taps into process management and separates application execution. Allows separating environments, application context and much more stuff that is required to invoke application efficiently and securely while keeping resource usage minimum. Implementation may vary depending on what server you are using but simple idea is, it's webservers responsibility to sandbox request response lifecycle and wsgi provides that barebones for any application and allows it to be build on top of already solved problems when you are writing web applications. This is same as (but not equal to i.e. similar to) using express in node, php-fpm, tomcat / websphere or any other application server, they are server appa helping in management and hosting of application itself while keeping them isolated from host system in one wya or other and providing specific standardised API at your application layer where you actually write business logic.

[@vfryhn2]

@@MaulikParmar210 thank you!

[@Ysh.CS_guy]

Great video 📹 👍 👏

[@Kostas_Panagias]

Great content! Very well explained! Thanks for all the effort you put into this.

[@nucleusnode23]

really good man!

[@_overide]

As always, great video. Thanks you so much, learned few new things!

[@ujjwalbansal5649]

Awsome content.

[@mlsandreas]

Thank you sir!! Awesome video!!

[@dipjul]

Very interesting! Can you please make a video on spring/springboot backend architecture?

[@alexsmart2612]

Hi Hussein, I don't quite agree with your discussion regarding context-switching near the end of the video. Your typical web-server is IO bound and like you explained the worker threads are just mostly sitting idly waiting for a response from the database. If that is the case, why would context-switching be a significant concern? Even if you have a hundred threads, the context switch only happens when there is an actual response from the db server, right? And when it does happen, the cost of the context switch (< 10 μs) is insignificant compared to typical db response times (at least a few milliseconds).

[@syedazzamzafar596]

I am beginner django developer. What course from you should I take first to improve my backend knowledge?

[@malekalhourani5930]

Hi Hussein, thanks for such content, it is really helpful and help me to evolve. I feel like using your old way of explaining stuff (using a PowerPoint presentation) is much easier to follow and make the topic you are trying to explain much simpler, I hope you return back to use it again, Thanks.

[@illichoskypiotr]

Got confused. Isn't WSGI spinning a new forked subprocess every time a new request arrives? How does that relates with this one connection per thread?

[@NavidRashik]

On another note is that from 4.1 it support async orm and views so yah async is supported there as well

[@andresgutierrez1804]

What about asgi with one thread how the Django will handle the transactions?

[@arcstur]

Awesome video!

[@frozeneye100]

Lol. Love way you say the bad kid dB. Really the orm. I am beginner Dev level but crap. I am forcing myself to do ORM way coz want to learn but just SQL seems so much easier than rubbing ORM into compliance. Really ORM is the hardest and doing Django way. But at times the SQL is way quicker than orm... But yeah learning is cool and your vids even better.

[@Tekecthegoat]

so a single thread only handles on cuncurrent user? Am I missing something?

[@mustaphab32]

Great stuff! it think it worthes mentioning that ORM only lately started being async in a very recent version, which means the thread will keep the request waiting and any other calculation as-well! we d love to see how to do parallel processing in python and especially Js and Python!

[@MukulAsija]

Thank you

[@rizwanfirdous]

I highly appreciate your video and the hardwork behind it. One request to you could you please make a same kind of video for hibernate and tomcat. Please...

[@noriking1467]

Is it possible to do spring boot architecture and how project reactor works ? love the storytelling here !

[@adityajoardar578]

I think this is one of the major drawbacks of doing backend processing in django, because if we have to update databases continuously in a background thread, that means we can't use persistent connection as it will keep one of the connections in idle state always

[@bibekjoshi3522]

Was looking at something similar and notification for this video came up

[@CamaradaArdi]

This is my Well actually comment: you have two CPU threads per physical core. Also it's the kernel who does the context switching, not the CPU Also a thread that's waiting for network can safely be idle. As long as it's not doing work. The kernel is smart enough to give it a lower priority. Since read()ing a socket blocks (unless you're doing epoll) you can safely have 100 threads waiting for responses without a significant performance degradation. The problem will be of course if all of those threads are doing work at the same time, where you're absolutely correct.

[@yashvarshney8651]

yep it is the OS's work to schedule and context switch processes

[@DanielTateNZ]

Very interesting, I would be interested to hear your opinions on Laravel a popular PHP framework.

[@TheLolEdits]

fast api next?!

[@riadalimammedov7209]

Hello Hussein,please record fastApi Architecture analysis.

[@MohdImran-uq5iv]

Thank you Hussain.. I am waiting for your 5th udemy course 😅

[@mehdi-vl5nn]

what about async orm (django4+)?

[@justvebsthings]

I have django project and now I hit multiple requests but its fails its only process one request at a time help me to process multiple requests in django parallel processing

[@brightlight1485]

what is role of Gunicorn?you did not mention

[@zeyadmoustafakamal]

Well I am here from 2024 and I heard that in python 3.13 you might don't have to use the GIL if you precompiled the library. So will this affect django or not ?? and also can we just use some rust code to improve this. by (for example) rewrite the template engine and maybe the ORM in rust ??

[@nandusraj7801]

Hi Hussein You said the database connection is only established when the request is received and not on application startup if I understand right. Isn't this the default behavior for most frameworks? You only establish a connection when it tries to do a database IO right?

[@holthuizenoemoet591]

I would love to see a video on a good multi client per thread model

[@fartzy]

So is it not good to use multiple threads? Even though there is context switching wouldn’t still go much faster than blocking I/o? Isn’t that why it exists? GIL or not?

[@101kawsar]

Thank you 🥺❤️

[@9899722801]

Hussein, Can we do data replication in django? I am very confused about it. Some says yes, some say it cant be

[@swapniljena8684]

I'm going to post this here as well as on Stackoverflow. So here it goes: I am running Django app using the WSGI server (gunicorn) on Linux. I am using the local memory for caching inside the application. I am using 3 workers (workers are altogether different forked processes that shared different memory spaces) and multi-threads using g-event under each worker. As the memory space is the same for each worker I'm safe having one single cache in one worker. But the load to the workers is balanced by the OS kernel so if the workers have different memory space so, the application cache 'set' by one worker can be made a 'get' by another worker or will it turn into an issue. I can solve this using unified caching using Redis or Memcached but wanted to clear this doubt before implementing another new service.

[@RajaKhan-dc4vs]

you actually are optimising reads. By having a cache pool and pluging it in with your application leverages the cache to fetch responses faster. at some extent mongo or mysql also uses a lru buffer to bring in the recent read pages into memory and thus saving a lot of disk reads as ordered. As per your specification you want to do it in the previous hop by putting a cache layer in between your database layer and your api layer. But you always have to trade latency with cost.

[@1412-kaito]

So, if in Django there is like 1 client, 1 thread, 1 query, then how does it tackle several (10k like) clients?

[@S--xc4rv]

Qns??? I'm downloading something in my laptop from mobile's network so very interesting thing is happening all the time while downloading. That is , whenever in every 2-3 sec there is spam of 1 sec in which the Mobile network speed goes to less than 0 kb . Why is that happening? And and there is no drop-down speed in laptop. It's normal

[@Namjoonsbonsai]

Can I learn backend without having any knowledge on front-end

[@isegofficial]

Lol of course. I know just Django,I download free frontend templates, then chatgpt helps me write js fetch requests to query my backend api endpoints until I starting doing that myself too

[@Namjoonsbonsai]

@@isegofficial oh Oka. I'm learning django currently. Just started the class for django and I only know basic python by now so I was really scared 😭 I'll learn the other languages once I'm done with this

[@Namjoonsbonsai]

@@isegofficial also ,is there a possibility of getting job after I have full django knowledge?

[@isegofficial]

@@Namjoonsbonsai of course. Don’t let the internet deceive you. Just build stuff. Like 2 good projects and upload on your GitHub and linkdln

[@DhavalAhir10]

​@@NamjoonsbonsaiThere is no possibility, nowdays companies expected Fronted, Backend, Docker, Kubernetes, Apache, nginx, FastAPI bla bla bla

[@Dinesh-babu]

Greatly informative like every other video. What do you think about netty and its use cases??

[@venkatavineelyalamarthy6958]

Is this video added in Udemy course as well. ? On another note, I have a question, if Python is not concurrent (can't run multiple threads at the same time due to GIL), how does Django handle concurrent API requests? Does it use multiple processes.

[@SB-qm5wg]

django have sqlite3 api?

[@amirhmahmoodi4607]

Hussein, what does happen if you put PgBouncer into this?

[@alexsmart2612]

Nothing happens. You are still limited by the number of threads, and since django only uses one connection per thread, your thread is still going to sit idly by waiting for a response from the db server. (PgBouncer would have helped with the first problem mentioned in the video, that is django by default creating a new connection for each request if django didn't in fact have a setting to enable persistent connections, but does nothing to help with the second one.)

[@mikopiko]

Could you do a similar video but for Rails? Would be hella interesting to compare.

[@vishal240993]

Any solution to this or we need to use db proxy like pgboncer ?

[@hnasr]

correct pgbouncer proxy helps this.

[@yashvarshney8651]

use starlette?

[@liorbm1]

@@hnasr i think you needed to mention it..

[@SachinDolta]

Much love

[@charank7852]

Hi Hussein, Thanks for sharing quality content. Can we expect a course on LInux OS fundamentals on udemy as u created for networking ? Please reply. Thanks

[@Fido1hn]

had to speed it up to 1.75 to make it to the end. A lot could be said quicker

[@tobidegnon4409]

I'm sot sure but the way you talk about threads here make me feel that you are actually talking about worker process, not threads, each process having it's own memory, I'm I missing something here?

[@buildmore9829]

You remind me of Firas Zahabi.. related?

[@uEisenhower]

Explaining before you criticise, i get you're making point but I'm learning nothing except disadvantages of django, which isn't the title

[@himanshukandpal5585]

The video did explain the architecture to you.