Radoslav Georgiev - Django structure for scale and longevity

Django structure for scale and longevity
[EuroPython 2018 - Talk - 2018-07-26 - Moorfoot]
[Edinburgh, UK]
By Radoslav Georgiev
Django structure for scale and longevity
Why
Django is great.
But as we add new features, as our dev team grows & the software needs to be stable on production, things can get quite messy.
We are going to look at some common patterns, derived from experience, on how to structure your Django project for scale and longevity.
What
Main topics are:
Django service layer or where should business logic live?
Using Django Rest Framework in a clean & repeatable way & combining it with the service layer.
Testing everything that matters, without repeating ourselves in different tests.
We are going to talk about when to rely on existing abstraction so it's actually helpful & when to avoid existing abstraction, and code things ourselves.
The examples showed in this talk are derived from working with Django in the last 5 years on projects with:
Daily production usage & production deploys.
Dozens of apps.
Hundreds of models & APIs.
Tens of integrations working simultaneously.
Teams of 5 to 10 people.
Who
Key takeaways from the talk:
Increased productivity when developing with Django.
Deeper understanding of the software development process with Django.
Demo project with everything mentioned in it.
The talk is great for all levels of Django knowledge - from beginners to advanced users & teams.
How
The main way of getting the point across is going to be by showing regular code, talking how it can get messy & then following up with examples of improving that code. Hopefully this talk will start a lot of discussion afterwards.
Breakdown of the talk:
Django service layer
Fat models or fat views?
Where do I put my business logic?
What is a service & what goes into a service?
What is a selector & what goes into a selector?
General Django structure
How many apps should I have?
Structuring your code so youр team can be more productive and have less conflicts.
Common modules & utilities.
Doing APIs with Django Rest Framework
Splitting APIs in 2 groups - "giving data" and "taking data"
Using a lot of generics for "give data"
When do to selectors?
Using no generics for "take data" (APIView + Services)
Handling errors from services
Inlining serializers & avoiding serializers reuse
A neat inline_serializer util
Introducing general error formatting for your API
Testing all of that - what should be & not be tested?
License: This video is licensed under the CC BY-NC-SA 3.0 license: creativecommon...
Please see our speaker release agreement for details: ep2018.europyt...

[@Trooper266]

James Bennett, one of Django’s core developers and author of books about Django, wrote two blog articles explaining why these “service layers” are a bad idea for Django projects. What he wrote matches mine experience 1:1 (I use Django since 1.1 and worked on several large projects).

[@samucancld]

Could you share those blogs?

[@12valmirjr]

I highly recommend the reading, Mr Bennett's argument did make much more sense to me. There's a set of "rules" he adopts that do help keeping code organized.

[@Trooper266]

@@samucancld Search for "james bennett Against service layers in Django". I tried sharing the link but YT keeps deleting my comment.

[@jiffey_faux]

Really appreciate your comment. Those blogs were a great read. My instinct from previous jobs is to always separate business logic into a service layer, but he makes a compelling argument for instead utilizing the model manager and query set subclasses. I am excited to have justification for trying this "the easy way" that I've always been discouraged from doing... If a project scales to the point where this becomes a hindrance or inconvenience, there will be so many other more pressing concerns to worry about that it really won't matter.

[@keinermendoza4631]

Thanks for comment, it's really nice to contrasting new ideas with the "voice" of experienced developers. I don't have much experience but there's something with the idea of putting the core app logic outside the models that makes me noise.

[@nikkovellios]

Model managers and queryset classes. Those are often the best place for business logic to reside. Consider reporting. All you're suggesting is taking all the code that was organized where it belongs and sticking it in one or two big mega files of helper functions, then calling it from where the could should be. Testing the API should be done. What if a model field changes? Filter and search fields fail in runtime.

[@KrishnaDasPC]

Brilliant explanation. Thanks a lot.

[@yunir844]

I am grateful for this video, because otherwise James Bennett would not have told what are the drawbacks of this solution. I saw its partial usage in the codebase of my colleague and also tried it in my own pet-project, but it becomes just too heavy to keep consistent in the long run. The idea of James Bennett with implementing it in QuerySets and Managers sounds much more reasonable right now for me. But till this video, I didn't know where to put the business logic and did not know about manager/queryset opportunities.

[@speedcuber-diary]

Did you use HackSoftware guideline?

[@yunir844]

@@speedcuber-diary I don't remember that name in my mind, so I think probably no.

[@codingelle]

make sense 🙏, except the selector havent tried that

[@liorbm1]

Services Pros Provided by the speaker: The general unit that holds business logic together. Service = A simple, type annotated function. Speaks the domain language of the software that we are creating. Can handle permissions & multiple cross-cutting concerns, such as calling other services or tasks. Work mostly & mainly with models. Services Cons by James Bennet: In short: Business Logic belongs in "the model", which incudes managers and QuerySets. I think it just an extended way to write the models file, and to migrate its complexed logic to another file(s)..

[@GuitaristManiaaac]

no

[@liorbm1]

@@GuitaristManiaaac Very detailed..

[@GuitaristManiaaac]

@@liorbm1 I was just fooling around. Just started learning Django 2 weeks ago, I worked in Spring Boot for the past 3 years and the Service layer was a no brainer there. It is kind of confusing for me that there is no such thing in Django 'by default'. In Django terms it really seems that the service layer is just an extension of the Model layer in another file, but I still think it is a better approach even if it is not supported by default. The model should be an interface, and the service layer should handle the business logic, it is cleaner in my opinion. With all these said I am still at the start of my Django learning and things that worked in Spring might not work as well in Django.

[@aksel58]

@@GuitaristManiaaac hm, i have a reverse case - 2 years worked with Django and started to looking closely into Spring because of topic of that video. Why do you moving away from spring boot to django?

[@GuitaristManiaaac]

@@aksel58 Changed workplace and we are using different stack here :)

[@HorstGassner]

Excellent talk and very helpful! Thanx a lot!

[@kurn14nt0]

Nice pattern, thanks to share!

[@elcreidoluis]

This was really useful for me

[@spagon]

Just what I needed.

[@kaloyanmirchev5006]

hah nice there, been exactly on the same conference you mentioned at the beginning :)

[@remy8587]

If you don’t allow to use ModelSerializer in create/update views, then you don’t benefit from any model validation such as CharField’s max_length etc...

[@yujinyuz]

I think that’s why they call the obj.full_clean() to still gain that benefit

[@kaiqiang3947]

very nice talk

[@omarosman7384]

selector: fetch DB service: write to DB

[@JacobKasperek]

This is quite a good and clean solution when prioritizing scalability - BUT - if you're using Django 4fun and small school/personal projects, this lecture should be considered just a "point of view" of professional developer (as video title suggests). In mentioned guidlines Django and DRF are used as frameworks for simply avoiding manual HTTP, SQL and serializing - this does not take advantage of MASSIVE support for fast and reliable development which both of the frameworks provide (and were meant for). As for professional use, this is a very thoughtful solution, not so much for "fun" projects this approach can take away the "fun".

[@muyewahqomeyour7206]

Very well said

[@adrenaline681]

Can someone explain what is the downside of using the ModelSerializer inside the CreateAPI?

[@kennethnwafor719]

According to him, it's just a separation of concerns. As the name goes, serializers are for transforming data to and from Python objects and shouldn't handle the other things like creation. You could have a rigorous object creation process that is needed in other parts of the code and so won't be very good to mix it up with serialization logic

[@soonshin-sam-kwon]

IMHO fat models underrated. Django's ORM is a kind of Active Record. If fat models cannot handle the complexity of business logic in your specific applications, it may means that the Django itself is not suited for your app, especially Active Record ORM. In that case better to choose other framework with other orm approach such as Data Mapper...

[@kevinpaullamadrid9621]

Thank you....

[@gauravbole3789]

Nice talk , but why not we use query sets and model manager. I think this methods are more optimized then service layer

[@rocomatic]

he answers this in the questions section ~38:00

[@yakob-g]

i like this

[@felixmorales273]

Hi, i have a question. How are circular dependencies handled when two services use functions between themselves? Thanks!

[@ManuTomeA]

The Mediator pattern might be a solution you are looking for

[@ibtsamahmad9582]

do it like this instead of importing like "from app.views import UserView" do it like this "import app.views as app_view" then use your view with "app_view.UserView"

[@idiannealvespiresdeoliveir8690]

Where is the source code?

[@funkindy]

Anyone else here has hands-on experience that reusing serializers is bad practice? This part is not quite clear for me.

[@IllevensKO]

I'm a newbie but I already had an experience of changing some little detail in the serializer from which others were inheriting/using and breaking my front end :/

[@oasntet]

@@IllevensKO Sounds like you need more integration tests. I don't think a service layer is going to fix that.

[@heroe1486]

Some useful informations but yeah you should maybe choose the appropriate level of abstraction on a per task basis and not as a golden rule although you're losing repeatability. But with such a reasoning we could also just argue that using Django isn't the right choice and neither is python if extrapolating

[@augustsbautra]

-None of the boxes [given by the framework] are appropriate for business logic. The exact same situation as Rails.

[@heraldo623]

Django is pretty flexible. You are not required to put all your code inside a Model or View. You can wrap all the Django ORM in a Repository and Unit of Work interfaces. Views are just one more input port for Commands, each can be mapped to an Use Case of the application.

[@italobarzola631]

What can I do if my service depends on a lot of validations, 10 for example .. Is it convenient to do them all in service?

[@oasntet]

This really just leads to a big, unanswered question for me: why use Django at all, then? Why not just use Pyramid or whatever, where the ORM isn't ActiveRecord and fighting your every attempt to abstract it away? And the cynical answer, to me, is that you can still say you're a Django company and you can recruit Django developers, despite not actually using Django in any meaningful sense.

[@heroe1486]

@@oasntet And python if we push the reasoning further.

[@oasntet]

@@heroe1486 At the very least, sqlalchemy is much more suited to this sort of use case. It's not ActiveRecord.

[@danilovmy]

i work with Django already 10 years. This Talk about escapism. And don't matter which framework Rado don't want to use. It can be Flack, Django, Tornado, FastAPI... And it very easy to say: i don't like abstractions, i take from framework nothing. I don't wane to use SRRO, and DRY - this is abstractions also.

[@kaiqiang3947]

very nice talk