Why the Plugin Architecture Gives You CRAZY Flexibility

[@ArjanCodes]

✅ Get the FREE Software Architecture Checklist, a guide for building robust, scalable software systems: arjan.codes/checklist.

[@enriquerodriguez9723]

You can make it even more flexible by passing the factory to the 'initialize' method in the plugin from the loader. That way you don't have to import the factory. If the factory changes, you don't have to touch the plugin code at all.

[@ArjanCodes]

Great suggestion!

[@Thristle]

Exactly! plugins should just implement a required interface (preferably also announce what version of the main program they are expecting) but not import anything

[@enriquerodriguez9723]

@@ArjanCodes thanks! Perhaps this video can be a good segway into the topic of clean architecture, by Robert C. Martin, since it uses the idea of building software into pluggable components. You could show how one could implement this architecture in python. Great stuff, keep it up!

[@pokemusicfan]

I mean you always have to worry about the register function of the factory still being defined but good idea

[@Zeedox]

Wouldn’t you lose type checking in that case? The Factory function call is the place where the data class is checked for compatibility with the protocol.

[@SkielCast]

Great video as always! Some suggestions: 1. To avoid the type ignore (15:40) I would make the PluginInterface a Protocol decorated with runtime_checkable. 2. To avoid using global objects, instead of defining the initialize as taking no arguments, I would pass an "App" object exopsing a public API of what can be changed. This may have been too much for this example but it seems like running something without parameters encourages people to use a lot of global vars and polution of the namespace 3. A more strict and parametrizable way of defining callables (5:25) is using ParamSpec or Protocols with __call__. Maybe instead of using plain dictionaries which are hard to type strictly we could have defined another Protocol called GameCharacterConfig and make GameCharacter have that as an example. That would allow dependency injection of configuration while still keeping everything type safe. When adding a new character, one should not only define a CustomCharacter class but also a CustomCharacterConfig that implements the GameConfig protocol. That way the type of that callable would be Callable[[GameCharacterConfig], GameCharacter]. If more strict typing is needed we could use a TypeVar but I believe it is not needed here.

[@deez_gainz]

Awesome! So far you have been choosing examples that fit well to the object oriented programming paradigm, where you got clear objects like Characters or Customers or VideoExporters. Im wondering if you can discuss and come with a counter example where you would rather not use classes but just functions separated to different files / namespaces. Thanks for great work Arjan!

[@ArjanCodes]

Great suggestion, thanks!

[@comradepeter87]

This should still more-or-less work in languages with no-inheritance-OOP, like Go and Rust (all that I know).

[@theRECONN]

Honestly, amazing videos! I've only seen 3 videos of yours so far and I was not expecting such great, methodical explanations followed by actual clean, useful code and advice. Moreover, spoken by a person with, what looks to be, an arsenal of experience and insight. In the KZbin space, there is plenty of bad coding channels, which for me as a software developer is so frustrating. THIS channel is definitely one of the best I've stumbled upon. Keep up the interesting stuff!

[@AlphaWatt]

You are the man Arjan! As an intermediate hobbyist developer, your series are all super super helpful. This plugin architecture layout is beautiful.

[@ArjanCodes]

Thanks a ton! Glad you're enjoying the content!

[@ACAndersen]

About 15:40 - To force the IDE Type Checker to accept your typing, you can use `from typing import cast`.

[@thomasburette9129]

This is an application of the Open/Closed Principle of SOLID. Quoting Uncle Bob: “Open for extension. : This means that the behavior of the module can be extended. Closed for modification.: Extending the behavior of a module does not result in changes to the source or binary code of the module.”

[@JaxxedNesmith]

9:30 don't mix broad exception catching for "is my factory function name in the dict" with exception catching from running the factory function itself. If the factory throws a KeyError it will get misdiagnosed as a missing character type.

[@aflous]

Exactly! easily fixed by moving the creation and return outside the try/except block

[@aliabedi6163]

I really love how you teach an architecture!

[@ArjanCodes]

Thank you, glad you like it!

[@susmitvengurlekar]

For complex / long typing hints, you can make custom typing definitions and use them.

[@cryp0g00n4]

The way you organize code is magical.

[@ArjanCodes]

Thanks, glad you like the video.

[@cryp0g00n4]

@@ArjanCodes yea itz solid

[@resatcanerbas3541]

Many classes out there but very few of them touch the architecture, and very few of those explain it that great. Great job! Thanks for the euphoria per video!

[@NGrimthrie]

Hi Arjan, really appreciate the videos. A suggestion for a video topic is the motivation behind choosing good architecture and design patterns. I've been reading the Pragmatic Programmer and they start with a section about how all of this is motivated by a desire to increase the "ease to change" code. I see that you understand this and as you often reach the point where you say, "ah now it's very easy to change x without changing y" but your viewers may not explicitly understand this. Thanks again!

[@DonGioification]

I think this is a very good point!

[@alessandroferrari2166]

This is neat, indeed! Thank you for providing us with new tools and mental frameworks to expand our software development skills!

[@ArjanCodes]

Thanks Alessandro, glad you liked it!

[@presstv]

the tutorial was awesome

[@hamzahakoun1072]

this channel is a real treasure

[@ArjanCodes]

Thank you so much!

[@k98killer]

I'm implementing something currently that uses a modified version of this to load a plugin from a file path after checking its content hash to make sure it has been validated/authorized. The idea is to make sure that only signed plugin code is run to avoid nefarious code from hijacking the application. (The plugins are actually contracts that specify correspondent time banking relationships, so there is an additional incentive for the system to be hijacked.)

[@FabriceNormandin95]

Hey! I've got another suggestion: Have you covered the singledispatch function from the functools package in one of your videos before? If not, I'd really recommend it! It really goes hand in hand with the plugin architecture too, since it lets you define generic functions and then the plugins can register their own versions of that function to handle the new classes they create (eg new characters in your example). I personally think it could potentially be a great topic for a future video, in case you're interested!

[@ArjanCodes]

Hi Fabrice, thanks for the suggestion, I'll take a look!

[@Ziggity]

"Toss a plugin to your code" 👍🏼👍🏼👍🏼

[@LucasMartins-op7oq]

plugin architecture is very useful to develop modular and adaptative systems. your overview is clarifying and helped to recap it. thanks for the video!

[@ArjanCodes]

Glad you found my video to be insightful! Thank you for the comment, Lucas :)

[@primeix]

I like the background Picture that says Toss a coin to your code!

[@devkhire2989]

This was good on Plugin dynamic loading and creating it. You explain it in very simple language. I was wondering whether you have any similar video on building python framework. Particularly on builds and distribution.

[@sagiziv927]

Seems interesting. In my code, I recently needed to do the same thing (adding variations without changing the existing code). I solved it using inheritance and reflection. I searched for all the subclasses of the base class and in every class I have the name as a static variable. Obviously the main issue is that it requires that all subclasses would be in the same module so they are loaded. I might try to use a factory like you did, which seems much cleaner because every subclasses would be in its own module with its own imports.

[@nandoflorestan]

At 9:17, I see an opportunity to improve the code. The return statement (and calling the character constructor) should definitely be outside of the try block. Always remember to check for this mistake: do not over-catch exceptions.

[@joaopedrorocha5693]

Cool! I think i've started implementing a project with an architecture that reminds this one ... but without the importlib mechanism. Thanks!

[@csavino]

Awesome videos Arjan ! Nice channel !

[@ArjanCodes]

Thanks so much, glad the content is helpful!

[@johanboone2696]

Awesome content, Arjan! Keep them coming!

[@ArjanCodes]

Thank you Johan, glad you like the content!

[@jordvcs1814]

This opened my mind. Thanks!

[@fierce1340]

Awesome video arjan!

[@ArjanCodes]

Thank you, Scott, glad you enjoyed it!

[@pthube]

Thank you very much for teaching design patterns and architecture, I think these are critical components to be a good software engineer..

[@ArjanCodes]

You're very welcome!

[@dedebenui]

thank you so much for your lessons on decoupling and composition, it helped me a lot in my projects and made refactoring/improving components of my application much easier !

[@ArjanCodes]

Great to hear! I'm happy the videos are helpful.

[@josecarlosmendozasosa3109]

If I could I'll give it ten likes. Mindblowing. Excellent explanation. Thanks.

[@ArjanCodes]

Thanks Jose, happy you’re enjoying the content!

[@djl3009]

Thanks for another great video. I liked the creative use of the PluginInterface class to provide useful static type info for a dynamically loaded module (plugin).

[@jakefischer8281]

Arjan! YOU THE MAN!

[@ArjanCodes]

Thank you Jake, glad you liked the video!

[@vorniy]

Instead of type: ignore, you can use the cast function from the typing module

[@ArjanCodes]

Good point!

[@andrzejdubiel3263]

Toss a plugin to your code! Nice one!

[@XRay777]

Really cool demonstration of how to build plugins in python. I am, however, not a fan of the use of Protocols here, since it is easy to miss adding the initialize method. It is also not explicit that the registration in the factory should occur in there. I prefer the more explicit solution of having a base class (say RegisteredCharacter) that implements __init_subclass__ to automatically register its derived types in the factory ;)

[@ArjanCodes]

Great solution! I agree that’s a really nice way to implement autoregistration.

[@virtualraider]

I also have to agree that protocol use is actually making the code more esoteric. I really enjoy your earlier videos when you can see everything tied in nicely in the IDE. But here it doesn't help you auto discover these relationships and requirements. Protocols seen like a lightweight solution that might help when execution speed is of the essence, but it seems like too much of a tradeoff against ease of development. Am I overreacting?

[@nandoaires]

Arjan, do you have any intention of developing some videos on TypeScript (or other languages)? It would be nice to see your approach on those. (The channel is great anyway, congrats and thanks :) )

[@manishankarsingh7748]

Can we use namespace packaging to create pluggable architecture?

[@ShlomoGottlieb]

Great video! I'd suggest adding those mini code project to a github account for everyone to enjoy.

[@ArjanCodes]

Thanks, and good point! I normally put the link to the repository in the video description, but forgot to do it here. It's been added now.

[@k98killer]

Very cool. I'll have to experiment with this design pattern.

[@RemotHuman]

This was very helpful, thank you

[@ArjanCodes]

You’re most welcome!

[@TheWolverine1984]

Could you please point me out to an explanation of what 'Protocol' is and where it's defined? Is it just what interface means in C++ or Java?

[@Phate777]

Hey Arjan, than you for the Content. I'm a big fan and I'm really learning new stuff. Can you make a Video about Project Structure? Like how to name things, what files belong in what folder? Where to put interfaces, dataclasses etc.? I have troubles structuring bigger projects. They start small but then once they are too big they get really messy.

[@ArjanCodes]

Glad you like the content and thanks for the suggestion, Antonio - noted!

[@ajflorido]

Very nice video Arjan! Also just looking for a more pythonic approach, example.py is doing a lot of things (declaring and registering classes) and since this is a plugin approach a "plugin.base" file with all those initial classes (Wizard,Witcher and Sorcerer) from line 12 to 36 could be added and then add it into the json file plugin list as you did with the "plugin.bard". All the initial registration from lines 43-45 are no longer needed as well as the lines from 12-36 as they're going to be defined in that "new plugin.base" file, letting the loader (line 52) to handle it Really good job, thanks a lot for your work

[@cheatoffchannel6320]

Nice work thanks!

[@ArjanCodes]

Thanks happy you’re enjoying the content!

[@Medan1993]

In english translation of Witcher series Jaskier's name has been translated to Dandelion for easier pronunciation and easier reference. Anyway beside thanks for sharing this stuff, will certainly have that in mind when I add new projects in the future.

[@ArjanCodes]

I probably still won’t be able to pronounce that correctly, haha.

[@Alche_mist]

In Czech, the bard is Marigold. Which obviously clashes with Triss's second name when comparing to English - it's Triss Ranuncul in Czech. Note: Czech is very close to Polish and our translator was amazing - Sapkowski himself once (in the far history before the Witcher games and series) stated that only Czech and Polish readers read what he wrote.

[@Tobbzn]

This seems like it would cause conflicts if you have multiple modders wanting to edit the same json file. Avoiding such conflicts would probably need a dedicated directory from which you load every json file rather than just a specific one.

[@k98killer]

Maybe. For Skyrim, there are tools that each compiles a master list of plugins to avoid conflicts, and these use metadata to figure out the load order; if one plugin depends upon another, it cannot be loaded first. This potentially violates the SOLID principle, but at the same time being unable to reuse plugin code violates DRY. Every architecture will have different trade-offs.

[@luersuve]

I wrote a mock for an embedded application using python. Since I wanted to implement a way to change the behavior like you showed in this video I also used the importlib module. However, I required a specific name for the files and functions. It is horrible! I will try the plugin architecture. Thanks for the great content!

[@ArjanCodes]

You're welcome, glad you liked it!

[@ИванНикитин-ч7б]

In fact this is a cross cutting concern thing. Because you can make plugins for any layer not only for domain as in this video.

[@munotpratik2009]

Awesome ! This was really helpful ! I was trying to do something similar for my project

[@ArjanCodes]

Thanks, glad you liked it!

[@raccoonteachesyou]

That's very neat content. Thank you Arjan.

[@ArjanCodes]

Thanks so much! 😊

[@lioneluran]

This looks great and all but, is it just me or these patterns combined make it very hard to create comprehensive unit tests? How would you accunt for your builder behaving as expected with such flexibility while importing modules and classes?

[@fschlz]

This is gold, thanks

[@themilkpirate]

I learned that try block should be as sort a possible. So it would be good an idea, to move the return out of it, after the except in def create(...). Whats your opinion?

[@miguelvasquez9849]

what exactly does "{!r}" do in 09:51?

[@josetapias2339]

It just calls the repr of the value supplied.

[@AnthonyPerot]

really nice! However, I'd register the dataclass using a class decorator which takes character_type as argument.

[@BrianRussell]

Thanks - great video! One qustion: I've incorporated this into my own project which I'm trying to package with pyinstaller. However the packaged program can't find the plugins folder when import_module() is called, even though I've added it through --hidden-import. Any idea?

[@raen714]

Do you have any information anywhere about how you set up your IDE? It looks like you're using Visual Studio Code, but I'd love to know what, if any, plugins you have for it in these videos

[@davidmatten8519]

a better version of importlib.import_module would support generics, to be able to specify the return type explicitly

[@alivecoding4995]

Isn't there any dedicated library for this?

[@ravenecho2410]

definitely had to review this one a bit, but really 😎

[@Empristis47]

Thanks for the content. As a developer coming from a Java background, why exactly do we need copy here? Is it just because we want to pop the type and for some reason keep the original dict?

[@nandoflorestan]

Yes

[@Hubert4515]

very interesting vid, as always

[@ArjanCodes]

Thanks, happy you enjoyed it!

[@brunodepaula5293]

How do i deal with real world implememtation in which i need a way to deal with the dependencies (external libs) that just my plugin will need to run ? Also, plugins will not be in the same path as of my application, specially when I install it as a onefile (using pyinstalller)..so how to deal with the imports? Please could you make a tutorial with a more "real example"?

[@MMarcuzzo]

How does that "raises ValueError(... " Works? From None? That's not usual for me. Thanks

[@davidmatten8519]

wouldn't a better decoupled plugin design not force the new Character classes to know anything about the mechanism of the plugin factory, but rather just have a factory that searches through a designated directory, instantiating any classes it finds (with certain limitations for safety, perhaps), and registering the classes it finds that implement the given Protocol or ABC, and ignoring/noting any others it finds that do not comply?

[@orestesom]

The whole creation of the characters it's pure magic. Congrats! The way the creator_func creates the objects its beautiful!

[@nicopic2381]

Here the PluginInterface class is an Interface working exactly like a protocol ? I find it strange that « class PluginInterface: » and « class PluginInterface(Protocol): » seems to do the same So what the difference ?

[@hajajmaor]

Why you didn't do the GameCharcter class an ABC class?

[@zacharythatcher7328]

I’ve been waiting for this one! I assume that there should also be some test code that generates the strangest plugin that will compile to pyc and sees your code-base fail gracefully, correct?

[@ArjanCodes]

Yes, that's a good way to test various edge cases of what a plugin should or shouldn't do!

[@mcalister1911Ab]

seems like an observer pattern with dinamic load to me

[@twistedsim]

Nice witcher references :)

[@ArjanCodes]

Thank you :).

[@huatsonmx]

most of the peeps here are suggesting, adding, modifying code, FCS, he KNOWS all of it, but he needs to have a base code simple for explanation, just enjoy the masterclass!

[@mahikumar8813]

How do you quickly remove the code? What is the trick?

[@superbroker.dxb007]

Hi Arjan, I subscribed to your newsletter but did not receive the 7step pdf yet

[@ArjanCodes]

Did it by any chance end up in your spam? If you haven't received anything, send me an email ([e-mail]) and I'll make sure you receive the guide.

[@TomTrval]

Ok this video project looks like labor of love :D

[@ArjanCodes]

It certainly is :)

[@thekgul]

it’s good to listen to you

[@ArjanCodes]

Thank you, glad you like the videos!

[@nicolabombace2004]

While this architecture allows a truly extensible framework, how would you defend the host computer against plugins that contain malicious code?

[@fat_pigeon]

Sandboxing. Use a declarative plugin interface, and also execute all of the plugins in a separate, sandboxed process. Essentially, the subprocess only has permissions to communicate with the host process. This requires explicitly defining an API for the plugin to call into, which is a good idea anyway for compatibility (plugins will be coded to an interface that you promise to support across versions, even if you refactor the host app), but reduces flexibility (plugins can only modify your app's behaviors in ways you've explicitly planned for) and takes work.

[@rhafedh]

Waw! Nice one :)

[@fuba44]

My python code is the neanderthal equivalent of your modern day humans python code... But I'm learning 🙂... love the channel.

[@ArjanCodes]

Glad you enjoy it!

[@robertbrummayer4908]

Great video

[@digiryde]

My favorite part. "Toss a plugin to your code."

[@bocckoka]

woo, functional programming!

[@ArjanCodes]

you bet ;)

[@Snookik]

Zelfs als ervaren dev heel nuttig Obligatory: Wanneer komt de TickTick 2015 XNA naar Python editie? :)

[@pariterre]

Thanks for this (again) amazing video! I was wondering, is there a reason why the new Character do not inherit from the GameCharacter class? Thanks again!

[@djl3009]

I think it is because he chose to use the Protocol class to define the GameCharacter interface instead of ABC inheritance. He provides some guidance on when to use the Protocol class vs (ABC) inheritance in another video (kzbin.info/www/bejne/sHjFgZdqj7Vmb7s). I also found the official Python docs explanation useful regarding the rationale behind the Protocol class: www.python.org/dev/peps/pep-0544/#rationale-and-goals

[@nandoflorestan]

The reason is, GameCharacter is a Protocol. And now you have to go read about protocols in Python.

[@budoray]

Toss A Plugin To Your Code. Noice!

[@josetapias2339]

Hi @ArjanCodes and thanks for sharing your knowledge. I download the project, but when I run it with python3 I got the following error: TypeError: 'type' object is not subscriptable in the factory module: in this line : character_creation_funcs: dict[str, Callable[..., GameCharacter]] = {} I wonder what I'm doing wrong.

[@josetapias2339]

I found the error: The error occurs because the code uses "dict" instead of "Dict"; also uses "list" instead of "List". the initial letters in lowercase are the built-in datatype but the initial letters in uppercase are the corresponding type. I made this little change and it works.

[@ArjanCodes]

Glad that you made it work!

[@arcowo]

16:16 instead of "the least horrible way of doing" you should consider it "the lesser evil" 😉

[@RyanJulyan]

Hi Arjan, Thank you so much for this. It is exciting. I am hoping to use it in a project very soon. While looking at your example code, I ran into an error in the loader.py: " def load_plugins(plugins: list[str]) -> None: TypeError: 'type' object is not subscriptable " It was not happy with the function parameter hint for plugins: "list[str]" I was running version 3.8.6 and noticed you were running version 3.9.6. When I upgraded to your version, the code worked perfectly. I was wondering if you could help me on how to find the lowest version of python this functionality would work on is. I am more asking for a method or approach on how to find this out than the specific codebase so that I can empower myself for next time :) Thank you for your amazing content and hard work putting these out. Kind Regards Ryan Julyan

[@adjbutler]

Love it amazing!

[@ArjanCodes]

Glad you like it!

[@badoli1074]

I love your tutorials, you're great at explaining! Just the jump between showing the code and showing you is quite jarring and jumpy sometimes. For a tutorial you're almost doing too much camera work, especially as you're typing really fast (wow!). I'm about half a quick and need to swap between the browser window and my text editor very often to pause the video.

[@maximilian1344]

This was a really cool video! But doesn't this open up our code to malicious code? From my PHP times everything inside me starts screaming when we take a string and pretty much execute it :D. I know this scenario is different because the end user doesn't really see the config file and plugin directory. Nevertheless, we allow arbitrary code to be executed purely by adding a line of code to our config. What are your thoughts on that?

[@ArjanCodes]

This is certainly possible. It’s therefore also important to have processes in place to protect yourself against that. For example, if a plug-in is to be distributed, you can require to check the code beforehand (kind of what is happening in app stores). You can also limit the things that a plug-in can access via the interface that you provide, but since Python has little protection in terms of access restrictions, this is dangerous.

[@malteplath]

@@ArjanCodes Python's introspection features will get you anywhere once the main program calls your code. That does not make the architecture invalid, it only means that you will have to have some process to vet the plugins. Like you suggested. There is no way Python and Python code can ensure 100% security against malicious plugins. (I guess this holds for any interpreted language.)

[@Ryndae-l]

Well, if someone can drop code in your plugin folder, what is stopping them from modifying your base code ?

[@maximilian1344]

@@Ryndae-l simply shipping a malicious plugin from some website and letting the user "install" it, is a lot easier than modifying the base code somehow. But thanks for the replys, everyone :)

[@nostromo9081]

I have not watched the video yet but I am sure it will be exciting. Thank you Arjan and keep on building :)

[@anhnguyenuc6543]

Awesome, I've seen you from the Write better Python series, keep it up.

[@ArjanCodes]

Thanks, will do!

[@nianeddavid7985]

hello, your tutorials sound very interesting to me but i'm having a hard time understanding. It’s interesting sometimes to start from scratch.

[@CrapE_DM]

Interesting. Everything else I've read tries to make their loader search the file structure for plugins, but yours simplifies the loader code a ton by having the plugin be registered in the config file...

[@ArjanCodes]

I often like introducing some kind of intermediate data representation to decouple things, in this case a configuration file. Later on, you can write code that automatically generates the config file contents, but the rest of the code doesn’t need to know anything about that.

[@naveenv3097]

Great game 😁😁😄

[@FabriceNormandin95]

Great video! (This is the third time I try to leave a comment, they seem to disappear as soon as I write them! Or are they getting removed? Am I going crazy?)

[@FabriceNormandin95]

I was suggesting the "entry_point" feature of setuptools, which is a great way to add plugins from outside your package!

[@ArjanCodes]

Sometimes KZbin filters out comments, but I’m not sure how it works exactly. Entry_point is interesting, I’ll take a look, thanks for the suggestion!