Python 3.12 Generics in a Nutshell

[@ArjanCodes]

👷 Join the FREE Code Diagnosis Workshop to help you review code more effectively using my 3-Factor Diagnosis Framework: www.arjancodes.com/diagnosis

[@ponysmallhorse]

after many years working as python developer, I must say that dynamic typing was a mistake. We wen't full circle and now type annotating our code to make it workable. I wish we just added static typing to python in the end, could make python faster.

[@IvanKleshnin]

Mojo will solve the last part.

[@resresres1]

As someone who has mainly coded in Python, in all the program i've made with python, i've never had a problem with type annotations and making sure i was getting the correct types..... I don't see how static typing would make anything better or easier

[@resresres1]

@@IvanKleshnin you don't need to guess though, the type annotations tells you what it should be, so what does it matter?

[@IvanKleshnin]

@@resresres1 then, perhaps I misread your post, sorry

[@nothingisreal6345]

faster is not the main concern. Maintainability and robustness are the real concerns. These type hints are STRANGE. If you read the error messages in your IDE you might be fine. But in reality the code being executed does NOT have any types. So it is still pretty likely that a caller passes arguments to a function that simply can not be handled in a reasonable way. What is especially weird, are type hints for return types. There is absolutely no guarantee that a function really returns the type hinted. IMHO this is even worse as not having type hints. Sorry - but computers do not run on hints... Left alone that the majority of Python developers and libraries don't use them.

[@herberttlbd]

I think we are getting to the point where statically-compiled languages with type-inferencing are starting to look more GvR Pythonic than modern Python. Anybody remember the days when Django was the villain because they were writing Python as if it were Java?

[@alexufba]

Generics reminds me Templates in C++

[@101Mant]

​@@alexufbaor generics in Java or C# or other languages that have been around for decades.

[@tikendraw]

Hey, arjan. Can you make a tutorial on how to use logging in production , what are dos' and dont's when logging in work environment, what function to log. how will it go with async , a tutorial on a real world example? I have seen the logging video on your channel , it is good , but some serious knowledge and real world use case will be helpful to understand the actual use. Make a tutorial when you are free and willing . Thank you.

[@ratulr]

mcoding has a in depth video on this topic

[@rockNbrain]

Generics is great, I use them a lot in typescript 🎉

[@nothingisreal6345]

Type parameters in a language that is dynamically typed and not compiled?

[@IvanIvanov-dk6sm]

After your first video about Generics i started to use it in my project

[@ArjanCodes]

I'm glad to hear, Ivan! Are you enjoying working with them?

[@IvanIvanov-dk6sm]

@@ArjanCodes actually, yes. But some moments are not that straightforward when classes are inherited from Generic-based class. And also when you need to create, for example, a dict attribute of a class. Which keys are types and values are generic-based object instances. While declaring If you highlight type of values inside generic-based instances, MyPy will complains about it. For example: self.my_dict: dict[type[MyObject], CustomSet[int]]. But if i remove int after CustomSet, Mypy will be satisfied. Usually i use generics to work properly with TypeVars when i need to bound main type. Also i used them to create custom OrderedSet[T] and DequeSet[T] objects

[@daviddelaney363]

Looks like Arjan is the new Knuth...good stuff.

[@ArjanCodes]

You give me way too much credit, haha. Knuth is next-level!

[@ravm84]

I just have an impression that implementing all those type-features will make Python even more complex than C++ with far less performance, and still it won't be type-safe. But anyway, this is interesting. Maybe there are benefits I can't see now. I started to use type hints quite long time ago, which makes the code easier to understand, especially with support with IDEs which supports that. Anyway, very cool video.

[@senzmaki]

well type-features are opt-in, you can always just write untyped python but the libraries you use for instance are typed making your dev experience much much better

[@jma42]

most of this will benefit library authors, no more T= t.TypeVar("T") bloat!

[@daze8410]

I suspect it will be a requirement for the removal of the GIL in the future

[@frustratedalien666]

I have worked on a large enough code base with absolutely no type hints and non-existent documentation and I hated every minute of it. I can't figure out what argument I am supposed to pass to these methods, so each time I touch these legacy repos, I make it a point to add type annotations that seem rational to me. Makes my life easier in the long run. You don't have to use it, though, if you don't like it.

[@Naej7]

I don’t see how it could be more complex than C++, nor be less performant

[@diegol_116]

Good video! It's a shame that this syntax hasn't been there since previous versions, it's much more readable.

[@dawidskreczko]

I love your content. Keep going!

[@ArjanCodes]

Thank you so much Dawid! ❤️‍🔥

[@TJ-hs1qm]

I think Python's dynamic type system is its selling point. It allows for pretty elegant code, ex. when programming dynamic plugin systems. But the type system is extremely limited for anything beyond standard linting. Coming from Scala and Rust, I was even surprised to see how the syntax knows all about covariance and contravariance types.

[@tulliolevichivita5130]

The best case of using generics for me was my own implementation of a B+Tree. I think the next step in python must be an implementation kind of a standart library like C++ has.

[@berdeter]

Well I come from the Java world and adding a bit more type checking always feels better for me so this feature is important. It's always a bit wierd that type errors in python are only actually warnings for your IDE.

[@ravm84]

Yes it is just a warning, and you must use an IDE which understands that or perhaps some type checking tools... I just wonder, if you employ all those type-checiking features why not use C++ or Java instead. You will get real type checking and far better performance.

[@NicolasChanCSY]

@@ravm84 If I understand correctly, the point is to have gradual typing (so this can be compatible to all Python code because they don't want another Python 2-->3) and development speed (Python is still strongly typed in runtime. Python developers can still write their scripts quick and dirty without caring any typing stuff, while providing an option in case some projects want to get good code quality via type checking and better autocomplete).

[@markuswagner5754]

​@@NicolasChanCSYin the end the choice of programming language depends on your use case. Python is good for quick and dirty scripting. But type annotations can help, when projects get a bit bigger and it becomes less transparent, which types you have to expect. I also find type annotations quite helpful for writing python libraries, as it reduces the amount of assumptions your library user will take and you may not thought about. Nevertheless for a high performance program or highly robust application Python maybe is the wrong language.

[@guruyaya]

I used it (in the old py10 syntax) to create a Pydantic collection type, that puts the return list of values into an attribute called "data" Why? I found that a lot of json parsers don't like to get a list as a reply - they expect a json objet. This solves the need to create a custom object for every time I want to return a list of multiple object

[@Alticroo]

I really do not understand the aversion to type hinting. I cannot stand working on codebases that have mystery functions, arguments, etc... IDE intellisense alone is worth it; At the very least, this largely improve productivity and interpretability. This isn't about solving runtime bugs by hard enforcing types; it's about avoiding those situation entirely.

[@glensmith491]

I think I now understand why I never understood generi type. Because they are not generic types but are generic types with value added. For instance, a generic drug is simply the basic drug sold without error checking (or at least minimal error checking), no extensions and no cool packaging.

[@orlanino]

This is ok when one uses a basic type like int, float, etc. But what if I want to build, let's say a Stack[myCustomObject]? What do I have to implement in the class? I assume __eq_, at least. Probably iterator as well?

[@ALINcibi-s6u]

Are there series about functional programming ?

[@ButchCassidyAndSundanceKid]

I prefer to use SingleDispatch. However, Python still doesn't support MultipleDispatch natively (have to pip install another package), it'll be great if they could include that in the next version.

[@LucasAlmeida-ef9io]

SingleDispatch is used to implement polymorphism. Think about two different classes with the method "add". SingleDispatch will be applied in the "self" parameter of this method, turning possible to behave differently depending on the type of self. So it's kinda low level thing that is used as workaround to implement function overload. Then, I think that it will not be implemented natively soon...

[@heinereniscaicedo7510]

What a Great video! new Python's Generics are so similar to golang's Generics

[@twelvethis3979]

Unfortunately, I cannot use generics like this at the moment. I would love to, though. My company supports only Python 3.9 and it'll be five years - I guess - before I can use Python 3.12 😅

[@Naej7]

I feel you

[@shlomokallner3180]

❤ i am stuck on 3.8

[@julians.2597]

@@shlomokallner3180 given then 3.8 is going to be End of Life even for security patches before the end of this year, meaybe you can motivate them to at leaste make the horrifying jump to 3.9?

[@evgeniy_toropchin]

yeah, another python type hinting which we will use only for ide :D so cool

[@DanielCordey]

I think I will use this feature. It's a safe approach in some situation, but I'm not going to generalize all of my codes with it;

[@ArjanCodes]

That's a good way to go about it, Daniel!

[@gaaligadu148]

What if there was a global variable named "T" in the code base ? I know that we aren't supposed to name variables like that. But would the generics still work ?

[@jeanhadrien]

What is the difference with just passing a type argument in the init method of the class?

[@cheukmingau983]

Say you have 2 instance of the class, 'a = Stack[int]' and 'b = Stack[str]', we can know that a can only stack integer but b can't, it can only stack strings. With type annotation we can leverage static checkers like mypy to verify whether our code is correct or not. If we pass the type in the constructor, as the attribute is defined in the instance level, we cannot annotate the correct type in the class level and the static checker or even inteliisense will not work. Also, specifying the type through generics is like a 'meta-configuration' where we can specify type alias, say in Python3.12 we can specify 'type IntStack = Stack[int]' or 'IntStack: TypeAlias = Stack[int]' in previous versions. This configuration works before the initialization stage thus another level of configuration.

[@jeanhadrien]

@cheukmingau983 Oh, I did not think about intellisense/type checking. Good point! Thank you 😊

[@ravenecho2410]

Interesting items in python i encounter pydantic - awesome for json validation Langgraph, langchain - do pretty well for api abstraction their pydantic parser is pretty good Sqlglot - pretty interesting capabilities for transpiling and inspecting abstract syntax trees NamedTuple - simple class to clean up access Dotenv - easy way to get variables Sqlite - built in DB which obeys the api pattern Sqlalchemy - an ORM which can be nice Mypy - static type checker Then i would like to point that a module with functions is exactly equivalent in access and calls for a class without any data I think finally, TDD (test driven development) will play a MASSIVE role in Artificial Intelligence and Machine Learning based features. Theres absolutely no way around it... i think😅 Great vid as always!

[@aaronsteers]

Hi, Arjan. Whenever new typing features come out for subsequent python versions, I always feel left in the dark about how/if they'll be backported with the "from __future__" magic - and whether prior versions of Python will at least tolerate the syntax, if not directly support it. If so, I can use them. If not, I have to wait a few years. I can never really enjoy videos like this unless I have those questions answered - otherwise, it's just another homework assignment for me to add to my backlog. Can you answer this for me?

[@NdamuleloNemakh]

What is the difference between using len(seld._container) == 0 vs self._container == [] on is_empty()

[@HandcartRule46]

Super useful, thanks!

[@ArjanCodes]

Glad it was helpful!

[@kiraleskirales]

The Python community is giving low priority to supporting generic variables a.k.a. Higher Kinded Types. It is currently impossible to annotate an operation that keeps a parametric container as is, but changes the type it contains (the package "returns" offers this functionality but you need to add a plugin to the mypy checker). They (Guido Van Rossum himself!) say that it is a lot of work and it is not worth it. I was surprised to read that because even simple operations cannot be fully specified in the annotations. What is your take? Should operations like that be avoided in the first place? Will they ever support them?

[@rikschaaf]

What about covariance and contravariance? So let's say I have a list[Cat] containing several cats. Can I assign that to a variable of type list[Animal] or is there some sort of way to define the type list[object extends Animal]? (or some similar syntax). If there isn't co-/contravariance modifiers, does that mean you can then add a Dog object to that list[Animal], causing actions on the original list[Cat] to break whenever Cat-specific methods are called or Cat-specific fields are accessed?

[@cheukmingau983]

In Python you can call whatever attributes / methods you want, just whether or not it throws an AttributeError. For your case of adding a Dog object to that list [Animal], normally if you follow the intellisense and static type checker, you would not be able to call Cat specific methods that Animal does not. When you annotate your variable as list[Animal], you have already told the static type checker that you only consider it as Animal, thus it will only allow (at the static type checker level) Animal methods. If you don't care about the intellisense / static type checker stuff, you can always call any method you want and result in AttributeError

[@NicolasChanCSY]

FYI, on top of the reply above, I would like to supplement a few things copied from PEP-0695 (proposal of this new syntax of generic type) regarding variance. The PEP writes "This PEP eliminates the need for variance to be specified for type parameters. Instead, type checkers will infer the variance of type parameters based on their usage within a class. Type parameters are inferred to be invariant, covariant, or contravariant depending on how they are used." (An overview of the algorithm of determining variance is in the PEP if you are curious.) `TypeVar()` also has a new `infer_variance` argument in Python 3.12 besides the original `covariant` and `contravariant` arguments. In the section "Explicit Variance" in PEP-0695, the authors explained why they rejected it: "We rejected this idea because variance can generally be inferred, and most modern programming languages do infer variance based on usage. Variance is an advanced topic that many developers find confusing, so we want to eliminate the need to understand this concept for most Python developers." Similar writings can be found in the Motivation section as well.

[@marckiezeender]

variance is inferred based on the signatures of the methods you define in the generic class. If there are methods that return T, then T will be covariant. If there are methods (other than __init__) that accept T as a parameter, then T will be contravariant. If both are true, then T is invariant. The type checker will determine this automatically.

[@rikschaaf]

@@marckiezeender but I can think of situations for parameters to be either-or. In java in a list, the addAll() method parameter would be Collection

[@marckiezeender]

@@rikschaaf It still infers the variance, based on the variance of those parameterized generics. I should have written "etc.," because the cases with the param type and return type are only the simplest examples.

[@littleconan7929]

The main issue I have with it is that it depends on modern typing that requires import. Docstring typing does not require import and thus avoids circular imports. but you cannot do everything with it.

[@BambuzMitZett]

Google: if TYPE_CHECKING

[@marckiezeender]

you can guard your imports with `if typing.TYPE_CHECKING` if you want to avoid circular imports. (also requires __future__.annotations until 1.13)

[@littleconan7929]

@@marckiezeender Wow, I did not know this statement existed. Seems great !

[@marckiezeender]

@@littleconan7929 Yeah, I use it all the time, to the point that I put ALL my annotation-only types behind the guard, just for consistency.

[@ReRubis]

Great video as always. :)

[@ArjanCodes]

Glad you liked it!

[@ZapD-q3k]

So python is now fully OOP and also fully procedural?

[@R15YZF]

Thanks for sharing! In Java what makes generics even more useful is to restrict the generic types to an interface. Ex. Numerics can be an interface which is implemented by both Int and Float. Is such a thing possible with generics in python?

[@marckiezeender]

Yes. You can use typing.Protocol to define an interface. Also, many abstract classes in the python standard library also behave as interfaces. I believe that Arjan as a couple videos on Protocols.

[@Naej7]

You can create a type which is equal to int|float

[@shlomokallner3180]

In pre-3.12 (i am currently using 3.10 and 3.8) enabled that on the 'TypeVar' construct via the 'bound' parameter. I use all the time.

[@WilliamWatkins-o6z]

Frankly at that point, just switch to rust

[@taraskuzyk8985]

I wanted to use generics but PyCharm currently offers no proper type hinting for them :( Very sad

[@GuihVicentini]

Why have generics in a non-typed language?

[@yickysan]

I use generics all the times in python and also typescript.

[@resresres1]

just curious , but what kind of stuff do you code that makes you want to use generics? I've made a lot of python programs for my company, usually around automation and mainly using win32gui, pandas, matplotlib, pyqt6 and sqlite3 libraries/modules.... but never found where using things like generics would ever be useful.

[@yickysan]

@@resresres1 I use generics mainly with collections like lists and tuples. Say I want a function that takes in a list of any type, then does something with that list, I use generics in cases like this.

[@resresres1]

@@yickysan ah I see, I usually just end up putting code into the function itself to do stuff with the list and explicitly transforming the type when I transform it. However I could see it being worth it do like in the video if it's something I would need to do in other functions as well

[@natnaelberhane3141]

At this point, it's just better to use a statically typed, high level language like Java or C#

[@rafiullah-zz1lf]

Love the way u teach. But cant afford the $ for the course.

[@sinasalahshour]

this is amazing

[@ArjanCodes]

Thank you!

[@smalltimer666]

So basically python wants to be Julia now ? I thought the only (and perhaps big) advantage of python over proper languages was the quick and dirty dynamic typing. Without that, python is just a slow C++.

[@jamesclarke7259]

Love this, could i request more jenkins please?

[@deathineyes]

Typescript in Python?!

[@hdtlab]

After many years, it’s sad to see Python is going to same route with C++, ton of features and syntactical sugar that essentially do the same thing.

[@Naej7]

People rejecting types annotations really scare me. It’s intellectually wrong, and I really don’t want to work on their codebases, it seems like a nightmare.

[@maleldil1]

I've noticed in Arjan's videos about type hints that you cover the surface well, but minor details suggest that you either don't understand things properly or you don't use them in practice. For example, if you're using Python 3.12, there's no reason to import `Optional` from typing. Instead, just use `T | None`. Also, using `Literal[0]` for your `sum` method doesn't make any sense. `Literal[0]` doesn't mean that the value might be zero - `int` already means that; instead, it means that you're explicitly writing down the value `0` in your code, which you aren't. This makes no sense for `sum` since you're never actually writing the value `0` in your code. In general, `Literal` is very rarely used, and is usually only helpful in adding types to existing APIs. Also, there's no reason to explicitly annotate the local variable `total` inside `average` since its type can be inferred to be the same as `sum`'s return type, even if it's generic.

[@andreacazzaniga8488]

such a limited use case that the chances of abuse are far greater than normal implementation

[@Deadlious]

These generics seem a mess. They do not feel intuitive or pythonic. I guess they can bring some positives if you are maintaining some sort of public library that handles all types of inputs or if you are working on a project with an extremely large codebase, but honestly, I don't think I'll ever use them.

[@kayakMike1000]

Here i thought it was Gen-ER-ics.

[@Nalewkarz]

I don't like the direction of typing changes in Python. Something wrong is happening in Python lately. I like type hints but it's going too far in my opinion. I mean generic types are OK but using construction like (int, float) instead of Union is just one big tragedy. Python core developers are making language for describing the code rather than just typing, so in the future in some cases You will have to know Python, and in other cases Type hinted Python will be the requirement like Javascript vs Typescript. In the same time there is no type checker in standard library. It's one big mess lately.

[@MarioDanielCarugno]

Please don't let Python become a monster like C++ !! Looks like it is starting to accumulate features and everytime you have more ways of doing the same things I really don't understand why languages are inexorably updated with new features... is it something mandatory or what ?

[@muditjha7235]

Agree 100% I wont be surprised if they make type checking a core part of the language.

[@MikhailLalyko]

Interesting, but I don't understand how to utilize it in real life :)

[@Naej7]

Generics are only useful for low level needs, where generalization in needed. Most of the time, what you must code is specific, not generic.

[@iankaranja7765]

Doesn't all this type obsession defeat the purpose of python the dynamic/duck typing language.

[@NicolasChanCSY]

It is optional. And type hinting can provide useful context to both IDE and code users what are the inputs and outputs of the functions and classes. Instead of reading the reference documents, you can just read the function signature to have a better idea.

[@marckiezeender]

Well, for one, duck typing is just the dynamic-typed version of interfaces (called Protocols in python).

[@Naej7]

No, it’s complementary

[@TheStickofWar]

I just wanted to point out that the thumbnail is a bad choice. If you removed the type hinting then it would still be just as generic.

[@Netum6am]

But can I have code like this? class Factory[T]: def create(self) -> T: return T() If not, then there are still no generic types in Python. Only a generic annotation.

[@marckiezeender]

@dataclass class Factory[T]: t: type[T] def create(self) -> T: return self.t() value = Factory(int).create() # value is known to be an int.

[@Netum6am]

@@marckiezeender I am not asking how to write a factory. That was just an example. My point about generic types stands. I know there are solutions to this example. I could also write a factory functions that creates factory classes. I could write it as a decorator. I assume there are many other solutions. Nevertheless, I had cases where type checking still failed me even with these workarounds. In my usecases I don't often need a generic object like in your example. I really need a generic class. In those cases, mypy often spetacularly fails. Although that said, it seems to sometimes fail even in some trivial situations.

[@marckiezeender]

​@@Netum6am The code I gave you behaves exactly how you want your code to behave. Both at runtime and during type checking. Factory(A) is is an A factory, and calling the create() method returns an instance of A. Also, using an arbitrary decorator on a class is currently not supported by mypy; it only recognizes @dataclass - like decorators.

[@Netum6am]

@marckiezeender Please, stop answering something nobody asked. I find it a bit rude. My original comment is a simple yes/no question. Does the presented code work in python 3.12?

[@marckiezeender]

@@Netum6am I don't mean to be rude. No, the syntax that you made up is not valid python syntax. But there IS slightly different syntax that allows you to do the same thing. I know the Factory is just an example, I was building on the example you provided.

[@UNgineering]

Oh no, python is becoming TypeScript with all the syntactic complexity

[@maleldil1]

Generics have been available for almost 10 years, since 3.5, via TypeVar and Generic[T]. This syntax is much better than before. Anyway, Python's type system is still very far from TypeScript's complexity, as you can't do much meta-programming with it.

[@nick.59]

I don't know man. If it wasn't for type hints I wouldn't ever use Python

[@UNgineering]

@@nick.59 I love type hints, it's when we get into generics with abstract factories, that's when the syntax starts to get in the way of development.

[@Naej7]

@@UNgineeringDeveloping abstract factories without any type is suicidal

[@UNgineering]

@@Naej7 I like to live on the edge

[@trns-7]

More and more Python are becoming similar to Java 🙄