Carl Meyer - Type-checked Python in the real world - PyCon 2018

Speaker: Carl Meyer
You've heard about Python type annotations, but wondered if they're useful in the real world? Worried you've got too much code and can't afford to annotate it? Type-checked Python is here, it's for real, and it can help you catch bugs and make your code easier to understand. Come learn from our experience gradually typing a million-LOC production Python application!
Type checking solves real world problems in production Python systems. We'll cover the benefits, how type checking in Python works, how to introduce it gradually and sustainably in a production Python application, and how to measure success and avoid common pitfalls. We'll even demonstrate how modern Python typechecking goes hand-in-hand with duck-typing! Join us for a deep dive into type-checked Python in the real world.
Slides can be found at: speakerdeck.com/pycon2018 and github.com/PyCon/2018-slides

[@polypus74]

5:17-6:38 should be required viewing for every programmer. well done

[@lawrencedoliveiro9104]

17:19 It became part of the standard “typing” module in Python 3.8.

[@cr8onpetty]

Very clear, concise, and well rounded intro to typing. Thank you!!!

[@matszczygiel5530]

Great talk, excellent slides, very clean.

[@luvelyvids]

An almost perfect overview of Python type annotations and how to use them to type check Python code. Well done

[@MartinThoma]

Awesome talk! 12:10 typing.overload 17:10 typing.Protocol - docs.python.org/3/library/typing.html#typing.Protocol ... I wrote a complete list of what is said when, including when he starts talking about protocols. Then KZbin autoplay deleted my draft -.-

[@loic.bertrand]

Oh no.. that's sad :'(

[@pyadaaah9866]

Thanks for your excellent, comprehensive introduction to type annotations in Python! It let me become aware of how well the static type checking of functions / methods provided by said type annotations (resp. mypy / pyre and others) and the dynamic type and range checking of variables / attributes provided by Pyadaaah complement each other!

[@xucongzhan9151]

Very good talk and cool slide deck. Thank you~

[@user-vv5lv4cy8k]

I am curious about how the animation implemented in his PPT.

[@lazerbro]

Excellent talk! Thank you!

[@quentinsf]

Very nice, clear, and practical intro

[@liron19]

good talk, thank you!

[@kindjacket]

how was this deck made?? looks awesome

[@kzorbadelos]

Amazing presentation!

[@brambeer5591]

Thanks, really useful!

[@JackQuark]

This is very helpful.

[@SerapioSergiovich]

Nice video shows methods to create a business..

[@hongleijiang]

"PEP 544 -- Protocols: Structural subtyping (static duck typing)" will appear in Python 3.8 ?

[@clip6399]

The slides page is 404

[@ZuLuuuuuu]

So should we write Python always with type annotations now? For small scripts I would imagine there is no need, but can we say for medium/big sized code bases we should always write type annotations?

[@cjm46543]

The value is probably less for small scripts, but so is the investment. Even there I find it handy, mostly as a form of documentation. But certainly I think the benefits are more pronounced in bigger codebases and with larger development teams, where it's more likely that people will have to frequently touch code they aren't familiar with.

[@Auslegung]

I'm not being sarcastic but an honest answer is, "You add type annotations to every function that you want to be made safer." If you have an unimportant 6-line script you may feel ok without the added safety. However, I insist on doing something 100% until I know the rules, and therefore when to break them, so I suggest you add type annotations to your cURL scripts ;) until it's drilled into you.

[@davenirline]

If I were to make the coding standards of a team, very much yes.

[@yvrelna]

No you shouldn't. Python isn't a static language, and type annotations aren't meant to convert python to a static language. There are mental costs associated to writing and reading code with type annotations. For smaller scripts, it's most likely quite redundant to add type annotations because you can see the code being called from just one place, and it's often quite obvious what those types are. Type annotations are meant for the situations where you need to clarify, in the a form of executable documentation, for when things aren't obvious from context. Even in larger code bases, the mental costs with trying to satisfy the type checker and with trying to read code that are full of junk annotations can be quite significant. You aren't paid when you get your types right, you get paid when you ship features. So use annotations sparingly, use it when it helps you reduce mental overhead, and don't fight the battle if you don't benefit from it. I often liken type annotations to comments. If you have to add comments/annotations to make your intent clear, then your code probably was not structured well enough to make it obvious. At the end of the day, sometimes it's just easier and more precise to annotate, and annotations have the advantage of being actually checkable compared to comments, so it's more useful in more cases. However, you should be wary about any code conventions that requires you to annotate everything. As a general rule of thumb, you should try to architect the system so there are clear boundaries different parts of the system. These parts is generally quite chunky (not one class/methods == one part), these parts are usually a constellation of many classes and modules that fits within a theme. Type annotations and checking are mostly useful in the boundaries of these chunks, as it specifies the contract between one part of the overall system and another, but it's usually just noise and distraction within a single part.

[@aoeu256]

Write type annotations if: your function or method is long, you don't have doctests, you don't have default arguments. Also nearly always use Sequence, Number, and str never use int, list for maximum code reuse.

[@BryanChance]

Oh yes!! Can we get rid of the indentation/white space significance too?

[@phudlow]

12:55 What if get_foo is called with an invalid foo_id, e.g. one that is negative / out of range, or with an id of a non-existent foo? In that case the get_foo method would return None. Should the second @overload definition of get_foo have a return type of Optional[Foo]? Manual type checking before referencing the id property of the return value seems necessary.

[@KanadaKid19]

It depends what your real-world scenario is. Your function might automatically instantiate a new Foo whenever one didn't exist with the given ID (like a defaultdict does, for example), but an Optional[Foo] solution is probably more typical.

[@frankt9156]

Why type checking is not dash option on python.exe?

[@archerreilly4881]

git repo don't have 2018 slides, and the upper link can't be opened

[@simonmasters3295]

Funny that: I got here via the slides!

[@Mahorir]

Instead of typing protocol, why not classes implement python abstract base class(ABC)?

[@DrorF]

I'd like to know too

[@sadhlife]

It's for flexibility. Protocols are accepted implicitly, you don't have to extend every concrete class from the ABC. It's also less code to type.

[@aoeu256]

Instead of having to define a protocol/interface for everything can't you just it'd be nice to write the type as r: able[render] or r: _.render.

[@skepticmoderate5790]

Well, then you don't know the argument types or return types from render.

[@AngelHdzMultimedia]

Are Protocols interfaces?

[@sadhlife]

go-like interfaces, yes. implicitly implemented.

[@Zergosss]

As someone else said, there are mental costs associated to writing and reading code with type annotations. Even watching this video is a cost. It's probably worth it only for a small percentage of the code bases, therefore know what type annotations are and why they're useful, then come back when you'll actually need them.

[@TheShanks]

Sure, reading a keyword that indicates a type takes mental effort. However, figuring out the typed from the doc or even inferring it from the code takes at least as much mental effort. And that doesn't even include the mental effort that comes from testing/debugging typeless code.

[@aoeu256]

They are great for when you'll be appending an object to a list.

[@dlf_uk]

Agreed. I don’t mind the overhead for straightforward functions and variables, but if I have to write multiple fake overload functions just to say what the argument types are, that’s horrendous for readability.

[@benhbr]

28:42 Pause for (well-deserved!) applause, alas, in vain

[@zeppelin0110]

Now is it possible to make this work in real time? Can an IDE like PyCharm or VSCode or Visual Studio offer suggestions in real time for the types? If it would be possible to also force the declaration of data types from the get-go, Python development could suddenly become a language that continues to be enjoyable even after your initial script starts becoming a larger program.

[@dachuanlin8625]

maybe pylint or something can do these jobs.

[@chiefir969]

PyCharm Pro offers type checking during code developing.

[@aoeu256]

Another idea is to code while your program is still running in the console by stopping at a breakpoint that way you get more information than just the type of your variables. You can also in some cases change classes and class methods at run time and reload(module).

[@saltrocklamp199]

Pycharm CE now has extensive type checking support.

[@olexandrklymenko]

I’m curious what the font does he use in his presentation.

[@cjm46543]

Titles are Graphik, code is Trim Mono.

[@Auslegung]

whatfontis.com is a great service for this, just take a screenshot of the video and upload it to www.whatfontis.com.

[@knpatel86]

During video, i was curious about the exact same thing and though of searching comments for similar question, and you know what, you did had the same question. lol.. thanks anyway for asking so i get my answer without waiting. :)

[@masakikozuki3214]

Wow, the slides are available on speakerdeck speakerdeck.com/pycon2018/carl-meyer-type-checked-python-in-the-real-world. I was looking for this in GitHub.

[@edwingarcia5043]

Uff It feels overwhelming, but types look useful.

[@doc0core]

In the real world, eh? So here @23:05 we have a Complex class with int attributes instead of float LOL

[@lawrencedoliveiro9104]

4:01 “Item” and “Sequence[Item]” denote types, but “None” is not a type, it is an instance of the return type. Synecdoche, much?

[@DrorF]

But None is a type in python... 😕

[@davenirline]

I use and prefer static languages a lot but I do dabble in Python at times. Why use Optional or Union!? That's just inviting bugs. Why not just use a single type? Makes code more readable and less error prone.

[@bloodgain]

Because a big benefit of Python is its generic programming, achieved through "duck typing." Many statically-typed languages offer generic programming through other means. Java approaches it by requiring explicit inheritance of interfaces, which must be fully implemented -- and at least throw UnsupportedOperationException, which unfortunately, won't fail at compile time. C++ does this through template substitution, which in practice, is like a (compile time) type-safe version of duck typing, and is a more pure version of generic programming than the interface inheritance model. So to avoid forcing your Python code into an interface inheritance-based, truly type-safe model, sometimes your type annotations need to be more generic. You still gain some benefits from using it, particularly communicating your intention to other programmers. EDIT: For what it's worth, the Protocol approach he demos from typing_extensions is basically the interface inheritance model, but more Pythonic. It shifts the burden to the API that requires the interface.

[@user-ip4zu6dp8q]

Postmodern fact: with asyncio, multiprocessing and typing, Python is no more a simple language as it seemed before ten years.

[@yvrelna]

Those are libraries and if you don't want to care about them, you don't have to. In its core, Python is a simple language with a standard library that implements everything. In the core python language, type annotations is simply a syntax to add metadata to a variable declaration. async/await are just resumable functions/coroutine, which is simply a generator function under the hood. They are pretty simple concept at the language level. Most of the magic really happens in the libraries.

[@RobertLugg]

You are exactly correct. Python written following best practices is now nothing too special.

[@saltrocklamp199]

These are optional tools that make it possible to scale Python to larger and more sophisticated code bases. You can still write simple scripts without them.

[@stumblinzz]

The whole god damn point of python was KISS and and readability and to get away from this kind of thing. Im irate because a project I want to inherit has all this garbage in the project and I know its [Facebook PC]Instagram's engineers driving this. Yes your point is valid and I understand why you're doing it, but the implementation readability wise is absolute dogshit. Now I have pointers in python and my code is starting to look like swift and C++ had a baby and I can't tell you how aggravating it is to see this bullshit back in codebase. I feel like this was the kind of thing that Guido Van Rossum would have shot down because it moves away from the original core tenants of the zen of python. Its not complexity that I can't understand that I'm afraid of, its needless readless code that could have been done using the language's native constructs that would have kept this simple. Python is already capable of type checking at the function definition layer, and by implying the type through a default value in the parameters. Even if you are using kwargs or args that only moves your type interpretation down a few lines. This needs to be seriously checked and made into a readable format.

[@thoperSought]

at 13:22, with the two get_foo defs that just pass, followed by the real get_foo, I scrolled down looking for exactly this take on the topic. I'm not a professional developer, but this seems very weird to me-kinda anti-python. why use get_foo in the first place, when you can parameterize a function later? why have three definitions for one function, just to define the type?

[@saltrocklamp199]

You can put type annotations into a separate "stub file" (extension .pyi).

[@davidblewett5703]

You realize that Guido is helping lead this entire project? mypy-lang.org/about.html

[@samdunggawaxy7314]

@@thoperSought yep lost me at that point. I will not support this practice at my company

[@user-fu6it4cm7x]

This is horrible compared to Scala.

[@Kitulous]

scala was a static language since the ground up, tho, but python was, is and will remain a dynamic language, and this is not even a static type checker, it's type hints

[@SerapioSergiovich]

Nice video shows methods to create a business..