Raymond Hettinger - Super considered super! - PyCon 2015

"Speaker: Raymond Hettinger
Python's super() is well-designed and powerful, but it can be tricky to use if you don't know all the moves.
This talk offers clear, practical advice with real-world use cases on how to use super() effectively and not get tripped-up by common mistakes.
Slides can be found at: speakerdeck.com/pycon2015 and github.com/PyCon/2015-slides"

[@bbckyotoku]

"Because his talk is on concurency, so I presume he is giving his talk and listening in this room at the same time" 20:50

[@kopuz.co.uk.]

I love Raymond Hettingers Python talks.

[@zes7215]

wrg, no such thing as power or done or not about it, no nerx

[@jonathandaniel7321]

what could possibly go wrong?

[@isodoubIet]

No accounting for taste I suppose, this is probably the most incompetent programming talk I've ever seen.

[@CodeOptimism]

Browsing Raymond Hettinger talks since they're awesome. Clicked this one thinking, "eh it's old and just about super() probably won't be very interesting". Hah! Amazing talk.

[@Rahuls-Insights]

This is great!!! Makes our life simpler!!! Thanks Raymond!!!

[@kenshinnanashi9469]

Very interesting speech. Thanks a lot !

[@nilindef9270]

Really helped. Great talk. Thanks!

[@swadhikarc7858]

Absolutely astonishing

[@DanVanderkam]

Great talk! Raymond suggests using multiple inheritance and super() instead of mocks in unit tests. Is this how mocking libraries in Python are typically implemented?

[@AlienValkyrie]

holy shit. When he gave the robots example, I thought "hm, yeah, this is pretty nifty, I guess"; then he brought out the OrderedCounter class and mind = blown

[@VCR47527]

Few questions about DI in this style. 1. How do you handle injecting a class that itself needs dependencies injected? Does the entire app become one class eventually? 2. How do you track down everything that can be injected and what method signatures are expected for each? 3. What happens if an interface is implemented wrongly and has a wrong return type? 4. What happens if two different dependencies that are both required happen to have the same method names? 5. Looks like you choose concrete implementations at the time of class declaration instead of within a configuration file (which governs class instantiation later on) like the DI libraries that I'm used to. What if I don't know some of the concrete dependencies at the time of class declaration of the base class and I only know that it will follow a certain interface? DI is usually a tool for class composition. In other words it deals with "has-a" relationships rather than "is-a" relationships. So it's really strange to see a language feature that facilitates class-based inheritance being suggested as a tool for DI. The result seems to be similar to a mixin pattern for sharing functionality (with all its criticisms) with extra attention to the MRO navigate the confusing nature of this language feature.

[@andreienache6290]

1. The same, just inject previously into the class you need. No, the entire app does not become 1 class. 2. Not sure what you mean by 'track down'? Don't overcomplicate things, ptyhon design is different. Don't bring here fancy terms like 'method signatures', they don't make sense in this context. 3. Change your code. 4. Good point. The method resolution order will resolve to the first one encountered if you use super with no arguments. Super can be used with arguments that direct its resolution. So, without arguments, it resolves with the c3 algorithm. With arguments you can call 1 or both in the order you need ( In that specific case you do introduce a little redundancy, but you have no choice). 5. Again, python is more abstract. An interface is just a contract or a declaration of behavior. You could just class Interface: def act(self): pass class Imp(Interface): pass You could use the ABC (Abstract Base Class) module to decorate and express and force the interpreter to put some restrictions, like, not implementing Interface.act method. Don't let them confuse you with fancy words 'implements' 'extends' and 'injection', behind the scenes there are just objects and pointers. In Python, I don't care if put my dependency inside the parenthesis of a class declaration if I'm given the dependency at the instance creation level or if I fucking give a framework a yaml file that declares my dependencies and makes the symbols available at the necessary scope. It's a matter of understanding the best approach for you. Again, in this case, is trivial, discussing the 'is-a' or 'has-a' thing is a waste of time and is inherited by old paradigms. The difference is just this: class Dependant(Dependency): # what you call 'is-a' pass or class Dependant: # what you call 'has-a' def __init__(self, dependency): self.dependency = dependency or, if you want a framework: dependencies.yaml Dependant Dependency -- Then in your code: Dependant().dependency.method_on_dependency() Do you see how dumb it is to waste time discussing this? It's nice as a learning exercise but at the end of the day, why the hell i would go and create a file and install a framework if I can just write class A(B): pass? What is the advantage of expressing this particular case in a file instead of in the code file itself? If you find dependencies later, just go and add a new dependency to your class Instantiation (Yes, classes in Python are first-class citizens). You are too biased. If your dependency is against an interface, and you know it, then add the interface. If you don't know it, it is not a problem with the language. That's it. When you call the method it will just do nothing, or again, you can decorate to enforce classic oo behavior like raising a NotImplemented error and stuff like that, or even raise a not implemented error. Python is very abstract and that is why it gives you so much flexibility. These ideas become clear to you when you become older. You need to work in too many different areas and technologies to understand all is reduced to a matter of convenience and automation. Some principles should guide you, but these are not dependent on the technologies. Behind the scenes, all is reduced to objects and pointers. How do you think a class Inheritance is resolved inside the compiler? So, for example, they say "Favor composition over inheritance". Ok. That may be good for Java | C# | C++ but, what if I have a language that has a method resolution and leaves me to declare my dependencies at class Instantiation level, I mean here: class A(B, C). Then what is the difference between receiving the dependency from a config file, or 'injected' at the instance instantiation level? I can test and refactor at any time. Convenience, Man. Hope you undersand.

[@TPHRyan]

I'm confused, what is a use case for the Root (stopper) class, where you would want to, for example, draw() on EVERY OBJECT in the MRO instead of just stopping at the first? To clarify, I mean a use case where there isn't already a definitive "Adam" that would actually do something useful with its method.

[@user-uq8jr3rv3g]

I've never seen such a humorous but useful talks like this !

[@evolagenda]

The guy with the self / super question had a point. If you inherit from a parent class it's methods are available on self. So super().method() is the same as self.method() at that point. Unless you're calling super().__init__() or something. I guess the point was if you use self it'll spend time looking at the current class for them method first and then go to the parent which I guess if you reimplement it in some way can be a problem, whereas if you use super it will just go straight to the parent.

[@thanhtan030t]

I learn how to use fake over mocking via Robot's example. Amazing!

[@BrendanMetcalfe]

The master!

[@HypocriticalElitist]

Raymond's examples are slightly contrived to illustrate uses of super(). He has a class like CleaningRobot call super().move_forward(), and then uses a subclass to join this logic with an alternative implementation of move_forward(). In real code, it'd make more sense to call self.move_forward() (implicitly inheriting Robot's implementation), and have a subclass override move_forward(), using only a linear inheritance hierarchy. super() should only be used in cases where the class containing the super() call *knows* that it wants to punt to the next class in the MRO. Generally, this means you should only use super() within a method foo() to call another class's implementation of foo(); if you want to call bar() instead, then make it self.bar().

[@Fulmenify]

I know it's a year late, but as far as I understood, the advantage of the use of super in the robot example is if you have multiple different robot implementations. For testing, you can replace the movement code of each robot as shown, without having to override move_forward() etc. in each of the subclasses.

[@magno5157]

@@Fulmenify super() by itself doesn't give you the advantage of dependency injection. What Raymond has done here is to use super() *and* exploit the diamond inheritance pattern for dependency injection without having to override any functions.

[@vela7447]

41:50 covers this?

[@isodoubIet]

super() should never be used, except in cases where there's only single inheritance and it doesn't make any difference. It's fundamentally broken.

[@paulallen1597]

This makes sense. I actually learned from this talk.

[@silence_coder]

Nice Talk

[@uppubhai]

The heck happened in 19:00 .can somebody explain?

[@rembautimes8808]

Excellent communicator

[@isodoubIet]

Too bad what he's communicating is completely wrong.

[@Neceros]

19:40 I had to pause it and really think about what he was saying. My brain just doesn't comprehend this stuff well anymore, but still he makes a lot of sense.

[@martialblog]

+Neceros It gets clearer at 30:30 - Children get called before their parents, and parents get called in the order listed. So MeFirst(MeSecond, MeThird).

[@skepticmoderate5790]

@@martialblog Isn't this just a preorder traversal, but with the child as the root instead of the parents? Why does he imply that this is some kind of revolutionary idea?

[@Han-ve8uh]

@@skepticmoderate5790 www.srikanthtechnologies.com/blog/python/mro.aspx Look at case 4 in this article. Pre-order traversal will do D B A C, not the same as MRO. Also case 5 is impossible for MRO but pre-order will be C A B

[@isodoubIet]

It's not your fault -- it's a truly insane anti-feature.

[@ChristophZwerschke]

Beneficial and hilarious as usual - Raymond considered super. The first part was a bit difficult to follow though, without a visual diagram of the family tree. Apart from that, I liked the presentation style using only Emacs.

[@skepticmoderate5790]

It's just a preorder traversal, but with the child as the root.

[@isodoubIet]

Raymond considered colossally incompetent I believe you mean

[@MrEternalFool]

Why would you put Raymond's talk and David Beazley talk at the same time?

[@MattJoinerPwns]

Which would you choose?

[@jl_woodworks]

I'd attend Raymond's just because he's a Core Developer. David is great though.

[@isodoubIet]

I don't know who the other guy is but his talk can't possibly have been any worse than this one

[@CrystalGamma]

I think the Python 2 syntax of super() makes it more obvious what is actually happening. That said I only suspected something like this happening under the hood before watching this talk.

[@barbarairena6714]

in the first (pizza-)example the the MRO gets changed only if a) OrganicPizza- class is created (subclass of Pizza and DoughFactory) and called and if b) the OrganicDoughFactory is created as a subclass of DoughFactory-class! But why b)?? I dont understand the necessity of b)

[@Han-ve8uh]

This talk did not have time to go under the hood to explain how MRO is working, you can draw out the inheritance diagrams of both the pizza and robot example in this talk and see that they match Case 4 in this article: www.srikanthtechnologies.com/blog/python/mro.aspx. In the talk, he was basically adding 2 more classes to create this diamond, and "hacking" the MRO order so new custom methods are called.

[@mishasawangwan6652]

reminds me of express middleware. in fact, i dare say it’s conceptually the same.

[@dimitristheodorou4959]

Um... I just ran this exact code with Python 2 and it does *not* work as advertised. I still get the insecticide treated dough.

[@dimitristheodorou4959]

Dimitris Theodorou Nevermind! The OrganicDoughFactory must inherit from DoughFactory for this to work, which I had omitted.

[@RadCirskis]

Great talk tanks! I am a bit confused with Python documentation on super. It says: "Return a proxy object that delegates method calls to a parent or sibling class of type.". I suspect it means the sibling of a parent, not the sibling of the type. Am I correct?

[@jebob1]

No, it is possible that the method called could be a sibling of the type. The cleaning robot's super points at sibling MockRobot when the robot is a MockCleaningRobot.

[@magno5157]

The term "sibling" shouldn't even be used at all. "Siblings" are obvious in the diamond pattern. However, if you have a very complicated multiple-inheritance tree, it can be very difficult to see which class is a "sibling" of which other classes. The only non-ambiguous way to determine super() is to apply the MRO algorithm as explained in www.python.org/download/releases/2.3/mro/, which turns out to be really easy (although you need to be really careful for complicated trees to avoid careless mistakes).

[@isodoubIet]

The only real way to make sense of this is to never use multiple inheritance in python. It's fundamentally borked and impossible to use correctly. Stick to single inheritance if you must use inheritance at all.

[@DheerajSharma-ol3go]

Thankyou sentdex

[@hangugeohaksaeng]

Thanks for the talk Raymond. One note: No one else in that room, or online, is familiar with your family tree. Why not choose characters from a popular movie, or celebrities? Anything else would be better than the relationships almost guaranteed to be known only to you.

[@morenoh149]

to be fair it's pretty close to being the minimal example needed to teach MRO right? It could've been classes A,B,C,D,E,F,G,H,I

[@laxmanapolisetti]

Even a popular movie characters would be unfamiliar to me. I just replaced with my family tree and it was hell lot clear to me after that. I also drew Raymond family tree to compare. One of the best lecture though, beautifully explained.

[@isodoubIet]

The obfuscation is intentional. If he used a clear example, it might become easy for the audience to realize just how broken multiple inheritance is in python. Can't have that, so we must contrive an example that makes absolutely no sense.

[@isodoubIet]

@@laxmanapolisetti If it was really "beautifully explained", it should've started and ended with the explanation that multiple inheritance is fundamentally broken in python, impossible to use correctly, and that the author of super considered harmful was correct about almost everything (the only error he made was in not making the recommendation to drop MI altogether). He chose to mislead instead.

[@FunkyELF]

Don't know why this was promoted to me just now but I don't like that this video says it's 9 years old and from PyCon 2015. It adds up, but seems like it shouldn't.

[@geshtu1760]

Seems I'm the first one to point out that if you remove the insecticide, you need some other way of dealing with the insects (and that other way may have a different set of pros and cons too) ;) Awesome talk though, but I actually think this way of rewriting the "supply chain" is horrible because it is too "magic" and easy to cause surprises

[@geshtu1760]

In fact this is probably a good example of why many coders are starting to turn away from OOP

[@lrebollo]

27:56 LOL

[@baksagimm5890]

wow now i get it. super is super complicated

[@anibaldk]

I think this guy is considered Super

[@BryanChance]

Oops. Let me go read the article now..lol I used to hate python until I saw some of his videos .

[@aaronbell5994]

:)

[@nuynobi]

FauxBot would have been a better name than MockBot.

[@BeYakko1]

"super does not call your parents it calls the ancestors of your children". um, the ancestor of my children would be me, um, the parent. that statement is so confusing. I have watched this over 100 times and I still don't get it.

[@twistedsim]

That means that it may goes further than just "it's parent"

[@Lexaire]

super() does not look at the parents of the class it's in. It uses the MRO to find the next in line for whoever created the object. So in a child (subclass) object, if one of the parents calls super(), it will not check the parent's parents but instead look at the object's MRO and see who is next in line. In the case of the family tree, when we have a Matthew object and Dennis calls super(), it goes through the Matthew object's MRO and finds the next in line: Sharon.

[@isodoubIet]

I cannot imagine how much brain matter must be lost before one calls super() "well-designed".

[@isodoubIet]

Already in the first 5 minutes we have a glaring error. Multiple inheritance in python _never_ works, because there's no such thing as "classes without overlapping methods", because __init__ always overlaps, which means it's impossible to call __init__ correctly, which means it's flatly impossible to use multiple inheritance correctly in python. The author of super considered harmful understood this. You don't. Drop the smug and try to learn something.

[@isodoubIet]

Second glaring error, C3 linearization is not a "best and optimum solution to this problem", because it's not a solution to this problem in any sense. Turning multiple inheritance into single inheritance means there's no multiple inheritance at all, so you didn't solve the problem, so the thing doesn't. fricking. work. Funnily enough, multiple inheritance in C++ works just fine. You can use it. It works. You don't have to make all the classes aware of one another and demolish encapsulation to make it work. The author of super considered harmful understood this. You don't. Drop the smug and try to learn something.

[@isodoubIet]

More glaring errors I didn't bother to mention, but Pizza inheriting from DoughFactory? Come on.

[@isodoubIet]

No, it's not awesome that you can change the parent from the child, actually. It's insane, brittle, and impossible to use correctly. "is this fragile? no, it's deterministic" so is everything in programming. It's _extremely_ brittle. That's all the incompetence I can stomach, I'm afraid.

[@stoneshou]

Every Raymond talk has one extremely funny joke that I did not know before. In this one: E(xclusivelyusedby)M(iddle)A(ged)C(omputer)S(cientists)..

[@dunder9444]

Sentdex brought me here

[@ruthlessadmin]

I'm still so fucking confused :'( I may be too drunk for this....

[@yukiousuzumi2595]

What? Can't believe one takes 45 minutes to explain a `super()` call. Let me add this to my favorite list for further watching though.

[@beep_doop]

a pizza-place is not a kind of dough-factory so it shouldnt inherit from it

[@spencer3752]

The purpose of subclassing is for code reuse. It doesn't matter if the subclass is not the same "kind" as the parent class. And in a way, a pizza place is certainly a specialized form of a dough factory. They produce an item, which is made from dough. Need to take a look at his "Python Class Development Toolkit" video and "Art of Subclassing" video.

[@internetzpotato383]

In this case the factory is thought of as a mixin type, not a base class you are inheriting from. It'd probably be clearer if it was called `DoughFactoryMixin`.

[@nicholasdewaal]

It's extremely ironic that someone doing so much awesome work in advancing tech spends half this talk beating the Luddite drum of not modifying the software/code in plants (GMO's). Honestly, adding vitamins to rice as a GMO (golden rice) could save millions of children from death and malnutrition. I love Raymond, but stop the Luddism, and stop risking the lives of children please. Your talk about super is harmful.

[@narnbrez]

i think he just uses it as a hypothetical. if you were designing a system for a customer and the customer wanted a particular type of widget, the system should give that particular type of widget. can you link to where he actually supports anti-gmo positions? He spends more time talking about insecticide than GMOs. (granted GMOs and insecticide are related since some strains are insecticide resistant and thus can be treated with higher strength insecticide without damaging the crop, although that has negative side effects for the ecosystem and neighboring farms not using the same strain of gmo. he does not make that connection himself anywhere though.) Given how candidly he remarked on how bell peppers are obviously healthier for you than pepperoni, I think if he actually was anti-GMO he would have stated it. It was merely a (largely) non-offensive way of constructing a hypothetical real-world example to demonstrate a concept. i consider myself a fierce advocate for gmo foods but i dont see your point.

[@internetzpotato383]

Business guy: You have to change the production code, we stand to lose millions! Developer: Nobody understands this legacy code after you fired the old team. It's been there for years and it's too dangerous to change without understanding more. There is much more that we do not understand than understand at this point. Business guy: If we lose millions, we will have to start laying off people. They all have families, for God's sake! Think of the CHILDREN!!!

[@SuperAhmed1337]

I only wished he didn't use references to a topic he has no clue about. "Organic". "Pesticides". Blah, blah.

[@litbmeinnick]

How can you know he's no clue about it?

[@internetzpotato383]

@@litbmeinnick Duh, because he doesn't have the same opinion as the complainer, a clear sign he is not well-informed.

[@isodoubIet]

That would mean no talk because he has no clue about anything.

[@SuperAhmed1337]

@@internetzpotato383 i've studied this, but do go on...