Python __slots__ and object layout explained

[@AWriterWandering]

“I’d like to thank me for sponsoring myself” 🤣🤣🤣🤣🤣

[@acif58]

Internet, its like sitting on a commode writing on toilet paper, and then the papers talk to each other.

[@nclt1978]

I would like to thank you a lot too.

[@Yotanido]

It's not just the memory savings. Restricting the ability to add new members is a feature itself. If I make a typo when assigning to an attribute, I want to get an error. I don't want it to be silently ignored, leaving me wondering why the value doesn't change. That is really the main reason I always use slots. Memory savings are just a nice side-effect.

[@mCoding]

You're right, it is also a feature when you look at it that way!

[@KappakIaus]

You could maybe also catch these errors with the right IDE settings (i.e. let it warn you about assignment to attributes not declared in __init__)

[@expurple]

I prefer mypy warnings for this purpose. Adding/editing __slots__ every time seems like extra typing

[@ninjaaron]

@@expurple Adding type annotations to everything is also extra typing.

If you use annotations you can add the following below the annotations of your instance variables instead of repeating their names: ___slots___ = tuple(__annotations__)

[@aaronm6675]

Finally, a great primer on slots!

[@mCoding]

Glad you enjoyed!

[@alcyonae]

The way you pick examples is phenomenally efficient at highlighting limitations and advantages of whatever functionality you present.

[@mCoding]

Many thanks for the kind words!

[@strandingstranger]

this channel is literally a gem im soooooo glad i found you

[@Roarshark12]

Thank you, your explanation was awesome! I've been doing Python for ages but never really bothered with __slots__. At least now, I know what they are, their trade-offs and what they're used for.

[@mCoding]

Great to hear!

[@mike50378]

no useless words in the content, all are knowledge intensive explanation. perfect!

[@goeiecool9999]

5:51 "We need to enter the matrix" Wouldn't this be more like leaving the matrix because you're peeking behind the curtain?

[@mCoding]

You're absolutely right!

[@rickharold7884]

Love it! As a C / C++ dev (in past) I can appreciate this. However as u mentioned we are rarely memory constrained. Thx !

[@AJMansfield1]

The reason to include __dict__ in __slots__ is because it makes it much easier to extend/modify the code, while _still_ giving you memory savings over the default. Anything stored in the __dict__ incurs additional storage overhead not just because of the dictionary headers, but also for (a) storing the _key_, and (b) due to the additional empty nodes needed to make a hash table efficient --averaging about 3 pointers worth of storage per entry, while a slot variable always only takes one. Therefore, any variable you can move into a slot represents a decrease in memory footprint, even if you can't move all of your variables into slots and still need __dict__. Plus, current versions of python don't actually create the underlying dictionary object that goes in __dict__ until a non-slot variable on the object is written to, so depending on the usage pattern of your code, it could actually be _more_ memory efficient to leave a set of variables that only a small fraction of the objects of some class will use out of __slots__ and just rely on the automatic creation of a __dict__ in those cases.

[@phuctran25277]

Nice. Your content is unique and not found anywhere else. Keep up the good work!

[@mCoding]

Thanks so much!

[@vinson2233]

I'm working as a data scientist for 3 years using python with no-CS background and the topic you cover is always interesting and mindblowing.

[@HypnosisBear]

Incredible as always...!!!

[@mCoding]

Thanks!

[@NStripleseven]

“I’d like to thank me, for sponsoring myself.”

[@davea136]

I would also like to thank you for sponsoring yourself.

[@MaxTechniche]

There's so much information, my brain is frying. I'll be looking back at your videos constantly!🤯

[@laurinneff4304]

Thanks, now i actually understand the previous video completely

[@daviddavidson2288]

I saw a comment on one of your videos saying "thanks Tony Hawk". Then I looked up Tony Hawk, and the comment suddenly made sense. Anyways, thanks for the videos.

[@mCoding]

Hahaha I'll take it as a compliment, cool dude. Thanks for watching!

[@ipbody]

Its crazy how all your videos are intresting and easy to follow!

[@AsgerJon]

That's the best sponsor spot ever!

[@n3141-d3s]

I find not being able to add a member actually a very interesting feature. This is something I often don't want to do, but can happen by accident. Of course, mypy can catch that often.

[@טלאיצקוביץ-ב4מ]

great video! I would like to see more c stuff , maybe even showing stuff on how python code translates to c code

[@b_kind]

1. Is there a non-hacky way add type annotations to slotted attributes? 2. On 2:36 you say you can still add class variables to the class as usual. Can we go one level deeper and restrict that too ...with slots? ...without slots?

[@mCoding]

1. You should add type annotations wherever you would normally initialize the variable (typically in init). 2. That would involve setting the slots on the metaclass, which gives an error if you make slots nonempty, and even if you make it empty it for some undocumented reason still doesn't prevent the class object from getting a dict, so it seems like you cannot prevent a class object from getting a dict.

[@b_kind]

@@mCoding I see, thanks for clarifying it! :)

[@quintencabo]

I had been waiting on this! Really great you explain things well you are an explainer

[@mCoding]

Much appreciated!

[@sbk1398]

Best sponsorship in a vid I've seen

[@kisawesome2981]

Love your videos keep it up!

[@mCoding]

Thanks, will do!

[@chrisgellrich6866]

I use __slots__ for a different reason. In the event that I write a new method and spell an attribute wrong, I find the bug immediately.

[@rodneylives]

"I'd like to thank *me*, for sponsoring *myself*! Did you know that I am available for consulting, contracting, training and interview prep services?" AWESOME

[@_maxt]

Really interesting to me and useful. A debug through the C code would be very cool also.

[@ashutosh_verma_fullstack]

8:46 => could you explain Instance memory map for inherited class B(A), where A is slotted class but B is not.

[@eladlevy9494]

Love your tutorials. Great work! Subscribed

[@QuantumHistorian]

Are classes with slots faster? One fewer dictionary lookup must count for something, but is it noticeable in reasonable use cases?

[@mishikookropiridze]

No, it doesn’t give noticeable speed difference, only memory

[@leobozkir5425]

yes, one lookup is not that much faster but the more you have the more you save. And why not if you can.

[@mishikookropiridze]

@@leobozkir5425 I would say to never worry about optimisation at first, but if software demands then you can.

[@1Hippo]

Does the benchmark at 4:10 not answer that question? It seems to be 10-30% faster, but this probably not really relevant if your have performance issues with python, there are other tricks that make a much bigger difference.

[@patricksmith1520]

I like your sponsorship very much :)

[@deemon710]

Wow that was very komakai. So the primary use case for using slots is when you have a lot of properties or when you're creating a lot of the objects?

[@ChristianBrugger]

This is such a nice video! Thanks. Would love one on weakrefs or asyncio.

[@Mustak310]

Really good video, as always. One request: can you do a video where you talk about making the Python classes in C? I use the CPython a lot to make modules (with only functions) in order to make faster codes. I was trying to make the Python classes directly from the C API but I have a lot of trouble. Thanks!

[@mCoding]

I will probably make a Cython video at some point, not sure about a pure C api video though. Thanks for the idea!

[@tahini245]

Really great explanation (as always!) Maybe a video on weakref in the future?

[@mCoding]

Thanks! Probably will at some point but it's not a priority at this time. So many other interesting things to cover!

[@arisweedler4703]

I'm going to try to summarize what I learned in 1 paragraph, as I find it a helpful exercise, and I want to feed the algorithm :) When python classes store data, is uses a hashmap (__dict__). If you want to use an array of pointers instead, you accomplish this by specifying "__slots__". This has clear benefits if you care about the differences between these two data structures.

[@mCoding]

That's pretty much it!

[@douglasmason6067]

what’s the recursive “getsize” function you use? great work as always!

[@nelsenedbertwinata7634]

I feel like I can use this as using C Struct, where we already know its member ahead of time

[@giraculum9981]

One tradeoff of slots that's not immediately apparent is that it can break multiple inheritance: A subclass can only have one parent class that defines nonempty slots. Presumably it has something to do with how parent and child slots are concatenated? Like in C++, for single inheritance you just tack on extra fields, and bam, the address of the child type is also valid as an address to the parent. But with multiple parents you need special casting operators. But since Python is using descriptors for access, I still don't quite get why that's an issue.

[@connorclub6244]

what about '__dict__' AND '__weakref__' in __slots__? will the class be slotted WITH dictionaries and weakrefs?

[@benyaminyakobi3652]

Very clear explanation! Thank you very much :)

[@idanhacmon5682]

How does changing a slots variable from an object instance work? I would assume that the value of that variable would change across all instances of that class. but it doesn't. can you explain why?

[@ДмитроПрищепа-д3я]

it's just an instance member, changing it doesn't affect the class or any other instances of that class.

[@Graham_Wideman]

James -- what point were you making in the title, listing dunder-slots and slots separately? (BTW, nice choice of topic, and delivered satisfyingly succinctly.)

[@mCoding]

The dunder slots are just the _extra_ slots a class defines.

[@Graham_Wideman]

@@mCoding Thanks for the reply, but ...well on the one hand, yes obviously slots are data that's 'extra" to whatever per-instance data Python needs on each object to manage it. But such housekeeping implementation data doesn't quite seem appropriate to refer to as "slots" in the context of user-useful data slots. Is your notion captured in your "graphic" at 7:27, where a label "each is a slot" refers to ob_refcnt, ob_type, x, y, z? In what sense are ob_refcnt and ob_type "slots"? Are you using "slots" here as a generic synonym for "sequentially-stored same-sized data members" (here, on a B object instance)? Or is there some realm (maybe in the C implementation) in which there's an actual array of these same-sized items named literally "slots"?

[@yuri0001]

This for micro-controllers is very interesting

[@Burbinox]

What about the case when we will have a list object as a slot? Does that make sens?

[@MrDowntemp0]

Can you create validator descriptors for classes with slots? It seems like setattr() and getattr() don't work the same.

[@XCanG]

I have a question: in my case I use fastapi with Pydantic classes for models, does using "__slots__" make any sense here? Or it not for my case?

[@lawrencedoliveiro9104]

I haven’t used slots to save memory, but I have used to them to avoid a certain class of errors. I have some classes with writable properties. In particular, I had a property named “dash”. One day, I was writing code like ctx.dashes = «dash-settings» and I was wondering why the dash settings were not being correctly set. It took a while to realize I had written “dashes” instead of “dash”, and so instead of updating the correct property, Python had silently created an additional, useless, attribute! From that point, I have been careful to add slots definitions to such classes. However, if you then want to have weak references to instances of such classes, you must remember to add the name weakref to the slots list. This is where Python stores some kind of backpointer for keeping track of weak refs.

[@Neuroszima]

what i miss from this video, is that, is the slots reserved for the variables you declare in "__init__" only or they go for the function definitions inside class themselves as well?

[@NicolasChanCSY]

I remember I watched a talk from Richard Hettinger mentioning the use of __slots__ but he could not go into the details probably due to time constraints. Although I am not likely to use __slots__, thank you for making this video! I learn something new today! One question though: when we define a __dict__ in __slots__, does attribute assignments work the same as if __slots__ is not used? Is this __dict__ in the __slots__ of the superclass still a dictionary? And does Python uses this __dict__ when we assign new attributes dynamically to instances, if possible? Thank you!

[@mishikookropiridze]

Yes, it will store variable not in slot into __dict__, also your question can be answered just by opening python repl yourself, play with it.

[@Mutual_Information]

Excellent! Maybe one day I’ll use slots now 😅

[@amidfallen]

Just curious, is there any reason why you are using .__dict__ instead of vars()?

[@haibrenner]

It seems like this slots thing should have been the default behavior of a class.... The ability to add attributes dynamically on-the-go is not the way most uses of classes are, even in python. I wonder why that's not the default, rather than the other way around, especially when it also saves on memory and although that's the standard case it is much less known.

[@blackswan2020]

How do you calculate the recursive size of an object? Would like to see the `getsize()` function :)

[@mCoding]

See the source code in the github!

[@Khushpich]

Best python channel

[@tunafllsh]

So the size of a slot instance without recursion may be much larger then that of a normal instance without recursion.

[@Checkedbox]

Hey, do you know where I might be able to find a similar colour scheme for VS Code?

[@wsrgs4]

looks like monokai

[@robmarks6800]

Are there any nice tools that can visualize the memory layout of python objects just like your diagrams?

[@mariamozgunova9584]

Nice video! Now I finally understand slots in Python How can I get the sizes of ssize_t and ptr in Python? It would be nice to know where you search for information to prepare material for videos

[@robinpipslayertekprofitsfa2644]

@0:34 #LOVEIT!! 🤣🤣🤣🤣🤣🤣👏👏👏👏

[@nixonkutz3018]

But those __slots__ pointers have to point to something - even if it's memory managed down at the C level (heap, presumably)?

[@mCoding]

Yes, nearly all Python objects are heap-allocated, so even with slots there is still one level of indirection. However, this is still better than the two levels of indirection used when a dict is used. Good catch!

[@ZeroSleap]

So this makes classes more work like they do in C++,C# etc.Where you got predefined fields(instance variables) and you declare them static if you want them to be Class only.

[@filiplangr7727]

Good video, but it doesn't really explain why slotted 7-attribute object is 5 times less memory demanding than the non-slotted one. In both cases we still need to store the data somewhere (either slot or a __dict__). Why storing data in __dict__ is so much more expensive? Where's all the additional memory consumption coming from? Genuine thank to anyone who takes time to explain me.

[@GHFishy]

Using a dict is just that much more expensive than array like storage. As for why that is, consider the extra memory a hash table (dict) needs to hold in order to reduce the chances of a hash collision.

[@filiplangr7727]

Good point, thanks. Apparently dicts saves hash codes and pointers to key+value tuple.

[@MrCmon113]

I've spent days trying to understand python code, which at runtime assigned attributes to objects, whose classes were also determined at runtime. Anything can be anything.

[@mCoding]

This is both the gift and the curse of the language. Such dynamic abilities allow for so many possibilities, but many of those possibilities are terrible ;)

[@johnniefujita]

at 1:54 just to add some info to the viewers. This is monkey patching. Although usually we use it to change or add some method instead of just an attribute. Very pythonic!

[@RayHorn5128088056]

Kudos on being full of yourself. What do you ask per hour of your time?

[@houstonbova3136]

This would be useful in the case of Cloud Functions where you have to pay for memory used by your program.

[@monsieuralexandergulbu3678]

0:30 oh yeah good ol' me ;)

[@theepicguy6575]

This is GOLD

[@mCoding]

I appreciate the kind words!

[@georgedicu7397]

U consult also on Dart?

[@mCoding]

Unfortunately no, I don't know Dart and therefore do not consult on Dart.

[@blackswan2020]

Can you do a video on weakrefs? 🙏

[@erikgiesenloo1871]

1. I use MATLAB for work - and MATLAB doesn't allow assigning variables to classes on runtime by default. Does that mean it uses slots? (Is that the same for C++?) 2. So the class with slots stores the pointers to the data as an array? Where is the actual data stored?

[@mCoding]

Most languages, especially statically-typed languages, use what Python calls slots by default, and hence there is no specific word for it in those languages. It just means pre-allocating some memory on an object for a specific variable, which is the simplest and arguably most efficient way to store attributes. What Python does by default would be laughable in most other languages as it is equivalent to proxying all attribute lookups and assignments through a dictionary member, which is wildly inefficient. But, of course, nobody uses Python for its runtime performance.

[@michadarowny3811]

Will watch later Cause didnt have Time now

[@arnoldwolfstein]

0:30 just a moment i was expecting Snoop Dogg speech: kzbin.info/www/bejne/rXi1d2Z9hpmKms0

[@prashantrana1089]

this guy rocks

[@1234minecraft5678]

Well, always have been a philosophical way of how to measure ones dict size correctly...

[@not_pockchan3418]

19th :D, yeah very specific ik

[@mCoding]

Confirmed!

[@not_pockchan3418]

lmao 1 year ago, still come back to watch your content as it is very helpful Keep going!

[@tshegomonama7910]

"I'd like to thank myself..."🤣🤣🤣🤣

[@_maxt]

Don't forget you hit the like button odd number of times in total, including all of your other visits to this video under the same KZbin account.😇

[@JorgeLuis-ts6qp]

I think this topic needs a bit of update regarding Python 3.11 objects with lazy dicts.

[@darvil82]

discord gang 😎

[@robertbrummayer4908]

Great

[@zaphbeeblebrox5333]

Yessir!

[@MithicSpirit]

Discord gang

[@HonsHon]

Hit that like button 5 times

[@redpred3502]

Too many commas in that title

[@mCoding]

Lol ty fixed

[@guilhermealveslopes]

Awesome video and great examples, but your face is too distracting 👀 Ye too handsome

[@markcuello5]

Help me

[@karamboubou8579]

3rd

[@26-dimesional_Cube]

A hacker give this code to you def password(str_input, str_key): -> int Password_num = 0 str_key_len=len(str_key) str_input_len=len(str_input) for i in range(str_input_len): Password_num += str_key.index(str_input[i])*str_key_len**(str_input_len-i-1) return Password_num This function can take a string and output the password needed to protect the string Puzzle: Make a inverted function where argument are password and str_key Input: Line 1: password Line 2: str_key Output: A string Constrains: str_key cannot have duplicate letter and must have at least character within the str_input

[@markcuello5]

HELP

[@alpers.2123]

I love python but OOP in python is ugly

[@mishikookropiridze]

Why is it ugly?

[@alpers.2123]

@@mishikookropiridze Verbose syntax: class Foo: def __init__(self, a, b): self.a = a self.b = b ...

[@cicik57]

this is amazing feature with horrible notation. Lets think about a decorator for that...

[@cutcrew2743]

Did not work, so I googled it. Works if changed to: Class A(object):

[@mCoding]

Hi, it looks like you are using Python 2, which reached its end of life years ago. Please upgrade to Python 3 in order to follow any of the videos in this channel.

[@cutcrew2743]

@@mCoding version 3.0 broke python whose development has been frozen. Please refer to the language you are using as Cobra 1.x.

[@virtualraider]

If I could make a suggestion, your very well made videos might be even easier to follow if you used more descriptive names. Instead of calling a class "A" and and it's instance "a", calling them ClassA and inst_a could make it easier to follow as you speak. Just an idea.

[@QuantumHistorian]

That's not the Pep8 standard though. Camel space for classes and underscore lower case for instances is the norm. And Hungarian notation as a naming convention is rather a pain

[@virtualraider]

@@QuantumHistorian I'm confused. "ClassA" is camel case, and "inst_a" is lower case with underscore What am I'm missing? 🤔

[@QuantumHistorian]

@@virtualraider A and a also has that naming convention. What you've added is a prefix describing the type of the object, which is known as hungarian notation. That's rarely done these days, especially in languages that dynamically typed

[@virtualraider]

@@QuantumHistorian oh, I see. Just to clarify, I'm not proposing that real code should be named like that, only for these videos. Usually the explanation runs fast and he always has to clarify that "little a" is the instance and "big A" the class and so on, so this would make it explicit.

[@kutilkol]

That was like super awesome!

[@mCoding]

Glad you liked it!

[@francisco_sbf]

actually, it can be an str itself (when you have only one attribute). check this out on documentation docs.python.org/3/reference/datamodel.html#object.__slots__ Transcripted from the reference: "This class variable can be assigned a string, iterable, or sequence of strings with variable names used by instances"

[@markcuello5]

HELP