Ned Batchelder - Facts and Myths about Python names and values - PyCon 2015

"Speaker: Ned Batchelder
The behavior of names and values in Python can be confusing. Like many parts of Python, it has an underlying simplicity that can be hard to discern, especially if you are used to other programming languages. Here I'll explain how it all works, and present some facts and myths along the way. Call-by-reference? Call-by-value? The answer will be clear!
Slides can be found at: speakerdeck.com/pycon2015 and github.com/PyCon/2015-slides"

[@sandysandeep7227]

Came here from Automate the Boring Stuff with Python Video 16. :)

[@huydong8774]

Same here. This is a really good material.

[@AmeerulIslam]

typed the link?

[@alaminbijoy1027]

from IRC. someone refer me to this link when I asked a question question about name-value assignment.

[@allmarketingtv]

Same!

[@abdallahdataguy]

same to me

[@mayanksj]

My favourite quotes: 1. Assignment never copies data. 2. Python is neither "Call By value" nor "Call By Reference", it's "Call by Assignment"! Epic! 3. There is no way in python where a name can refer to another name. A name can only refer to values. Oh my!

[@domagojbosnjak75]

Absolutely agree! Even though call by assignment comes down to the call by reference in C/C++

[@bradywb98]

C has ONLY pass by value. There is no pass by reference in C.

[@skylerquinton1367]

instaBlaster.

[@miraculixxs]

@@bradywb98 the fact that C is passing pointers by value is an implementation detail. It is still call by reference semantically.

[@bradywb98]

@@miraculixxs The behavior that passed arguments exhibit is most certainly not an implementation detail.

[@serpentphoenix]

Oh man, he's probably the best Python teacher I've seen on KZbin.

[@hrmIwonder]

I feel like that very last statement about a list as a default value will stick around forever & and potentially grow forever, is a rather important point.

[@markandrews5508]

This was brilliant. It dealt with some of the subtleties at the heart of simple python statements and how these statements work differently to similar statements in other languages. Most people, myself included, don't quite get all of these subtleties until after many years of using python.

[@hangugeohaksaeng]

Great talk as always Mr. Batchelder. You are always succinct and articulate.

[@aaron41]

I've been programming in python for almost 10 years, and I still learned something new today!!!

[@TimothyApe]

I am thankful for the pause button. I am able to follow, I just have to let my brain catch up with the speed of him talking. Great explanation none the less or for this very reason! :)

[@joecode5153]

many...many... python tutorials later, something educational at last! :D It's one thing to say: "do it this way. now, do it your way" when in fact, "it" always does whatever you say, in a very particular way. Answered so many questions no tutorial ever teaches. Thanks Ned!

[@monikaparmar2061]

That has unfortunately become the reality now. There are countless number of mediocre lectures/tutorials out there claiming to teach python. However very few do teach these internals of the language that help in understanding python in depth.

[@PRS-0317]

AHA! For the old asm/C programmers out there, the money shot is 20:15. Names/variables have scope, whereas all (allocated) values are on the heap and only 'leave scope' when the last reference to them leaves scope.

[@UtahHeroes]

There is one more case that should be detailed here: my_lambdas = list() for x in range(10): my_lambdas.append(lambda y : y + x) my_lambdas[0](5) will return 14 and not 5 To fix, you have to write it like this: my_lambdas = list() for x in range(10): my_lambdas.append(lambda y, x=x : y + x) my_lambdas[0](5) returns 5 It's an esoteric case, but it's useful in frameworks such as Twisted.

[@bharatgulati1903]

this is an event listener problem, quite famous in js world. another approach is to use iife (aka immediately invoked function expression) in case one does not want to allow the x field to be tampered with in the lambda added to my_lambdas my_lambdas = list() for x in range(10): my_lambdas.append((lambda x: lambda y: y + x)(x)) my_lambdas[0](5) # returns 5

[@seandockery368]

Thank you for highlighting this, @UtahHeroes. It is actually a very salient point that names inside lambda suites aren't dereferenced until the lambda is executed... not when the lambda is assigned. I think that it is clearer to write it like this because it explicitly eliminates the confusion about an inner "x" shadowing the outer "x". my_lambdas = list() for x in range(10): my_lambdas.append(lambda y, z=x : y + z)

[@dinethdewmina3530]

epic explanation completely shock my understanding of variables and lists

[@domagojbosnjak75]

23:18 question can have actually multiple answers. Depending on the types of the given elements list and the type of the modification the guest was talking about. So we could have this scenario: a_list = [1, [2, ], 4] x = a_list[1] a_list[1].append(3) print(x) # prints out modified version of the old value! List are mutable and the change happened in place! In contrast to this scenario: a_list = [1, 'ned', 4] x = a_list[1] a_list[1] += ' is awesome!' print(x) # prints out the old value. (The address x is pointing to actually never changed in both scenarios) print(a_list[1]) # prints 'ned is awesome' and of course now is: id(x) != id(a_list[1]) cause strings are immutable. P.S. Great stuff, thanks for the super instructive facts!

[@das6109]

Oddly enough the thing I wanted to understand most was addressed as the last question in the talk. Thanks to whoever asked it haha

[@work_account_]

what an awesome talk!

[@Whatthetrash]

Excellent talk! Thank you! :)

[@kusharora1435]

wow.. fantastic stuff. am so glad i saw this

[@luckyboy20021]

Great video, it help me a lot

[@Naz-yi9bs]

Amazing, thank you!

[@edchelstephens]

Amazing talk!

[@MagnusAnand]

Amazing talk

[@scottcampbell2707]

I found this video very useful.

[@timzhang6253]

great talk!

[@fidelpalma5164]

Came here thanks to Al Sweigart and Automate Boring Stuff with Python. Thanks Al, thanks Ned.

[@movax20h]

I always recommend Python as a first programing language to others, but sometimes especially to people really far off CS background or with some mindset, really do have trouble with name passing, especially to functions. Just like this a_list = a_list + [val, var]. It is simply hard to understand and lengthy to explain to a person with zero previous coding or programming background. The cases like x += y, and special iadd for arrays, or apparent mutability of integers (x = 7, x += 3), makes it simply confusing. The big problem might be our attachment to saying "variables" for these thingies (x, a_list, etc). Where in fact they are not variables. We also often say "variable of type x", "this variable has this type". Where this is technically not true. Are "variables" are of the same type, they point (refer) to values. These values can have different types. The easy way of seeing that is that x = y, does exactly the same thing no matter what is the value (and type) of what y refers to, and no ability to overload equal sign operator (it is not an operator). I am starting to think that functional programming might be a better first programming language for many, because it basically forces you to do return a_list at the end by design (and do not allow you to do a_list = a_list ... anyway). In functional languages (Erlang, Haskell, Ocaml, Elixir, etc) you do not need to worry if you pass data to function by value or by reference or by name, and whatever it is copied or not, because in all these cases the result is the same. We usually call it "by value and copy", but really in implementation it is passed by reference (pointer). There is no difference because you can't mutate data anyway.

[@PRS-0317]

Not just zero previous coding experience - Python treats basic concepts like stack/heap variables differently in its syntax than every other programming language, and reuses established terms of art to mean different things that only apply to python. At least it's internally consistent? No wait - it's not. Some types are mutable and some aren't, but using the same operator (=) on them will yield fundamentally different behaviors (ptr/name reassignment vs memory allocation and then reassignment). Well, as a native speaker of English, I really can't complain about arbitrary, context dependent language rules. "I before E except after C, unless its a weird neighbor"

[@joseville]

Great talk. I learned many of these the hard way, but seeing it all summarized with clear succinct rules is great. 8:15, wouldn't assignment have to copy data in that example? Call by assignment makes so much sense!

[@ivolol]

What happened is the `+` on the right hand side copied the data (under the hood its like "hello".__add__(" there") returned a new value) and then the = just referred to that new data. But the = by itself never copied any data. It just referred to the new value that ended up on the RHS.

[@joseville]

@@ivolol thanks, ok that makes sense. The '=' itself didn't copy the data, it was the '+'.

[@rifatbhuiyan2543]

15:45 "yeah that was that weird time that that list changed and I don't know why" Exactly that's why I'm here.

[@Glicerol]

20:00 Names have no type & Values have no scope - really cool "duality" :)

[@jonassteinberg3779]

ned is the man. find him on #python on irc. dude is always down to help with whatever. he's like the world's free python tutor, it's really quite remarkable.

[@serhiy2020]

To illustrate that last point, this is what happens when you write a function like this: def func(nums = [1, 2]): nums[0] += 1 print(nums) >>> func() [2, 2] >>> func() [3, 2] >>> func() [4, 2] ..and so on

[@arthurpenndragon6434]

This example does a great job at making the theory part understandable in practice, thanks!

[@thomasw4422]

If I'm understanding correctly, it's Becuase nums is only made when the function is being defined, but the change to it is made every time the function is called?

[@BayesianHorse]

I think at 11:30 the problem of understanding the docs is that the __iadd__ operator should return the result of the operation(docs.python.org/2/reference/datamodel.html#object.__iadd__), which may or may not be the same object. The particular detail of list's += behavior is not mentioned, but "extend" is supposedly the equivalent of a[len(a):] = [x], which is a range assignment and thus modifies the data. It's unfortunately not equivalent to a = a + [x], which is unfortunate I think.

[@saurabh75prakash]

Must watch for a Pythonista.

[@pranjalmittal]

Excellent talk. Thank you. One question, in reference to 19:25, where you suggest to make a new list in function body.. Why not just pass the function a tuple instead of worrying about not modifying list arguments. I would assume it is better to correct things at the data type level (rather than taking care every time variable of that data type is used)?

[@aaronhall8039]

Pranjal Mittal Your suggestion is a perfectly legitimate approach, and that's what I would recommend if you're just replacing things. However, I think Ned is talking about giving advice over IRC, in which case you just want a quick solution so the asker thanks you and goes away instead of dragging the whole channel into a discussion of mutability.

[@nirajraut9408]

@@aaronhall8039 Hey, at 18:34, there is a little reference created on the frame object while executing _return a_list._ If the memory of the frame object created for the function _append_twice_good_ is reclaimed, why isn't the memory of the "little" reference reclaimed even if it is present inside the frame object? In other words, how is it that the memory of the "little" reference not reclaimed and it can hold the reference if it's present inside the frame object and the memory of the frame object is reclaimed?

[@aaronhall8039]

@@nirajraut9408 That's an interesting question - I'm not quite clear on what you're asking, though. If you mean "val" isn't reclaimed, that's because Python uses reference counting and there's a reference in the outer scope to the object val points to. Similarly, a_list isn't reclaimed because now there's a reference to it in the outer scope as well. If objects that you think should be reclaimed aren't, then there must be a reference to it somewhere (perhaps a reference you created to try to track it?) Another point, ints from -5 to 256 are interned so they are always available and aren't duplicated in memory. Finally, don't worry so much about tracking memory in Python unless you've got a memory leak and are running out of it...

[@solamarpreet]

Ty ned

[@thomasgandalf4111]

great pres, thanks. just my 5c there to towards the end, "python is call by assignment". that's really a non-statement because in principle both values and references can be assigned. that said, python is call by reference as can easily be seen by the nice drawings on the slides - all names are references to values (objects, in fact). function calls assign (or bind) these references to names in the local scope. that is the very definition of call by reference. (it kind of puzzles me why people get confused by this -- there is really no difference to e.g. how pointers work in C, or how objects are passed in Java)

[@drygordspellweaver8761]

Passing a value by reference and modifying said value does not always leave you with a modified value in Python, unlike in C++ where it always leaves you with a modified value. Hence it’s both/neither.

[@miraculixxs]

@@drygordspellweaver8761 yes it does allow modification as long as the object is mutable. Mutable objects include e.g dict, list. It does not include str, int, float or any other scalar type. Regardless what gets passed on to a function is a reference to the object, never its value. That's the text book definition of call by reference. Mutablity of objects is a different trait entirely. It has no relation to and no impact in the nature of passing values on function calls.

[@drygordspellweaver8761]

@@miraculixxs I never said "it does not allow modification". I said it does not ALWAYS leave you with a modified value. "Regardless what gets passed on to a function is a reference to the object" ... Everything is technically an object in Python. Did you even watch the video where he is explaining the specific implementation of a value? "Mutablity of objects is a different trait entirely. It has no relation to and no impact in the nature of passing values on function calls." Okay- who or what are you even arguing against at this point? Your replies have nothing to do with the context of OP's conjecture and my response.

[@miraculixxs]

@@drygordspellweaver8761 Well yes, mutability is the key concept you seem to be confusing when you say Python does not always change the received object. The reason it does not in these cases is because the object is not mutable (e.g. a string), hence it cannot be modified. It is still passed by reference though. Call by reference and mutability are different concepts, we should not confuse them. Call by reference means passing, well, a reference to something. That's what Python does, always. Modifying an object requires for it to be mutable, regardless of how it is passed. That's a property of the object, not the calling style. You seem to imply that Python's way of variable passing in calling a function is somehow distinct from "by reference" *because* the received values are not always modifiable. That would be an inacurate conclusion - as noted above. Call by value oth means that a copy of a variable's value is created (typically by pushing the value to a stack, where the called function pops it off from). Python never does that. Actually there is an exception: in a multiprocessing setup, passing an object to a target function follows in fact call by value semantics, in that the value gets copied and reinstantiated in the receiving process. The actual function call again receives a reference though.

[@drygordspellweaver8761]

​@@miraculixxs I think your confusion here stems from misinterpreting my statement "Hence it’s both/neither." I'm referring to both/neither "call by assignment" and "call by reference", not "call by value/call by reference" as you seem to think.

[@easy2useinsurance79]

At 9:34 He says nums.append(7) does not make a new object, but x=x+1 makes a new object. But nums.append(7) does change the list to have a extra value at the end right?

[@Am_Ende]

Yes, append modifies the list in place.

[@TommyCarstensen]

I would like to drink a beer with Ned :) He seems friendly and knowledgeable :)

[@gametimewitharyan6665]

Came here from "Beyond the basic stuff with python" :)

[@amnanajib8167]

For anyone having a confusion between what is told in 11:11 and 18:26: Actually the __isadd__ method is run under the hood of x += y and and not for x = x + y. So, if you reproduce the example under 18:28 with a_list += [val, val ], it should work fine, since then you are just mutating the val of a_list.

[@aaronhall8039]

I've got my Python gold badge on StackOverflow, and I watched this talk from beginning to end, and I have no nitpicks to make over it. Good job, Ned.

[@ChrisLaffra]

At slide 18, at 13:40 into the talk, Ned is avoiding the use of an explicit iterator (probably due to time limitations). However, I still would have been more formal, not use sequence[...], and instead say the for loop is the equivalent of: it = iter(sequence) while True: try: x = it.next() except StopIteration: break something(x)

[@aaronhall8039]

Chris Laffra Sure, that's correct. I think his level of sophistication was fine, though your example makes it even more clear without the hand-waving. To further refine your example I would call next(it) instead of it.next().

[@drygordspellweaver8761]

You are the student, not the teacher. Humble yourself.

[@michaelstreeter3125]

Remember: *"Names have no type; values have no scope"* - is this the genius of Python? I'm going to have to go away and think now.

[@drygordspellweaver8761]

“zip turns a pair of streams into a stream of pairs” Ned is brilliant

[@_elkd]

For the 2D list in the last slide, here is Amy's talk section covering that kzbin.info/www/bejne/qXmXiXlspbB-o80

[@RonJohn63]

23:56 IOW, no pointers?

[@Fire_Rice]

Lol I was dealing with the list problem today. I thought I found a glitch or something xD

[@Lolwutdesu9000]

At 22:32, the guy talks about the two versions of the 2D list code. I just tried them out, and the "bad" one runs about 5 times faster. Why on Earth is it the "bad" one?!

[@henryzhang2992]

Because then I believe that you then create references to the same list, which would mean any modification would affect all the rows instead of just that one

[@vwwvwwwvwwwvwwvvwwvw]

What is the answer to the puzzle? Start->8549176320->Null ? Significance?

[@einSteppenwolf]

The digits are in the alphabetical order of their English names.

[@movax20h]

a_list += [2, 3], that it iadd returns self, is generic guidline of iadd operator: object.__iadd__(self, other) "[...] These methods should attempt to do the operation in-place (modifying self) and return the result (which could be, but does not have to be, self). [...]" The fcat that x = 7, x += 3; produces new value 10, is because type int do not have iadd (in-place add) at all. Python tries x = x.__iadd__(3), but because __iadd__ do not exist, it does x = x.__add__(3) (which will success), or if that doesn't exist either, will try x = (3).__radd__(x). So the x += 3, is not special at all for int vs arrays. It is because x += y, is not equal to x = x + y in general at all (they can both make x completely different values at the same time, and make x refer to different or the same things at the end at will).

[@bethfiore]

How did he figure this all out!?

[@paraglide01]

Came here from kittens meet puppy's first time.

[@gustavonobregadossantos1223]

I came here because of "Automate Boring Stuff with Python"! If this is true for somebody else, I have a question. What did you do next, after "Automate Boring stuff with Python"?

[@drygordspellweaver8761]

I just started writing scripts and looked into software design principles

[@tbabbittt]

Python is a class oriented programming language.When you do something like x = 23 x becomes a integer class complete with all the properties of the integer class. x never refers to a value, a name refers to a class instance that lies in the namespace and the data that lies within that instance. I think that when he says value he actually means class instance or he is not talking about Python at all.

[@tbabbittt]

Remember that x=25 and y=25 create two seperate instances of the integer class in namespace. x and y are not referring to the save value but to different class instances. When you say x=y that says that y now also refers to the class instance in namespace (the dir() function lists namespaces). Now x can affect the properties and methods (no values here) of the class instance in the namespace.

[@iffinity7704]

> Remember that x=25 and y=25 create two separate instances of the integer class Not entirely true. For some range of values, it's the same instance Python 3.5.2 >>> x = 25 >>> y = 25 >>> x is y True >>> x = 123456789 >>> y = 123456789 >>> x is y False And it's logical, it will be a big overhead to create new int object every time you need 0 or 1 or 10. IIRC, strings for up to length 20 behave the same way.

[@drygordspellweaver8761]

Ints from -5 to 256 are always interned so they won’t duplicate memory use.

[@patriotir]

19:28 he recommends to create a new list every time but isn't that counter-intuitive? creating a new list means consuming more memory space while we can easily mutate the list. to me the first function is more efficient.

[@pwbpeter]

I thought this was about names in Monty python Mr ftang ftang ole biscuit barrel and such like ,Imagine my dissapointment!

[@aberezh]

I am still not following how this is different from e.g. Java. In Java it is clearly "pass by value", just some values happen to be references. Well, in Python all values are references, still in the original meaning of the terms it is "pass by value".

[@aberezh]

I other words, once you speak about variables, they are all references. Once you speak about names, they are "bound to values". So, as I see it now, the talk does not deny the existence of variables, just suggests an alternative way to look at what happens in Python.

[@shivampandiya]

legend.py

[@CristiNeagu]

I would say Python definitely uses call by reference, but it deals with references in a different manner than C/C++. So while in C/C++ you know that passing a value by reference and doing something to it will always leave you with a modified value at the end, in Python that is not always the case. But it's definitely 100% call by reference.

"Python has no variables" is not a silly way to explain "variables" at all, Ned.

[@nedbatchelder]

Jürgen Erhard We'll have to agree to disagree.

[@nirajraut9408]

@@nedbatchelder Hey, at 18:34, there is a little reference created on the frame object while executing _return a_list._ If the memory of the frame object created for the function _append_twice_good_ is reclaimed, why isn't the memory of the "little" reference reclaimed even if it is present inside the frame object? In other words, how is it that the memory of the "little" reference not reclaimed and it can hold the reference if it's present inside the frame object and the memory of the frame object is reclaimed?

[@andysondur]

19:10 doesn't work. Just adding return statement to the def function doesn't change the reference (nums still refer to the original nums list). The 'best advice' doesn't seem to be best advice after all. The append command is the best way to modify a list inplace.

[@m1k1a1]

But he didn't *just* add the return statement. The previous slide showed how he used the returned value: nums = append_twice_good(nums, 7) . That re-binds nums to the new list. As someone who used other languages before Python, this is the way I wrote my first function to modify a list.

[@JethroYSCao]

Reread his code. He didn't only put "return" as the last line of the append_twice_good function, he put "return a_list"

[@andysondur]

@@m1k1a1 thanks (a year a later)

[@andysondur]

@@JethroYSCao Yes, I realised my mistake. Thanks for the help.

[@anudeep13]

Came here from WhiteHat Jr course. Yes, I am a 5 yr old.

[@Alpacastan21m]

Automate the Boring Stuff.

[@tonyravindran]

Came here from Automate the Boring Stuff with Python Video 16

[@lucashallo4760]

AUTOMATE THE BORING STUFF

[@daronieogarniaro]

AUTOMATE THE BORING STUFF