python 3.12 was ALMOST javascript

today we're talking about AttrDict, maintainability of cpython, and the burden of maintenance.
==========
twitch: / anthonywritescode
dicsord: / discord
twitter: / codewithanthony
github: github.com/asottile
stream github: github.com/anthonywritescode
I won't ask for subscriptions / likes / comments in videos but it really helps the channel. If you have any suggestions or things you'd like to see please comment below!

[@oakley6889]

As a wise man once said "when building a language, only add what you really need and set out to do, adding neat little conviences will lead to the most issues later". I dont remember who said it and thats paraphrased This is where JavaScript probably fails the most.

[@paulk314]

I think you may be thinking of Ryan Dahl from his 2018 JSConf talk "10 Things I Regret About Node.js": "What I've come to learn now that I'm aging is that whenever you're designing a program, there's things that you think might be cute to add in, you always regret those... don't do them!"

[@maksymiliank5135]

JS fails the most because of poor choices when adding a new feature and not being able to remove them in the future because it could potentially break some 20 year code on a website nobody visits anymore. Back when js was being designed dynamic languages where all the fuss, everybody wanted to use them. Everybody hated static types, they wanted everything to be determined at runtime with no safety checks whatsoever. The creator wanted to make the developer's life easier, that's why every number is a float and you can compare strings with numbers without any annoying conversions, because that's surely never going to cause any problems in the future. There is also the 'null' and 'undefined' which are basically the same but not the same at the same time. Now things have changed and statically typed languages like rust are getting popular.

[@modernkennnern]

Almost all of problems with JavaScript(which are a lot) comes from its original version. Newer features are generally well thought through

[@SamualN]

and python doesn't?

[@BradenBest]

If it adds a little context, JavaScript was written in 10 days and was supposed to be an implementation of LISP/Scheme in the browser. It only has C-like syntax because Brendan Eich's boss at Netscape told him to give it Java syntax. Hence why the language's name changed at some point from LiveScript to JavaScript (though now its name is technically ECMAScript or ECMA262). I think there was also almost a lawsuit over that. But one area in particular where the Scheme influence really shines through is in the prototype system and the way it implements object-oriented inheritance through these meta-objects rather than classes like how Java does it. This very much resembles LISP and LISP-like languages because of how it lets you essentially redefine large parts of the language.

[@neildutoit5177]

I'm literally a data scientist and I think Pandas is bizzare and I avoid it as much as possible. Fortunately I don't need it much because I mostly work text not tabular data, though I have been using it every so often for almost a decade. But man the syntax is weird, unpythonic, not even internally consistent, impossible to predict/guess. I feel like any time I use it I have to look up everything. In my experience you can go to SO at any time of day, search python, and at least 30% of the page will be "how do I do X in pandas". There's no other library in existence where people have such difficulty figuring out the syntax. I think the only people who really like it are people who learnt pandas before really learning much Python, and people with R/matlab backgrounds. But then again, I have an R/matlab background. And I actually enjoyed both of those. I've never enjoyed pandas.

[@AlexanderThomasHaselton]

THIS

[@DaveParr]

+1 for coming from R, and being happy, and then going to pandas and being confused.

[@yuchsiao2]

pandas is the very library I always need to look up for everything before every time I use it. The syntax is so unique and the error message is so arcade. I has always been trying to avoid it except for simple table display in notrbooks.

[@yeelam4291]

that's why I use polars😂

[@hansdietrich1496]

@@yeelam4291 Yes, but polars has it's own issues. Mainly that they want to stop you from doing row access by indexing. Just stupid.

[@paxdriver]

That "object_pairs_hook=lambda attrs... " made me laugh so hard lol "yeah, it's *little* clunky" 😂

[@GrahamAtDesk]

I totally agree with all your points. Thank god for all the folks who are looking after consistency in Python.

[@_Robin_]

I like to go by a rule which is that if something is supposed to be complicated or expensive, it should look complicated or expensive; so that it's discouraged from being used or at least approached differently. We are also currently starting to suffer from over-abstraction of concepts in Python (and web-dev), so it's nice to see that they didn't go through with this feature.

[@TheAulto]

Oh no, yeah pandas is not a prime example of good API (not entirely its fault tbh). It’s an awesome tool, very convenient but so inconsistent in its API it’s not even funny. Same deal with matplotlib (which inherited much of its API from matlab). It’s one of the best we have, but far from perfect. Loved the video! Glad to see them popping back up.

[@darrenlefcoe]

python 3.12 broke matplotlib: kzbin.infoYkZxLanqWmI?feature=share

[@BrianJorgensenAbides]

It is kinda hilarious. The stdlib is so huge that people have tried to reimplement the same thing a bunch of times. I agree with you that this style is not broadly useful for design. Hehe, even though I use argparse all the time, I am not sure I ever thought of using Namespace/SimpleNamespace for this purpose. I've rolled my own solutions over the years for exploring APIs interactively (back when HATEOS and _links were in high vogue). But like you say, that's almost a toy. (And was meant to be)

[@neildutoit5177]

Why was HATEOS a toy? BTW the HAL Google group is still going.

[@BrianJorgensenAbides]

@@neildutoit5177 nah, my tooling around it was the toyish thing. HATEOS is a great idea with the worst possible marketing. Anyway the punchline was really that my bespoke solutions have always also been named AttrDict. It’s an obvious thing to want, when ya want it! 😇

[@princeroshan4105]

This a good example for me to understand what consistency means for a big project like this because this can be helpful `obj.a.b.a` can be helpful for diff tools to find diff between deeply nested dicts but again as you explained this can be done using object_pair_hook

[@hakansvensson8314]

Lot's of comments here bashing pandas for the dot notation, but I actually like this feature a lot, even though I realize that there are downsides to it. Main reason being convenience - not having to type square brackets and quotation marks (which sucks on a keyboard with Scandinavian layout), and also that autocomplete works even in a terminal.

[@hansdietrich1496]

Same for me. Also, if you have tables with familiar columns, where you control the name, the difference between "structured object" and "unstructured container" gets blurry. To me, the video is a bit too doctrinal. nevertheless, AttrDict is something that shouldn't belong to json in the same way as SimpleNamespace doesn't belong to argparse. It should both be in dataclasses or collections.

[@Squeemos]

My biggest complaint about pandas is when people do the attribute access from tables with dot notation. I find it really bad practice to do so especially since in tabular data you can have column names be strings with spaces, and it's like a weird patchy thing to do, especially since it may have the same name as something already attached to the dataframe. Whenever I find code from friends or other students using dot notation, I almost immediately tell them to use the key access instead. Great video! Also really neat to know that SimpleNamespaces can handle it anyways

[@hansdietrich1496]

If you work on tables with a familiar set of columns and you have control over the naming on a regular basis: The line between "structured object" and "unstructured container" gets blurry. Also in terms of convenience: To me it's highly understandable, to replace the hundreds of clunky [""] with a simple dot. Especially if you're not on an English keyboard and typing the brackets forces you to leave the main typing position for your fingers.

[@etis398]

For debugging or rapid prototyping in notebooks I use dot notation, for finalized code I use brackets. Another pitfall of dot notation is that you cannot assign with it

[@amir.hessam]

cool; I recall that you tweeted about this. QQ: how do you explain `Enum` in python that you can look up entities with both `dot` and `bracket` notations

[@anthonywritescode]

enum also suffers from the "not enough maintainers" problem and has quite a few weird quirks in terms of the object model. the non-identifier attributes of enum are also kinda problematic -- I'm currently trying to fix a bunch of usages at work. imo it shouldn't have `__getitem__` and just let people use `getattr` if they need the weirdness

[@TreyKeown]

I believe also causes problems for how Python does generics now, if there were ever a desire to make that work with Enum. The Enum metaclass implements __getitem__, so Generics would be a behavior change.

[@BradenBest]

Much like how you can't write a dynamic language these days without having some support for javascript syntax (because you need to support JSON), you also can't implement a compiled language in general without that language "speaking C", because although C does not have an ABI, C is so deeply rooted into the way that everything works that you cannot make anything work without making it binary-compatible with C programs. C and JavaScript are unique in this regard, as C is the language that everything in Unix is built on top of and JavaScript is the language that everything on the modern web is built on top of. These languages are so deeply embedded into everything that you're very likely to know (or at least be more than a little familiar with) them. All this is to say that saying python was "almost" javascript because it did something weird in the name of JSON support is kind of like saying Rust is almost C because when you look under the hood of the compiler you see a BUNCH of stuff related to ABIs and libraries that interop with C and C programs, not because the developers of Rust chose to make it compatible, but because it literally won't work if they don't.

[@migueldeirala]

Clickbait.

[@2sourcerer]

How could that JSON thing not go through PEP?

[@dixonstroi]

Totally agree, I wish Pandas did not do this. It's complex enough without that added ambiguity.

[@gardnmi]

Did they forget that dataclasses are part of the std library?

[@fred.flintstone4099]

The 'json' module in Python sucks, it cannot deserialize JSON into a class or a dataclass. For that you need third-party libraries.

[@gardnmi]

@@fred.flintstone4099 Wouldn't be too much of an effort to add a to and from json method to the dataclass module. A way better approach to this problem imo.

[@crides0]

@@gardnmi There is the `dataclasses-json` package that does that. Maybe they can add that... (would be very useful; not having auto-deriving these impls is.. not very scripty)

[@jlpeyret]

Or pydantic. Which has dict() and json() methods.

[@fedoraguy5252]

Great video good job explaining the issue. We can't just add features for the sake of adding features.

[@kintisheff]

agree, great catch - i hadn't even noticed this was trying to sneak into 3.12!

[@tswdev]

Just case across your channel, pretty cool stuff! Q: What do you think of pythons module import system? For me, its the most limiting factor of python. In C# (just for examples sake), you can access a class from basically anywhere within a namespace without circular imports. This is specially useful when using classes as types (as one should) rather than as "aggregators of functions". In python you have to use something like var: "MyType" or use fence check your imports to see if you are on "IDE" mode or "RUN" mode (i forget what its called now) to avoid importing those types at runtime, where you will hit circular imports. Do you think its a good idea for python to mature on this front?

[@anthonywritescode]

c# has the advantage that there's a compilation step and a smart compiler to influence all of that. unfortunately python's roots are in scripts so its "module system" (and I call it that loosely) is really just a bunch of side-effects and a mega-global (sys.modules). a lot of scripting languages have basically the same problem I wish it weren't so -- but I don't see a good way to fix it without greatly limiting what you can do at the "module scope" and making it more difficult for scripting

[@maximefilippini383]

Did that structure not allow for keys that could be cast to integers then? Since calling them via the dot notation would be a syntax error.

[@anthonywritescode]

indeed -- it only allowed for valid identifiers -- everything else still had to go through key notation

[@RogerValor]

This is super really easy to write yourself. I did this already in python2, where such an object came in handy for a specific parsing task. If I need this I just basicly write it down.

[@subjekt5577]

There's also AttrDict for when you actually want this semantic, a super common and short library .. oh wait was that literally the exact same thing getting added in here?

[@sopidf]

great content, thanks!

[@zanityplays]

😭 was looking forward to this

[@tonychia3824]

how do you install multiple versions of python so you can test out the new python features? do you use brew install or pyenv?

[@anthonywritescode]

I use (and maintain) deadsnakes

[@tonychia3824]

@@anthonywritescode it seems deadsnakes is not supported on mac. is that correct?

[@anthonywritescode]

yep -- just on ubuntu. I don't really use a mac all that often

[@danielcs88]

As a heavy Pandas user I refuse to use the dot notation for columns and find it unnecessary.

[@EvanOfTheDarkness]

I have a similar class, that I use a lot, I call namespace. It's very similar to the built in types.SimpleNamespace, but I use a single dictionary to initialize the attributes, and not the keyword args.

[@kuyajj68]

Thanks for that AttrDict. Ihave a TOML config and it was really annoying to access dict with deep levels. Will try at work tomorrow 😂

[@jfb-]

i've used my own implementation of this for quick and dirty code (e.g. advent of code solutions) but yeah shouldn't be encouraged in production code so not a good fit for a standard library

[@oscarljimenez5717]

In javascript this differences don't matter because if you use a formater it will transform everything the way you want.

[@cleverclover7]

I was waiting for you to finish to say exactly what you did about pandas. It's not even consistent within pandas. DataFrames still rock though.

[@lukekurlandski7653]

Insane that this didn’t have a PEP.

[@po6577]

as a junior python user(not even dev), it's fun to just hearing this stories(include those in the comment), I like how people are friendly in this conversation. Keep up!

[@anthonywritescode]

hah -- they are NOT friendly I've just moderated away ~100 very nasty comments

[@mrswats]

Yeah, so glad it was removed

[@isodoubIet]

I don't think you need to tergiversate much re pandas; it's certainly an 'organically grown' library with useful, extensive functionality, but at times clunky, inelegant, and with poor defaults. Even one of the core tasks in pandas -- reading and loading from a csv -- is harder than it needs to be. By default, if you load a csv and immediately write it out, it'll be different from what you loaded unless you include the incantations: df = read_csv('foo.csv', float_precision='round_trip') df.to_csv('foo.csv', index=False) That's sort of besides the point, though. As a whole, I'd say the library exemplifies what you mean by "no is temporary, yes is forever". Pandas just does too much and a lot of its more annoying behaviors are probably impossible to change at this point.

[@blueman333]

I would like to hear your opinion on polars or any other good libraries that implement "data frames" in a proper way.

[@mydetlef]

There is no reason to include something like this in the standard library. It is easy to implement and use as a separate library. This also provides the incentive - if necessary - for different implementations.

[@sergeyf6536]

Pandas be pandas lol

[@user-zt2lp6hq7l]

it is not difficult to make dict into js like object

[@tunafllsh]

I was about to comment about pandas but your point is convincing.

[@JohnZakaria]

It's Raymond Hettinger one of the python Gods

[@anthonywritescode]

gods or fallen angels?

[@moo_goo]

hadn't heard anyone had a less than positive opinion on Raymond.... (was sad to see it was him though) definitely an oversight/ too assuming on his part but you aren't contributing unless you make a few mistakes along the way. especially a feature we straight up don't need. Thank god for roll backs.

[@anthonywritescode]

not my story to tell but there's some really problematic things that raymond has done and personally I've not had a positive interaction with them

[@moo_goo]

@@anthonywritescode oh i see. eek. Thats unfortunate to hear. My interaction is highly limited. Always a shame to hear the quality of character of someone youve learned a lot from being problematic.

[@JohnZakaria]

I never hat a personal interaction with Raymond, but his talks "writing Idiomatic code" and "python class toolkit" were very inspirational

[@antoniofernandodiasjunior8896]

when python will have multiline lambda functions, still waitin'

[@Alexedtion]

Now I understand why Mojo, Python superset is not open source til they finish the language implementation. Python sometimes is a mess like JavaScript

[@anthonywritescode]

I think mojo is closed source because it's a vaporware PR stunt

[@hansdietrich1496]

I guess the biggest mistake of mojo is, that it's a superset of python and not a subset. Superset introduces totally pointless new syntax and forces separation instead of a steady progress of the whole python universe.

[@kaninchengaming-inactive-6529]

@@anthonywritescode true

[@kintisheff]

I think python docs actually recommend using collections.namedtuple over types.SimpleNamespace in cases such as this - >> import json >> from collections import namedtuple >> doc = '{"a": {"b": {"c": 1, "d": 2}, "e": 3}, "f": 4}' >> hook = lambda mapping: namedtuple("foo", mapping.keys())(*mapping.values()) >> obj = json.loads(doc, object_hook=hook) # note that this works with the object_hook parameter of json.loads, not with object_pairs_hook >> obj.a.b.d 2 >> obj.f 4 >> type(obj)

[@darrenlefcoe]

this line is horrendous: hook = lambda mapping: namedtuple("foo", mapping.keys())(*mapping.values())

[@shhossain321]

it is fine as long as it is not added in built in objects.

[@yorailevi6747]

i really wonder why raymond did that, that's unlike him

[@KangJangkrik]

I like how he also recorded his arms, good for hand fetish 👍

[@COLAMAroro]

I think having it as an *option* (while not being enabled by default) was a good idea. It's not because "JavaScript does it" that it makes this immediately a bad idea. The thing that shock me the most is that Python isn't supposed to be "pure OOP", but convenient. What happened to duck typing ? No later than a few hours ago I just injected into a library's object some attributes for me just because it was convenient. If you want absolutely pure OOP, there is a lot of awesome other options, C# and Java in mind, but this would have been really great for people (like me) who writes a 200 lines scripts here and there.

[@anthonywritescode]

there's plenty of space on pypi for such a bad idea -- but it does not belong in the standard library

[@COLAMAroro]

@@anthonywritescode that's assuming you even can access pypi tho (because your project manager is doing the right thing and vetting dependencies, or simply because you are behind a corporate proxy, and are only allowed vanilla python)... And I still can't see why it's an issue if it's not even enabled by default

[@anthonywritescode]

perhaps you missed the whole part of the video about maintenance and surface area? cpython already suffers from "not enough maintainers" and adding more junk that most people should not use is going to make that worse even if you can't access pypi it's trivial to copy the ~5 line recipe or use the already-existing alternative if you want to shoot from the hip. or you can use a real data serialization layer instead of a hack

[@COLAMAroro]

@@anthonywritescode That's a +241/-1 patch. I've seen worse in terms of maintenance burden. Especially so when if it's going to be 100% unused in your opinion, so raising 0% of bug reports. Well, I guess that's fair if the actual part is 5 lines, that's a file I can easily copy from GitHub and import manually (which is what I do at my job, because no pypi). But I still can't use a "real data serialization layer", nor does I want to. I'm not going to write 3k lines of DTO just to transform my JSON into a CSV that I can email to someone who doesn't even need to know what Python is, because they are business workers, not IT. It looks like you don't want to understand that people might use Python as a *scripting* language. Where it's one of the most convenient, because it provide those kind of tools out of the box. I might be wrong, but I assume you also think that duck typing is a bad idea ? And that being able to monkey-patch objects is dumb ? Because Python here is ***GREAT***. Where writing a complete application is NOT the end goal. Some people just writes code that needs to work once or twice then they throw it away. Is using AttrDict on a full-blown app a bad idea ? Sure, but for a "just once app", this feature could be great.

[@b_0rk]

@@COLAMAroro 200 LOC that includes new APIs is a maintenance burden, especially for codebases used by millions of people and kept alive by volunteer labor

[@_scourvinate]

I thought we were supposed to move away from JavaScripts terrible design decisions not add them to other languages 😂

[@CyrilCommando]

If it were JavaScript, maybe I would actually use it.

[@kamurashev]

So the good could happen… alas

[@sweetshit4931]

Interesting

[@Fabian-ff4cq]

No that would've been so useful I'm kinda disappointed.

[@Mekuso8]

If you know the structure on JSON objects in advance and want to treat it as such, you can always just use Pydantic. So having this feature is kind of redundant.

[@darrenlefcoe]

it is interesting that you draw the distinction between square brackets for "unstructured" data and the dot method for "structured" data. However, i find the dot really convenient for avoiding typo's and massively helpful for intellisense and would prefer it to be natively there.

[@anthonywritescode]

then use a parsing layer to get structured data!

[@darrenlefcoe]

@@anthonywritescode whilst that is the "proper" way. A parsing layer would add more inconvenience (clutter) and would probably lend itself better to large code blocks.

[@adityag6022]

Thank you sir

[@y7o4ka]

python drama

[@nexovec]

Should have named it the PeekabooDict, as when you assign to the built-in methods, the original method then goes peekaboo. Where are these people getting these ideas OMG :D

[@cbbcbb6803]

Python may be the best programming language ever created. I still do not like it. I do not think a lot of JavaScript either.

[@fred.flintstone4099]

I think Python is a great language for non-developers, for laymen, for scientists, tinkers, mathematicians, biologists, physicists, doctors, mechanics, and others who are not programmers but may need to write some code. It is good for small scripts. For professional developers I think Python sucks.

[@CottidaeSEA]

@@fred.flintstone4099 That's basically my stance on it as well. I think it's great for when you have a basic job that needs to be done. I think one of the issues is the horrible performance. It's not a big issue for smaller scripts that you just run, they complete and you get the result from a file or db inside your main system, your system might use more resources than necessary, but it's not like it matters too much. However, when everything is Python, it's really noticable.

[@fluktuition]

​@@fred.flintstone4099 As a scientist, I can say that I still dislike Python. Modeling anything that requires 10 files starts becoming a huge mess. Think of things like machine dynamics, fluid simulations, computer graphics, chemical synthesis.

[@LukasSkywalker_]

yeah, so no problem with the feature itself but it was the speed of the PR being approved without revision. The JavaScript "problem" is inexistent then.

[@anthonywritescode]

maybe you missed most of the video but the feature itself is a problem too

[@theerrornan4456]

Love the way you look down on data scientists and JavaScript devs!

[@anthonywritescode]

to be clear I am not doing that. data scientists have different goals that don't always require the same level of reusability and structure when exploring problems

[@philsburydoboy]

While I would never use this, I don't have any issue with it. Developers should be opinionated, not languages.

[@anthonywritescode]

that is my point actually. this is the standard library not the language

[@minandychoi8597]

‘…very much like javascript … in a rather problematic way’ - that’s redundant 😜

[@frankyboy4409]

cmon that would have been an actually useful feature ... i hate how cumbersome parsing json is in python compared to literally any other language I've come across

[@fallingintime]

Just use type dicts,

[@blueman333]

As a data scientist and a pythonista, I feel a little offended at the pandas joke. Although I find the use of AttrDict to be completely counter intuitive, I don't feel the same for pandas. I guess that is because we consider each column as separate "features", so it kind of makes sense to access those column objects through dot notations. I will agree that the first time I worked with pandas, I found it weird to be able to access the column with dot column names. Also it was hilarious when I realized you can `df.col` but can not `df.column(space)name` because python just does not allow that (that non sense which exist in R). But trust me, having a worked a lot on a vast amount of data science projects `df.col1 = (df.col2 + df.col3) / 5` seems a lot more clean / pythonic than `df["col1"] = [(v2 + v3) / 5 for v2, v3 in zip(df["col2"], df["col3"])]`.

[@anthonywritescode]

"I've been lazy a lot so trust me it's good actually"

[@b_0rk]

pandas python behaving different than cpython python is weird and bad

[@sv_gravity]

% python3 -m this

[@aznpwnsalot]

Would it not be df['col1'] = (df['col2']+df['col3'])/5? Seems just as clean to me.

[@blueman333]

@@aznpwnsalot Yes it does. But somehow I still prefer df.col and it does not bother me. Maybe I have worked too much on pandas to know the difference. When I think about a dataframe df, it does not appear to me as "just another python object". I find naming a column "sort" far worse than, pandas ambiguous outcome of `df.sort` in that case.

[@LCTesla]

still looks like javacript to me. glorified, overhyped javascript with too many otherwise worthwhile projects wasting effort on it. this language stagnated the code world for +/- 20 years limiting the potential of everything it touches.

[@ShadowKestrel]

The more I learn about python, the less I want to learn about python. There aren't many languages I feel that way about.

[@fred.flintstone4099]

Good point. The basics of Python are good, and Python is good for when using the basic things in it, but for more advanced stuff it sucks. It is great for non-developers because, but for professional developers it sucks.

[@davidtaylor6124]

I used to feel that way but decided to dig in and use it for some 'resume driven development'. And now I quite like it. Some things drive me mad, and I can't think why people consider it a 'simple' language - it's insanely complicated and has all sorts of footguns. But it is pretty good to code with if you have a good IDE and use type hints. 3 years ago I would have unconsciously chosen java to write just about anything other than firmware. Now I automatically choose python unless there's a good reason not to. Plus everything is JSON now so python's dicts are awesome. The pandas API is a shocker. I can never do the simplest thing with it without looking at the docs and SO, and I use it every few months.

[@DrewryPope]

Saying it's existing functionality is just, not a useful comment. It's a programming language of course you can make it do that.

[@anthonywritescode]

so let me understand your point of view here: you're saying that because a programming language can do anything then garbage should be in the stdlib?

[@fltfathin]

i wish `dict.get_value('the.path.that.you.want', None)` or something but oh well