pip freezing (==) isn't safe (intermediate) anthony explains

[@thisfred]

This is one reason I like pip-tools: it adds the hashes to the requirements files, so they can be verified on install.

[@James-ln6li]

This is my preferred method as well

[@TravisPickle]

im new user to poetry, but i hate it doesnt autoload .env files

[@madumlao]

Doesn't sound like a bug per se. From the name of the option itself, a "build number" would be the same version of the software, just possibly rebuilt with different options / settings and not justifying a version bump. I'd say if you fundamentally can't trust the source of your package at the level of package semantics, you have much bigger problems, and a requirements specifier isn't supposed to be used as a security tool to protect against that.

[@anthonywritescode]

I never claimed it's a bug -- just a massive anti-feature

[@lonterel4704]

Maybe its a bug in packaging.SpecifierSet? Because pip uses it internally, maybe its already fixed in main branch of packaging repo, but there is no release

[@anthonywritescode]

nope, this is a very intentional feature

[@qexat]

@@anthonywritescode can you link something that shows it is intentional? thx

[@anthonywritescode]

it's specced: peps.python.org/pep-0427/#file-name-convention and implemented -- it'd be hard to "accidentally" add such a feature

[@con-f-use]

I was gonna say pipenv and poetry do locking using the checksums, might be a "solution" though has other problems part of which you mentioned. At least the lockfiles of those tools take into account, that users might need multiple wheels for the same version of a package (e.g. one v2.0.0 wheel for mac users, another for v.2.0.0 on linux, yet another one for windows, ...) and lock those with their checksums, too.

[@anthonywritescode]

yeah I also talked about this -- they address the risk, but pull in another set of problems which is that the lockfiles *break* on any package change

[@con-f-use]

@@anthonywritescode Only solution, I can think of is, if developers signed packages and you could lock-file your trust in them. But given how slow python packaging evolves and how difficult it is to maintain much needed backward compatibility, that will be a pipe dream.

[@mfeickert07]

@@anthonywritescode I'm a bit confused by what you mean by "break" on package change. You mentioned around 5:00 "if someone uploads a wheel later that isn't in your hash set pip will fail", which is the part that I don't understand what you mean (by "fail"). If you're using pip's "secure installs" recipe with something that has hashes enabled then you want to install that exact wheel and not something that has been patched later, no? The only way that I see having hashes break something is if the wheel that you have the hash for got _deleted_ from the index, and that you would certainly want to know. I just gave a lightning talk at SciPy 2022 on reasons people might want to consider using lockfiles and treating data analysis environments in a similar manner to deployment environments for applications, so I'm interested to hear people's criticisms and feedback around the concepts of lockfiles in general as well as the multiple implementations that exists in the Python world today.

[@anthonywritescode]

any time I've added a wheel to an already-released package I get a hatestorm of pipenv and poetry users whose lock files have been "broken" by introduction of another hash (the package now resolves differently). I think the actual fix to this problem is: - remove build numbers - release files are immutable after creation (with a small grace period (~an hour)) - forbid deletions (yank only) - native support in pip freeze for the collection of hashes of a release

[@mfeickert07]

​@@anthonywritescode Ah this is a pipenv and poetry implementation specific problem then (I think!). (Also sorry to hear about the hatestrom). If you create a lockfile with pip-tools's pip-compile and require hashes you just restrict your installs _to the hashes in the file_. So if other wheels get added after the fact they are ignored when installing from that lockfile. I believe this can be shown with pip-tools by just removing a hash (that you don't need for your install, like a Windows hash if you're on Linux) from the lock file and then trying to install from it. It will happily do so as it is only querying the hashes of the file, and the fact that there are other hashes available on the index (the one you deleted) doesn't matter. I think your suggestions are pretty good. This isn't the forum to bug you more on this, but as some of us have been thinking of writing a PEP related to this would you be interested in at least looking at it? If so, I'll follow up in the far future.

[@dumbqs]

Anthony how do you implement hash validation?

[@anthonywritescode]

for pip or in general? for pip I've used `pip-tools` which works well -- otherwise `sha256sum --check` (in bash) or hashlib + secrets.compare_digest in python

[@ChristianBrugger]

Isn't it still that packages can be overwritten on PyPI? By removing and re-uploading it under the same name. Or has this been changed?

[@anthonywritescode]

you haven't been able to replace the exact filename for at least 5 years -- I think it even wasn't possible before warehouse either

[@atugushev]

All hail `pip-compile --generate-hashes`! ^___^ Thanks for the great topic! That's funny how easy to inject a malicious package into a private PyPI index server.

[@amir.hessam]

I like poetry export without hashes but idk if it solves the problem you mentioned here; gotta check it out

[@anthonywritescode]

it does not

[@-morrow]

pip/pypi is a security nightmare. it's worse than npm, just was lucky enough to not get the headlines.

[@alexprengere]

So it would be nice to have an option to "pip freeze" to include the build numbers?

[@anthonywritescode]

probably better would be to just kill the feature entirely -- I've only ever seen it used once and new version numbers are free and plentiful

[@soberhippie]

You can also download all your required wheels and keep them in your git repo along with the project, and use pip install --index-url=./my_wheels_folder

[@anthonywritescode]

you mean `--no-index --find-links ...` ; and checking in GBs of wheels is a poor use of git (and doesn't help you if you ever want to upgrade). you may have solved one problem but you've introduced many more

[@soberhippie]

@@anthonywritescode Yes, sorry, that is exactly what I meant ) "Wheels in git" is not my preferred solution, but it may work for the cases when it's difficult to transition from pip to other solutions. Either pip or wheel also has the ability to download wheels for the exact python version and platform combinations, which makes this approach relatively working. For me, Poetry looks like a significantly better solution; however, I haven't found a way to transition to newer versions of Python when project dependencies are managed in poetry. As far as upgrading dependencies with Wheels-in-git is concerned, you can have several versions of the same wheel in your repo, and run your tests in tox with different dependency combinations, until the newer one gets stable, and then switch to the new combination and remove the old deps. But it is a hassle, I agree.

[@soberhippie]

@@anthonywritescode Having looked through the other comments: with poetry: it is a cruel thing to do to upload a modified wheel with the same version number; with the 'wheels-in-git' approach, the form I've seen was one where the project was developed using virtual machines on developer computers, so there was no problem of dependencies being incompatible with the developer machines.

[@dhruvakashyap]

Wait, this sounds like some sort of society crumbling bug waiting to explode, right?

[@anthonywritescode]

yeah kinda :S -- for a sufficiently popular project a takeover and build-number upload would potentially reach millions very quickly

[@johnflynn880]

Another point for conda. “Conda env export” pins build hashes in addition the version. The whole python packaging ecosystem is such a mess and always has been. Conda is the only solution that makes it make sense in spite of PyPAs best efforts to keep it bad.

[@phxees]

I probably used it wrong, but Conda was too heavy for us. Plus, their packages weren't up to date w/ all vulnerability fixes according to Dependabot.

[@domochevsky4703]

how we can get a hash of the package?

[@anthonywritescode]

it's in the url on pypi -- or you can download it and use `sha256sum` or you can use a freezing tool (such as pip-tools)