Use Shelve For EFFECTIVE Persistent Data Storage In Python

[@Matt-es1wn]

For someone who self taught with python and has built loads of stuff using barebones knowledge, these tutorials are so amazing. Ive spent so long writing messy code wishing i knee a better cleaner method, and this guy teaches it so well

[@Alister222222]

I wasn't aware of this, and think it is both elegant and useful.

[@JohnSmith-nm1jk]

Oh God. I've wasted so much time in my daily work which involves editing and loading large amounts of data. I've spent tens of minutes almost daily to reload things that could have been stored this way. Thanks a lot!

[@6midlan]

Is it impossible to use shelve to write concurrently to a database across multiple threads? Even if I make sure that each time I write to file I use a unique key?

[@Виталя-н8о]

Watch it from Russia, thank you good person!!!

[@rogerskiome9138]

7:34 I am aware that when you shelve a class instance, you do not have to have the class definition within the same python scrip where you will open the shelve.

[@TheSzybas]

Does it support search like TinyDB? Or do you have to use filters or loops to look for specific keys or values using structural pattern matching?

[@Isa-dw7wo]

what editor are you using?

[@NoidoDev]

Not sure if I want to use this in many cases. If I want to load a set I can you load the file, and I can still modify the file as a text file in some editor.

[@yafethtb]

In what occasion it will be benefit to use shelve? Is it better using this to create a kind of database than using sqlite3 to create a database? Is it better than storing data in JSON format?

[@aeghohloechu5022]

You can think of it as the sqlite3 of mongodb. It implements dbm, which is one of the earliest NoSQL databases created. However since it stores pickles it is a bit more dangerous, as it opens your code up to arbitrary code execution, and that's about it. But that's also it's selling point in a way. You're not limited by the type limitations of json, or the rigid schema and normalisation horrors of SQL.

[@DrGreenGiant]

Is the database file portable? Particularly across architectures?

[@FidosTeasers]

Wait, would it be possible to store a keras.Sequential this way? Just theoretically?

[@darrenlefcoe]

i was going to make a video on this, but this is a great video. Well done... the only question is, when would one use persistent data storage as a use case ?

[@DrGreenGiant]

Logging of raw data would be very helpful, to be used later for a replay simulation. Especially useful if that db file is portable so can be used on one particular hardware then replayed on another, something I do a lot at work.

[@SkyFly19853]

Does it improve the performance?

[@furrane]

Could you shelf the Fruit class itself and then retreive it ?

[@mahmodaldahol]

Amazing 😍

[@ellukayou]

Nice

[@freepythoncode]

I love your videos so much 🙂❤

[@Indently]

Thank you!

[@LaCompulab]

Thanks for your videos. For me the key is: is it safe in a multithreading scenario?

[@thatsunpossible312]

The docs say no concurrent read/write but concurrent reads are ok. So I guess multiprocessing is fine for reads, but for writes you would need to let each process have it’s own file and then combine the shelve dictionaries in the main thread.

[@LaCompulab]

@@thatsunpossible312 thanks

[@elpiloto100]

You say that pickle can only pickle one object, but you can simply call pickle.dump multiple times for unlimited number of objects. The only advantage I see for shelve is that you don't have to remember the order the multiple objects are pickled. So simply to me, shelve is like a dictionary whereas pickle is like a list.

[@Indently]

Essentially that is what it is. "A shelf of pickled objects"

[@Chalisque]

Under Windows, with cygwin, gdbm is available, but not under anaconda, so it seems.

[@emre42960]

What extensions do you use in vscode?

[@Indently]

This is PyCharm, and I don't use extensions. When I do use VSCode, I use life refresh for my JavaScript websites.

[@wrichik_basu]

Is creating foreign keys, indices, unique constraints, multiple tables in one DB etc. possible using this library? If yes, could you make another video covering these?

[@aeghohloechu5022]

It's not SQL, it's just pickle dumps.

[@Meige24]

why your pycharm looks different its cool

[@murphygreen8484]

Could this be used instead of an .ini file for a configparser?

[@aeghohloechu5022]

It can be used for arbitrary code execution, so just be careful

[@murphygreen8484]

@@aeghohloechu5022 yeah, I would only load my own pickled files

[@philtoa334]

Nice.

[@Collins183]

Nice

[@rushhour6444]

hey man that intelisense type hint dropdown is sick! it clearly distinguishes between classes and methods and stuff. id really love to install it. does anyone know what it is?

[@Indently]

I believe it's part of the experimental NewUI they have for PyCharm

[@thatsunpossible312]

We’ll now I feel like an ass inheriting UserDict and implementing this myself.

[@Indently]

Going through uneccesary effort to create our own implementations of a feature will probably never be avoided 😂 BUT, at least you get a much higher appreciation for the built-in modules.

[@thatsunpossible312]

@@Indently to be fair I looked into shelve after watching your video and the inability to support concurrent writes limits my use case. It’s the reason I used on disk storage for my pickles instead of a database. But thanks for this video, this might be my go-to now when running into memory issues. I’ll benchmark it against my solution.

[@Indently]

That is fair

[@zebraturner7103]

Can i store a lot of data in one: db[a] = 1, 2, 3 etc

[@aka1021]

Try it 😡

[@Chalisque]

I think so: the 1,2,3 will be turned into a tuple (1,2,3) just as if you did x = 1,2,3 db[a]=x

[@JordanMetroidManiac]

@@Chalisque Yeah, you can pickle a tuple, just like any built-in or user-defined object in Python.

[@tfr]

Another profile picture update??

[@Indently]

And not the last

[@rishiraj2548]

👍🙏👍

[@xushenxin]

I think it is garbage. There is no reason for Shelve to exist, except causing confusing. I don't want to know this.