Git Annex Is The Coolest Program You've Never Heard Of

[@aacolive]

0:23 Git Annex 1:20 Sync files between local folders 3:28 Turn the directory into an Annex repository 5:40 Git annex add 7:12 Getting both annex repos in touch (Like a Tinder match) 8:46 Sync files between repos via link 10:04 Getting the real file (--content) 11:26 Using the Gitlab as another example

[@tejing2001]

Git-annex is absolutely amazing. You really have only scratched the surface. My personal favorite thing you can do with it is distribute large files intelligently through repos on external drives, aka sneakernet. Each repo keeps track of what other repos have and need during syncs, so you can set the external drive to "need" anything that the other side of the transfer needs but doesn't yet have, and then the sync --content will load exactly what needs to be carried to the other side, automatically. Once it gets the other side, the content will of course be loaded by the target, but it will also be cleared from the external drive, since the target repo now has it, and it no longer meets the criteria.

[@ChristianF2]

Pro

[@123anonymous456]

Where to find a detailed article or video showcasing this use case?

[@griof]

A veru good use of git-annex is machine learning and data science. In my company we use git-annex to synchronize machine learning models and huge data sets that git can't handle. We storage them in whatever cloud provider and download them locally on demand. Super useful

[@nathanielsabanski]

git annex really needed a modern quickstart guide. Best video you've ever done mate.

[@hydejel3647]

One of those programs which you probably didn't heard of but are incredibly useful: entr. It watches a file (or files/directory) and if any change was detected, it runs an arbitrary command. Super useful.

[@cafkafk]

The timing of this was so perfect. I was just about to dive into the docs of this, but overwhelmed I ended up opening youtube to procrastinate and... Godspeed DT, goodspeed.

[@0x007A]

I am curious what use case git-annex addresses that rsync does not provide? If the data source is offline or otherwise inaccessible and you need the referenced "bigfile", what happens when you try to pull the full content of the referenced document from another computer?

[@rsmith31416]

As far as I understand, git annex keeps an inventory of your files without actually storing their contents unless you want to on demand. This is useful for huge files (100GB+) that cannot be stored easily but a version history of the repository is still needed across computers. In short, it is not a backup solution. It is, for the most part, a distributed synchronization system of metadata which can also retrieve the actual files if you really need them.

[@MagyarUS]

The big one is you can version control the files.

[@leftaroundabout]

Yes - Git Annex really is a program more people should be using! But two notes of caution: 1. `git annex sync` has a habit of trying to be too clever, performing automatic multi-way merges. For experienced Git users who use commands such as `rebase`, this can lead to weird and undesired behaviour, perhaps inexplicably rolling back changes. I personally usually use the more manual `git annex copy` or `git annex get` commands to prevent this. 2. The ability to version-control lots of big files can make it tempting to use it for _lots and lots_ of automatically generated data files, and you think that this repo can still be checked out on a resource-limited machine (by just not fetching the big files). However, because each file is represented by a symlink, which file systems don't store very efficiently, even such a repo can end up taking a significant amount of extra space and in particular inodes. One workaround is to pack the data files into fewer, bigger `.tar` archives and only put those in Git Annex, though this isn't a great solution either - because Git Annex is not based on diffs but always entire files, replacing one of the files in an archive would mean a new compy of the whole archive needs to be stored.

[@dany08011]

One of those game changing apps for me is noisetorch, it's great to suppress background noise when using the mic, works really good.

[@shatterstone3045]

Unrelated, but I just want to say that I did my first successful installation of vanilla Arch in a VM without archinstall yesterday!

[@gladwinmohlamonyane4033]

Congrats dude 🎉

[@gladwinmohlamonyane4033]

I used to do that too, but I build my env for work so I use the archinstall command even though I dislike how it handles partitioning my drive. Mainly because I will def add a lot of bloat 🤣

[@tejderha]

What is the difference with Git LFS

[@DataLad]

Git LFS uses an all-or-nothing centralized approach. Git-annex is a truly decentralized network. It excels even when not all data live (or can live) in the same place.

[@DataLad]

Also, git-annex can use Git LFS as *one* of the many special remote types it supports

[@tejderha]

@@DataLad thank you 👍🏻

[@DevDungeon]

Have you used jigdo before? In Debian, it's the `jigdo-file` packae. It is used for downloading massive files in pieces. Debian uses it for downloading the full set of packages. For example, the 19-DVD set that contains every single package. I have an offline copy of that and wikipedia just in case the internet ever goes away. It's about 170G of storage for both the full Debian repo and Wikipedia combined which can fit on a single thumb drive. Just try to think about how many minds and man-hours and how much information that is. In a way, you could consider that thumb drive one of the most valuable items in the entire world.

[@toxiccan175]

That's awesome. I've always wanted to accomplish something similar. How do you navigate Wikipedia offline? Are there any other sites you've archived?

[@rahilarious]

I've seen jigdo mentioned on debian but never made an effort to understand it. Maybe a video showcasing it would be helpful

[@rmcellig]

Excellent Derek!!! 🙂

[@DaraulHarris]

Git annex is how I used to manage my music, using various free warehouses.

[@Kludgedean]

I like the sound of that, could you give me a heads up on where I can acquire these my dude? :) Here's hoping

[@DaraulHarris]

Essentially, use the rclone remote to store your music on dropbox, google-drive, etc. It can encrypt them, too. I had made a longer comment, but I don't see it :/

[@rafeu2288]

@@DaraulHarris I think that youtube sometimes eats the comments that include external links, that may be why your comment disappeared. Did the same for a time myself, but now I copy-paste my comments in a file before sending them, that way I can tweak them if KZbin doesn't like the links or what. Good luck with that. :)

[@squ34ky]

How does this compare to git-lfs?

[@huantian]

Yeah exactly, as soon as I saw this I immediately thought it was similar to LFS LFS seems more supported than annex though

[@leftaroundabout]

The main difference is that with Git Annex, checking out contents of large files is by default separate from checking out the repository at a given commit. This makes it a bit more difficult to use than git-lfs, but it also gives more fine-grained control about where the contents actually are stored, in particular useful when you have huge repos and still want to be able to quickly make clones on a resource-limited machine. Also, Git Annex repositories can be used without having the Git Annex program at all (it's really just standard symlinks but under version control, you can view the contents and in principle even copy around the actual files manually).

[@AndersJackson]

I followed and used git-annex from when it was developed. It was even better when it had Jabber support, but it is still amazing.

[@krishenbhatti]

I still haven't figured out Git completely

[@DistroTube]

You and me both. :D

[@alexandrosvangelatos9979]

Same... Especially since github changed the login process. I cant get in via the terminal :/

[@gamerking64]

Maybe a Git tutorial soon?! Would be very helpfull

[@lawrencedoliveiro9104]

Nobody has. My philosophy with regard to open-source tools is, I figure out bits of them as I need them. This is how I work with Emacs, Git, Bash, the Python library, CMake, Debian packaging ... everything.

[@drishalballaney]

​@@DistroTube can you please make more videos on git basics ?

[@sergeynikiforov8012]

Can anybody tell me how to configure the shell like in this video?

[@TecnocraciaLTDA]

why use git to sync folders in your own system when there is rsync? git would be necessary if you want to track the changes made in that files and have the option to rollback...

[@yarikoptic]

Well -- I might tend to overuse git, e.g. keeping all my configs in git, and using etckeeper (from the same Joey who develops git-annex) for keeping /etc under git. One use case for git-annex could be "on-demand throw away rsync". E.g. if I want to try something out, I would create a local clone, 'git annex get' only needed files, "experiment", see the result, and possibly bring it back. With rsync, it would be a bit more cumbersome.

[@theodorealenas3171]

Hey that looks awesome! But what if someone wants to pull your big file from gitlab? It's on your local machine so your machine needs to be online when the other person wants your big file, no? It does sound proper, because gitlab gets rid of bloat, but is it convenient enough to use it with git lab? Some comments talk about use cases of git annex, and they convinced me it's a useful tool, but I want to know more about the online aspect.

[@brunoais]

Same question

[@tatotick8513]

I have been using git-annex for 10 years now, and is my most trusted way of storing all of data. I consider myself a git-annex "expert" so happy to help anyone if you need some help with it.

[@theodorealenas3171]

So you can link to a big file that's been deleted and it's only kept in .git? Is it a way to link to an old version of a big file?

[@FARDEENKHANQWE123]

creative thumbnail there buddy...

[@cheako91155]

RCS, it's git for a single file... Great for /etc or if for testing you want to replace something in /use/bin with a wrapper script.

[@brunoais]

For what you did, usually I'd end up using git-lfs. Also, were git-annex files actually uploaded to gitlab?

[@123anonymous456]

TBH, I would have preferred a short demo with actual huge files created using dd which then get modfied slightly and synchronized over a low-bandwith connection.

[@haj2.025]

Your on the edge of audio feedback today.

[@jfftck]

I had to use Git Annex many years ago, but have never had a use for it for quite awhile.

[@Flackon]

you rarely hear of git-annex because people use programs that supersede it like git LFS, rclone or restic or what have you (depends on the use case)

[@leftaroundabout]

But git annex unifies those use cases in a way that is more powerful than any of those tools, even when they're combined.

[@tatotick8513]

The reason git-lfs is known more is purely because of marketing. Git annex doesn't have the marketing power of git lfs which is backed by one of the largest corporations in the world. In actual functionality, git lfs is unusable once you have gotten used to git annex.

[@Flackon]

@@tatotick8513 Do self-hosted git services have good support for git annex? because Gogs and Gitlab have built-in support for git lfs out of the box, which is a big factor in my adoption (not to mention Github itself, of course)

[@tatotick8513]

​@@Flackon It used to be fully supported in git-annex but was deprecated due to not having sufficient volume of users. However, the important thing is that this only means that the files cannot be stored in gitlab. It can still be used to sync the git repo. I personally self host a gitea server for the git repo, and a minio server for keeping track of the files. It doesn't practically change much since a "git annex sync --content" magically works once the setup has been completed in either case.

[@academicalisthenics]

I am currently using git lfs for this kind of stuff. Is there any meaningful difference between those?

[@Fanaro]

Maybe compare it to git-LFS?

[@anantgupta1188]

Git Annex is a game changing program as it is a haskell program*

[@leonardonovara9348]

Hello DT, please, make a video about Lazygit, a tool that I didn't know existed but now I think I need it.

[@PicyPoe]

Terminal looks so sexy. What theme is that?

[@shastro6939]

Nice

[@DistroTube]

Thanks!

[@THIRSTYGNOME]

it's neat, but I don't think I would use Git that way. I would rsync or sftp "large" files around. I think it would be more impressive if it took a large file and chunked it in to smaller archive chunks in Git, and recreate the item back into the original file after running a git pull. This could allow you to version control larger archives, and get around git size limits, and help on upload bandwidth saturation Say you have a 5 GB encrypted tar.gz file. Program would turn it into say 10 small 500MB files. You could then run the tool on the other end to get the large file back. Now that could be cool, but might go against terms of service for GitLab/GitHub/etc.

[@leftaroundabout]

Git Annex actually uses rsync under the hood, but it saves you the trouble of having to remember which file to copy from where to where.

[@WilliamLDeRieuxIV]

Git was never meant to be a cloud storage solution -- no wonder why it doesn't do well with large files (which really shouldn't be in your repo anyway).

[@LionKing-qp1lk]

FIXED: git doesn't do well with large files (that's why they shouldn't be in your directory under git)

[@auroradraco9974]

Damn, looks sick

[@alankjohn9263]

fill the comment section with all the other underrated cool programs but offcourse foss only!!

[@DeshierArchitecte]

Meh, why not jut use rsync?

[@slavko5666]

Anyone got an idea on how to sync game saves from Retroarch?

[@vincentas1]

soyface thumbnails always get a click from me

[@ShaunakDe]

Thanks for demoing this

[@lanpartylandlord6123]

quit it w the soyface thumbnails brooo! ur channel is better than that!

[@mrCetus]

Have you used jigdo before? In Debian, it's the `jigdo-file` packae. It is used for downloading massive files in pieces. Debian uses it for downloading the full set of packages. For example, the 19-DVD set that contains every single package. I have an offline copy of that and wikipedia just in case the internet ever goes away. It's about 170G of storage for both the full Debian repo and Wikipedia combined which can fit on a single thumb drive. Just try to think about how many minds and man-hours and how much information that is. In a way, you could consider that thumb drive one of the most valuable items in the entire world.