Optical Media For Digital Data Archiving - My Complete Guide (Jan 2024)

[@henryj.8528]

Daniel, great info (I'm referring also to the entire series). However, if we look back at the actual problems we've had with long-term storage of media and data, finding working hardware that can read/playback the stored media is equally problematic. Simply having media that can be read years down the road is "necessary but insufficient." In fact, the problem with all potential storage solutions is viable hardware to play it back. (The record on the Voyager spacecraft included a rudimentary player.) In the case of HDDs, the magnetically stored information will certainly survive longer than the mechanics and electronics of the drive itself (personal experience). I made the first optical laserdisc in the US way back in 1981. In those days, small production runs were not stamped but were laser-etched glass masters. Those will last forever. However, try finding a player than can play them back. RCA Selectavision discs are still playable (unless they've been stored in an attic) but finding a working player with a usable stylus would be a challenge. In most cases, playback hardware still exists but transfers can be expensive and are getting more expensive every year (quad tape). If reasonably priced playback hardware is not readily available in, say 25 years, it won't matter if the storage media is still viable or not. In an attempt to get around this problem, I am also storing a few players (in hermetic containers with dehumidifying packets) in the hope that it won't be too much of a hassle to playback the media which (based on your research) will probably make it.

[@danielontech]

What an honor to meet! Laserdisc is fascinating technology! And yes... a worry. Love the idea of keeping a few on hand. I just bought a cheap Blu Ray player which I had intended to have as a spare but... I'm tempted now to do something along your lines and shift the plan to preserving it as well as I can.

[@AbsurdScandal]

You make a good point about how HDDs will keep the data on the magnetic platter much longer than the mechanical parts would be able to last - that's often overlooked. If one is willing to accept a migration strategy for data archival where every 10 years or so one moves data onto a newer hard drive, that would be sufficient to last for a lifetime. Optical media removes that hassle due to lasting longer, but there is the worry of whether optical readers will be available decades later to read the disks. Now blu ray and dvd aren't nearly as niche as laserdisk is, and (correct me if I'm wrong on this) are more widely used than laserdisk ever was, so if hardware for optical were to cease existing, it's not gonna be as soon as it was for laserdisk. Still, hard drives seem to be much more immune to hardwate extinction than most other forms of storage media.

[@henryj.8528]

@@AbsurdScandal Good points. I used HDDs as long-term storage and had mechanical failures much sooner than 10. years simply sitting on a shelf. And yes if you copied them over to another drive every year that would help. But I am trying to store video that will be playable in 30+ years w/o any effort. By playable, I mean fairly easy and cheap to watch (given the content). Even today one can get a dub from 2" quad for a couple hundred bucks, and other deprecated formats, but if it's too much trouble and too expensive to play back, it won't really matter if the media survives... So I am storing un-opened DVD players with the media in the hope that a USB or HDMI viewer will be readily available in 30+ years. (We still have SO-239 connectors, so I think I'm safe there.)

[@AbsurdScandal]

@@henryj.8528 Yeah, that's a bit of an issue for HDDs. From what I've read, there are multiple things one can take into account. For one, it seems some brands are of lesser quality compared to others. Seagate for example is infamous for some of the drives it makes being doomed to fail catastrophically, and is often regarded by some as being on the lower end of the quality spectrum. From the data recovery experts I've talked to, it looks like Toshiba is one of the better HDD manufacturer's out there, and that in their experience Tosh's drives tended to fail less over time. Specifically, the Hitashi / HGST line of drives is considered one of the best out there, and is widely present in Toshiba but also some Western Digital models. On the other hand, I've talked to people who've had sufficint luck with Seagate such that in their experience Seagate turned out to be the most reliable in comparison to other. So....that's interesting. Blackblaze (or is it Blackdaze?) also compiles statistical data for various failed drives they use in their data centers and Tosh seems to be somewhat better in terms of how long the failed drives lasted, but the amount of Tosh drives they use is noticably lower than Seagate and WD, so that could maybe be due to not having a large enough sample size. Either way, that's for drives used on a constant basis. Keeping drives in cold storage is another matter. From what I've heard, lubricant drying seems to be an issue, and so it may be beneficial to plug in an unused drive once a year or so and maybe read stuff on it for a few hours or over a few days to keep the lubricants stable. On the other hand I've also heard that modern drives in particular don't actually have lubricant problems so....yeah. Another issue is SMR vs CMR. Data recoverers seem to prefer CMR as it doesn't have TRIM enabled so deleted files can be more easily recovered, and it's also said CMR is more reliable over time. But I've also read articles and commentators say SMR is uniquely good for archival purposes as well when kept in cold storage....either way, most if not all 2.5" external drives use SMR, so if one wants to use external HDDs that's the most likely recording method that's gonna be used. What's more, some also say that 2TB or so is like a nice stable maximum for external drive storage space, and anything beyond that makes the drive less reliable or more fragile as it has to pack all that storage in small platters. This seems to be most likely true, and I wouldn't be surprised if higher storage capacity drives tend to fail more often than lower capacity ones for above and similar reasons. For what it's worth, I have a 500 GB internal laptop drive that's 10 years old and still going strong with the data on it intact, though it does have 18 reallocated sectors over that time. I also have an internal drive that's almost 20 years old, and hasn't been used for 8 years straight yet is still functional as of this year (last used it in 2016). So it seems whether a drive will survive for a decade in cold storage is up to the quality of the drive, maybe some of the above factors I mentioned, and some luck. You did mention you kept some drives on a shelf but which failed in a few years or so. A possible way to maybe mitigate this that I thought of would be to cover the drive in foil, or put it somewhere where not much dust can enter, and where accidental bumps or falls can be minimised, but so far I dunno whether that would make a difference to cold storage longevity. Besides dust, moisture, intrinsic industrial imperfections that eventually fail a drive, and other such stuff, there's also the mechanical parts themselves failing over the years, which is why some people recommend not keeping a drive for more than a decade without a new backup. I myself strongly prefer a data storage medium that doesn't have to be backed up with a new product every year or so, which is also why I'm looking into bluray burning and archival disks for long term storage, though I also haven't given up on hard drives either. There may unfortunately be some bad news for optical, since someone who pays attention to the market for this & is also into archival told me that they've looked into their bulk supplier and while BDR media has gone up in price significantly the number of drive makers has been reduced to basically one, with many rebrandings, all the while drive prices are going up, which is a sign that the technology is basically dead such that come another 10 years and it may not be possible to buy drives anymore while the media will either be from the old stock or a cheaper manufcaturer using expired patents to make lower quality products of questionable value. Furthermore, this already happened with floppy disks in the 90s where many 720 kB disks from the early part of the decade became unreadable in the same drive 5 or so years later Dunno how accurate this prediction is, but it's at least worth considering.

[@AbsurdScandal]

​@@henryj.8528Ok so I made a long reply but it's buried under Newest First. Hope you see this and catch it!

[@craconia]

Thanks Daniel for the updated presentation! It was great.

[@danielontech]

Glad you enjoyed!

[@TofuInc]

Something not talked about much, possibly because of the lack of first hand experience with tape media. Is that LTO is not backwards compatible beyond one generation (up to LTO-7 is 2 generations) . For instance if someone was to back up to an LTO-6 tape throw it in a box and forget about it for 10 years, they would not be able to go out an purchase an LTO-9 drive and read it. They would have to go out and specifically purchase a LTO-6 or 7 drive. That is also what makes optical media very attractive. I can purchase a new $20 DVD drive and read a CD from the 90's. Backward compatibility is always maintained. If the data cannot be read, it's effectively lost.

[@danielontech]

Excellent point!

[@stuCameraman1]

A very good breakdown👍🏼👏🏻I'm about to start capturing 34 Hi8 family tapes to apple prores via a Átomos monitor/recorder. Then I'm applying heavy colour correction, sharpening, and finally noise reduction. My final archive destination I've decided is BD-r in the form of a Authored BD-r video disc so I can send it to relatives around the UK so they can play it on their BD movie players or ps3/4/5 ect. So I'm guessing BD-r 50GB are still very good for compatibility with BD movie players🤔 I probably have around 400 dvd-R over the last 15 years that are still working fine today😂🎉 I do fully trust optical media over hdd as it only let's me down when dvd-R surface is scratched or chipped. Any suggestions on highly rated external usb BD burners? Thanks in advance👍🏼

[@MichelOnLine]

Daniel, I have watched quite a lot of your M-Disk video’s you made last couple of weeks. The 100/128gb limit of one disk and the worm state of the discs, let me to choose an second hand refurbed LTO-6 drive+hba card ( about 1000 euro) and a bunch of tapes (they are +/- 35 euro for 2.5TB) LTO-7 (6TB) is quite a bit more expensive for a second hand drive (around 2500 euro), the tapes are only 10-20 euro’s more expensive as LTO-6 ones. New drives are 3.5k to 6k euro depending on the LTO version, as you mentioned. The fact that the LTO tapes can be reused is quite welcome while using it as a backup medium, for archival that’s not really an issue. But the cost per TB should be a bit lower for LTO, if you use LTO-6+

[@Mae-nr7wr]

blu-ray actually used to be cheaper per TB before all those cheap 8TB HDDs flooded the market one of the reason i made backups with blu-ray was because it was actually cheaper per TB, but yeah ever since those 8TB HDD's became very cheap, the blu-ray market has never been competitive enough to beat HDD's in terms of $ per TB. i keep seeing news about new optical disks having been developed in labs, but they never reach the point where you can buy them

[@mindrover777]

Cheaper 8tb hdd. In which country? Be specific

[@Von_D]

36:30 I find it funny how Japanese writeable DVDs and Blu-rays are all explicitly said to be for video recording right on the packaging despite how strict Japan is regarding copyright (especially with regards to anime). Some of the units mention regular data storage too, but video recording use is pretty much always made very clear. Heck, not too long ago, I found a video from a large electronics retailer's KZbin channel explaining which disc type people ought to use with their DVR machines that save to optical rather than a hard drive.

[@jupeterczech1340]

I would NOT trust ANYTHING in the cloud! 😡🇬🇧

[@JamesSmith-ix5jd]

I think there was CD-Audio or something like that specifically for music that's how CDs became popular. But when I got my first computer and burned some music on CD I was confused why my music player could not play it...

[@JoaoFerreira-um7wf]

Hi daniel. What program do you use to burn archive discs?

[@danielontech]

Hey! Via Windows VM - BurnAware Premium On Linux/Ubuntu - K3B (I find it a bit buggy for BDR media so for those projects I use the VM). Nero and ImgBurn also seem to be very popular.

[@chrisw1462]

Why is the audio so distorted???

[@janisozols5255]

Over-compressed? I can understand everything at 10% volume

[@cmoneytheman]

​@@janisozols5255ill fix it

[@stankobulanov8428]

I don't want to be rude, but unfortunately optical media as a archival media is dead already for a long time. I have more than 20 years of experience with optical media archival. I've started 1999 with CD-R, then 2003 switched to DVD-R. Until 2012 i had 160 CD-Rs and over 1300 DVD-R from different brands. In 2018 i bought external LTO5 drive for 400USD from Ebay and then copied all my data from the optical discs to the tapes. From all those 1300 DVDs and 160 CD-Rs i had couple of them failed to read completely, mostly some movies i re-downloaded later on the Internet. Regarding LTO tapes - from that i have seen on your presentations i could say you have no idea what you are talking about when describing LTO tapes :) I fully understand your fascination about optical media as a technology for storing data, i am also fascinated :) but it has too many drawbacks. Will give just some examples along to the ones you provided: 1) no error checking during write - from all optical media formats only DVD-RAM is performing read right after data is written, that's one of the reasons why DVD-RAM is so slow For example LTO drives are verifying written data immediately after it is written on the tape and that's a standard! 2) speed - reading/writing to optical discs is SLOW! LTO5 tapes (the ones i use) have average read/write speed 80 - 130MB/s compared to couple of MB/s best case for opticals 3) longevity of the magnetic media - it stays magnetic much longer than you think :) etc, etc.

[@AbsurdScandal]

He did say that many optical disk types aren't good for long term storage, but that some media like blu ray or mdisk are of higher quality and intended to last much longer than other forms of optical media. And as for error checking, couldn't one do that by using burner software that checks if the burn is successful instead?

[@michaellin4553]

The right optical discs have lower bit rot and can be augmented with dvdisaster, lasting much longer. Plus, they have better random read access, and doesn't require tedious rewinding. The drives are much cheaper and more accessible than the ridiculously expensive LTO tape drives. Eventually, the old LTO drives will run out on the secondary market, and if your drive breaks, what are you going to do? Even now, optical drives support CDs despite being a very outdated optical format, while costing a fraction of a tape drive. I would only ever consider LTO if you can afford the latest standard and are working with very large volumes of data (corporate, government, or something like the Internet Archive) that justify the large upfront cost.

[@pxt22]

Man, the audio is terrible. It is clipping bigtime! So tiring to listen to. Absolutely unbearable. How on earth did you forget to set your levels correctly?

[@cmoneytheman]

I'll fix it