My recent experiences with archiving irreplaceable home videos has prompted me to post a reminder for you to ensure your valued home videos are freshly archived in a format that is not obsolete.
Most of my family’s home video was recorded to VHS or Mini DV tapes. Over the years I used a Pioneer DVR to record these tapes to its hard drive then burnt DVD’s using high end Verbatim or TDK discs. I made several copies of each and distributed them to family members for redundancy, about 180 discs total.
I have recently upgraded my computer which can process video conversions 10 times faster than my last, so I decided it was time to convert all the DVD’s to MP4 files and store them on HDD’s. I found the burnt DVD’s that were around 10 years old struggled to be read and the 12 year old DVD’s would not read at all. Very worrying as I hadn’t considered these discs to fail in that time.
Luckily, after many hours of trying other methods, I found a magic program that, given some time, managed to recover all the data from the unreadable discs. Roadkil’s Unstoppable Copier http://www.roadkil.net/program.php?ProgramID=29 is old but still works on Win10.
I discovered my Pioneer DVR still had unburnt family videos on its HDD but would shut down after being on for a short time. The HDD in these will not play in any other player, even the same model. I was able to obtain another unit and rat a power supply board from it to get mine going.
I then found my old Panasonic MiniDV camcorder which had tapes with it I hadn’t transferred, including my own wedding. I transferred three out of six remaining tapes before the camera went fatally faulty. It is not easy to find a cheap working Mini DV camcorder now and professional transfer services are expensive. I have managed to borrow a camera but this whole ordeal has been an eye opener.
I have made a number of copies onto HDD’s and sent a couple offsite. I feel HDD’s are the safer way (and cheapest) to store important data and should be ok in the long run if they are re-written at least every few years.
Good points. Another aspect is the compatibility of old media with new systems. A good example are enterprise tapes. Every few years the standards change, and periodically there is a disconnect whereby an old standard drops off the compatibility list. Enterprise sites spend/spent considerable funds to refresh archival tapes as well as copying them from old to new media every few hardware generations to stay supported by hardware and software, not just because of reliability.
I believe the venerable IDE HDD might be the oldest lasting common media that can still be read (media and formats) on modern PCs albeit with an IDE adapter on newer ones, roughly 25 years running. In comparison consider the tape formats of 1990 and current hardware/software availability to read them, and generations along the way.
As for being re-written every few years, HDD utilities read and re-write every sector on an HDD when doing surface scans; eg that is part of routine long term media maintenance. For readers who discount the (un)reliability of data, I occasionally encounter long forgotten files (even on NTFS, a very reliable file system on very reliable HDD) that can no longer be recognised by applications as their original formats, although bit dumps are possible and a serious techie or specialist recovery program could possibly restore the original.
Optical disks should surpass HDDs while USB sticks might seem attractive, but a troublesome HDD (sans catastrophic head crash) can usually be fully or partially recovered by specialists at some cost, but a USB stick works or does not. As for optical products,
I’ll add some context regarding my failing DVD’s. They were dye based -R high end brands burnt at a speed of 4x, stored indoors in a cool, dark, low humidity environment and stacked in a spindle case. The redundant copies stored in similar conditions at other addresses behaved the same way after the 10 year mark. From a total sample size of 180 discs, 30 of them were over 10 years old and proved problematic. Even after assuming the publicised life expectancy was overly optimistic, I’m still a little surprised.
Some of the high end DVDs were sold as ‘archival quality’. If I remember correctly they claimed a 50 year life.
Thank you for your timely reminder. I had issues recently trying to transfer from miniDV tape to HDD. Gave up in the end and wasn’t prepared to pay the outrageous prices being asked by businesses to do this.
I’ve also had issues with multipe SATA HDDs (Western Digital) which I use for backups failing after just a few years. They were stored in a dark, cool, low humidity environment. Fortunately, I do multiple backups, so the data was not lost.
It would seem that for the average consumer there is nothing that can be relied on to last reliably for more than a few years at best as a backup repository.
That is a solid reason NOT to store backup HDDs in an unused state. When they are routinely used one usually has some notice they are degrading; when they are in a closet for a few years and connected, who knows?
That leads to a reason for cloud backup services that use enterprise quality products where data loss is all but non-existent because the online systems are RAID6 with tape backups for the most part, and some have duplication in geographically diverse locations to protect against natural disasters, terrorism, etc. Normal operation routinely tests the devices for file system and data integrity and readability. The downsides are network speed and data limits, and costs. The more secure and protected, the higher the cost.
Some of us, usually the more technically inclined as well as the most naive, often manage our own backup risks with more or less rigour as the case may be.
My backup HDDs get fired up at least once every few months with surface scans every (not so) often. Even so, [usually old] files occasionally are found to be corrupted, but as you wrote, when the backups are duplicated the odds improve.
What applies to backups also applies to archives, but is more complex (sinister?) since one expects to save something, and retrieve it in future, no worries in between. @meltam’s point is well made.
Out of curiosity I dug out a few old IDE hard drives I had stored in my big uninsulated Colorbond shed for the last 11 years to test. The shed temp gets into the high 50’s during summer and condensing temps in the winter. All three showed no signs of data loss. Also stored in there was a big box of audio tapes, all over 30 years old. I played a number of them on a high end cassette player I have and they still sound pretty good.
I know my sample size of three is insignificant but their storage conditions were woeful and may indicate a decline in reliability as data density increases and price decreases with these items. The audio tapes go to show how reliable the old magnetic tape systems can be and reinforces the trust put in LTO (Linear Tape-Open) cartridges for archiving.
LTO cartridges are very cheap for the data density but the tape drive can run into $$$$. It is probably cost effective to employ a transferring service but without the machine in your possession there is no way to verify the cartridges you have for archiving actually contain the data you expect on them.
One of the false economies - staying with old reliable hardware. Upgrading every couple of years keeps one within the ‘support zone’ so to speak, at least with tape tech. Stick on an old one for too long, and of course you want to upgrade to the latest - yet the old won’t ‘read’ for restores. I love it when the beancounter types say “but can’t we just …?” - A few times in my career it has been, no, you can’t - those 600 tapes you say you need to be able to read for the next two years have a couple of options - retention of unsupported hardware often one, re-dumping of a lot of tapes another. It’s really hard not to grin when the beancounters start to ‘get it’ … of course they have forgotten it all by the next sensible hardware refresh time … blame the technology is the cry of those bearing calculators …
Out of curiosity @meltam, are these drives portable types or SATA types used in a caddy?
I only use 3.5" drives mounted in USB 3.x caddies with separate power supply for speed and reliability. The portable units are much slower at transferring and seem less reliable in the long term.
They were 3.5” HDD periodicly placed into a caddy to back up, & then removed.
I have quite a few HDDs I rotate through so I don’t overwrite anything for at least a year. Depending on what of our data I am backing up, sometimes it’s a matter of creating a new folder to backup, or it could be a reformat & fill the disc.
I find out the HDD is defective when it’s turn comes up, and it is kaput.
I have taken them apart and couldn’t see any obvious problems. I have even tried swapping the PCBs between identical spec drives. Nothing worked.
One or two OK, but four in under a year? That’s a lot of money down the drain.
Its worth pointing out that solid state technology - USB sticks, SSD and the like - should NEVER be used for archival purposes, as these lose data if not plugged into a power source for extended periods. This has caused havoc in some long running court cases where such devices have been seized, examined, but not copied, and when used as evidence have proven unreliable.
A good point, additionally an archive not tested ‘recently’ I feel can be assumed to be lost data until it is again tested, where ‘recently’ is measured in months not years. This covers all forms of archive - even with secure climate controlled storage and multiple disparate media across physical geographic separation … paranoid archivists ‘re-archive’ regularly, I’ve seen it done annually or bi-annually on a large scale. It’s amazing how often even routine backups fail to deliver to expectations when tested, over weeks and months, not years - more often due to ‘pilot error’ but worryingly often enough due to media integrity/etc. Expecting to put any media down blindly for 10 years or so/more is an act of pure optimism test test test, and have more than one copy …
This is a huge problem due to manufacturers constantly upgrading and improving technology. Good for them but bad for us!
I have also experienced all those issues starting with the old VHS tapes transferred to DVD’s and also new video recorders not being able to read the DVD’s. Perhaps saving the videos on computers and laptops will be easier?
That is, if you want to make sure it’s going to be there when you need it you need to have multiple backups - including one offsite (e.g. using cloud storage). You will find that cloud storage providers deliver much more redundancy and security than you could ever hope to have on your own. In particular, any hard disk you buy today will be running its own error correction, as the density of data is such that they just cannot get it right every write!
My understanding of DVDs is that they can be extremely stable if they are kept out of the sun (and away from water, and not fed after midnight, of course). I am somewhat surprised that yours failed so quickly. Spinning hard drives can fail, but remain the most reliable local storage option other than magnetic tape (yes, it’s still around). SSDs should not fail quickly, because they do not have the same design issues as RAM - but will still leak electrons over enough time.
Finally, if you want to try to recover data I keep hearing about Spinrite on a security podcast to which I subscribe (Security Now, with the program’s author). It gets some good feedback, but I have never felt the need to buy it - yet. If you do visit the site, you may even look up SQRL - a possible password replacement, if it’s ever finished.
I do quite a lot of transfers from VHS and DVDs and find that quality differences can be quite large. I have transferred some VHS tapes of the same age - some pretty well perfect and others filled with static. I think we have to consider that there are different quality tapes, different quality players as well as different storage quality. I recommend that people transfer out of the old technologies as soon as possible rather than leaving old tapes and disks in storage until it is too late. I certainly agree with having multiple copies of priceless family videos. The “off-site” copy needs to be in a geographically different location whether it be “in the cloud” or another town. Whenever I see floods or bushfires on TV I wonder how many family movies and photos have been lost.
In regards to archival DVD and Blu-Ray media there is a type that LG I know at least has the burning support for and that is M-Disc. There are a couple of producers of the media Ritek and Verbatim are two of them. The supposed lifespan is 1,000 years. The original company was called Millenniata which went bankrupt in 2016 or so. The company assets were taken over by the creditors and it was recreated as Yours.co. I cannot find any recent reviews of the product/s with the latest being about 2014 but I don’t think it would be a heavily used media.
The media is available from Amazon at least as one store and I guess you could contact Verbatim or Ritek to find out other resellers of the product.
For the pdf’s about M-Disc that are still on the M-Disc site see:
The trouble is that storage of data by itself is insufficient to deal with a major solar storm. What exactly do you do when every computer on Earth is inoperable an irreparable - including those that hold the blueprints for building other computers and those that operate robots in factories?
That kind of solar event would turn the current pandemic into a mild incident by comparison. No telephones, most cars would not start, the electricity grid would be dead - of course no Internet or computing. What happens to a global society that is highly interdependent and reliant upon technology if all the technology dies at once? And no, we don’t tend to have backup devices buried in the Arctic tundra.
Even ships carrying cargo would suddenly find themselves all at sea - GPS is dead, and sure the captain can navigate using charts in a pinch but her watch is also dead, and so even 17th century navigation won’t help. (Ignoring all the electronics that are used simply to get the ship to move.)
A city starts to starve within a week if it does not receive regular food deliveries. Australia is fortunate enough to be self-sufficient in food - except it has to get from the farm door to the supermarket somehow. Horse and cart? And we are also one of the most urbanised countries on Earth, just to make things even more interesting. Surrounded by oceans on one side and deserts on the other.
Burying GitHub’s repositories in the Arctic makes sense in the long term - thinking multiple centuries into the future. It does nothing for the short term, and a large solar storm could take more than a century to recover from.
In part and in theory, the archive deals with that via the Tech Tree. In other words, it could allow an intelligent but primitive society to haul themselves back to our current level of IT. It would be a lot faster than having to rediscover and reinvent all the needed tech - but is no panacea.
The article contemplates these kinds of scenarios:
Anyone wanting to read the archives might need to have at least a basic understanding of creating a magnifying lens (something humans achieved about 1,000 years ago)
Even if in 1,000 years something dramatic has happened that has thrown us back to the Stone Age
That could be so. The goals of this archive are two-fold then, I suppose:
the archive survives long enough for that recovery to take place
the archive helps to shorten the recovery period.
There is a risk though that throughout centuries of disruption and dislocation, everyone forgets that the archive is even there.
Those encased in a Faraday cage may survive the solar blast. Some do exist for the purposes of super sensitive information storage beyond air gapping them. These may form the start of the recovery rung.
test test test, and have more than one copy …