CHOICE membership

Archiving and Backup of Data, Home Video, etc - Electronics and storage media have a finite life


#1

My recent experiences with archiving irreplaceable home videos has prompted me to post a reminder for you to ensure your valued home videos are freshly archived in a format that is not obsolete.

Most of my family’s home video was recorded to VHS or Mini DV tapes. Over the years I used a Pioneer DVR to record these tapes to its hard drive then burnt DVD’s using high end Verbatim or TDK discs. I made several copies of each and distributed them to family members for redundancy, about 180 discs total.

I have recently upgraded my computer which can process video conversions 10 times faster than my last, so I decided it was time to convert all the DVD’s to MP4 files and store them on HDD’s. I found the burnt DVD’s that were around 10 years old struggled to be read and the 12 year old DVD’s would not read at all. Very worrying as I hadn’t considered these discs to fail in that time.

Luckily, after many hours of trying other methods, I found a magic program that, given some time, managed to recover all the data from the unreadable discs. Roadkil’s Unstoppable Copier http://www.roadkil.net/program.php?ProgramID=29 is old but still works on Win10.

I discovered my Pioneer DVR still had unburnt family videos on its HDD but would shut down after being on for a short time. The HDD in these will not play in any other player, even the same model. I was able to obtain another unit and rat a power supply board from it to get mine going.

I then found my old Panasonic MiniDV camcorder which had tapes with it I hadn’t transferred, including my own wedding. I transferred three out of six remaining tapes before the camera went fatally faulty. It is not easy to find a cheap working Mini DV camcorder now and professional transfer services are expensive. I have managed to borrow a camera but this whole ordeal has been an eye opener.

I have made a number of copies onto HDD’s and sent a couple offsite. I feel HDD’s are the safer way (and cheapest) to store important data and should be ok in the long run if they are re-written at least every few years.


#2

Good points. Another aspect is the compatibility of old media with new systems. A good example are enterprise tapes. Every few years the standards change, and periodically there is a disconnect whereby an old standard drops off the compatibility list. Enterprise sites spend/spent considerable funds to refresh archival tapes as well as copying them from old to new media every few hardware generations to stay supported by hardware and software, not just because of reliability.

I believe the venerable IDE HDD might be the oldest lasting common media that can still be read (media and formats) on modern PCs albeit with an IDE adapter on newer ones, roughly 25 years running. In comparison consider the tape formats of 1990 and current hardware/software availability to read them, and generations along the way.

As for being re-written every few years, HDD utilities read and re-write every sector on an HDD when doing surface scans; eg that is part of routine long term media maintenance. For readers who discount the (un)reliability of data, I occasionally encounter long forgotten files (even on NTFS, a very reliable file system on very reliable HDD) that can no longer be recognised by applications as their original formats, although bit dumps are possible and a serious techie or specialist recovery program could possibly restore the original.

Optical disks should surpass HDDs while USB sticks might seem attractive, but a troublesome HDD (sans catastrophic head crash) can usually be fully or partially recovered by specialists at some cost, but a USB stick works or does not. As for optical products,

this might be informative for the general reader.


#3

I’ll add some context regarding my failing DVD’s. They were dye based -R high end brands burnt at a speed of 4x, stored indoors in a cool, dark, low humidity environment and stacked in a spindle case. The redundant copies stored in similar conditions at other addresses behaved the same way after the 10 year mark. From a total sample size of 180 discs, 30 of them were over 10 years old and proved problematic. Even after assuming the publicised life expectancy was overly optimistic, I’m still a little surprised.


#4

Some of the high end DVDs were sold as ‘archival quality’. If I remember correctly they claimed a 50 year life.

Thank you for your timely reminder. I had issues recently trying to transfer from miniDV tape to HDD. Gave up in the end and wasn’t prepared to pay the outrageous prices being asked by businesses to do this.

I’ve also had issues with multipe SATA HDDs (Western Digital) which I use for backups failing after just a few years. They were stored in a dark, cool, low humidity environment. Fortunately, I do multiple backups, so the data was not lost.

It would seem that for the average consumer there is nothing that can be relied on to last reliably for more than a few years at best as a backup repository.


#5

That is a solid reason NOT to store backup HDDs in an unused state. When they are routinely used one usually has some notice they are degrading; when they are in a closet for a few years and connected, who knows?

That leads to a reason for cloud backup services that use enterprise quality products where data loss is all but non-existent because the online systems are RAID6 with tape backups for the most part, and some have duplication in geographically diverse locations to protect against natural disasters, terrorism, etc. Normal operation routinely tests the devices for file system and data integrity and readability. The downsides are network speed and data limits, and costs. The more secure and protected, the higher the cost.

Some of us, usually the more technically inclined as well as the most naive, often manage our own backup risks with more or less rigour as the case may be.

My backup HDDs get fired up at least once every few months with surface scans every (not so) often. Even so, [usually old] files occasionally are found to be corrupted, but as you wrote, when the backups are duplicated the odds improve.

What applies to backups also applies to archives, but is more complex (sinister?) since one expects to save something, and retrieve it in future, no worries in between. @meltam’s point is well made.


#6

Out of curiosity I dug out a few old IDE hard drives I had stored in my big uninsulated Colorbond shed for the last 11 years to test. The shed temp gets into the high 50’s during summer and condensing temps in the winter. All three showed no signs of data loss. Also stored in there was a big box of audio tapes, all over 30 years old. I played a number of them on a high end cassette player I have and they still sound pretty good.

I know my sample size of three is insignificant but their storage conditions were woeful and may indicate a decline in reliability as data density increases and price decreases with these items. The audio tapes go to show how reliable the old magnetic tape systems can be and reinforces the trust put in LTO (Linear Tape-Open) cartridges for archiving.

LTO cartridges are very cheap for the data density but the tape drive can run into $$$$. It is probably cost effective to employ a transferring service but without the machine in your possession there is no way to verify the cartridges you have for archiving actually contain the data you expect on them.


#7

One of the false economies - staying with old reliable hardware. Upgrading every couple of years keeps one within the ‘support zone’ so to speak, at least with tape tech. Stick on an old one for too long, and of course you want to upgrade to the latest - yet the old won’t ‘read’ for restores. I love it when the beancounter types say “but can’t we just …?” - A few times in my career it has been, no, you can’t - those 600 tapes you say you need to be able to read for the next two years have a couple of options - retention of unsupported hardware often one, re-dumping of a lot of tapes another. It’s really hard not to grin when the beancounters start to ‘get it’ … of course they have forgotten it all by the next sensible hardware refresh time … blame the technology is the cry of those bearing calculators …


#8

Out of curiosity @meltam, are these drives portable types or SATA types used in a caddy?

I only use 3.5" drives mounted in USB 3.x caddies with separate power supply for speed and reliability. The portable units are much slower at transferring and seem less reliable in the long term.


#9

They were 3.5” HDD periodicly placed into a caddy to back up, & then removed.

I have quite a few HDDs I rotate through so I don’t overwrite anything for at least a year. Depending on what of our data I am backing up, sometimes it’s a matter of creating a new folder to backup, or it could be a reformat & fill the disc.

I find out the HDD is defective when it’s turn comes up, and it is kaput.

I have taken them apart and couldn’t see any obvious problems. I have even tried swapping the PCBs between identical spec drives. Nothing worked.

One or two OK, but four in under a year? That’s a lot of money down the drain.


#10

Its worth pointing out that solid state technology - USB sticks, SSD and the like - should NEVER be used for archival purposes, as these lose data if not plugged into a power source for extended periods. This has caused havoc in some long running court cases where such devices have been seized, examined, but not copied, and when used as evidence have proven unreliable.


#11

A good point, additionally an archive not tested ‘recently’ I feel can be assumed to be lost data until it is again tested, where ‘recently’ is measured in months not years. This covers all forms of archive - even with secure climate controlled storage and multiple disparate media across physical geographic separation … paranoid archivists ‘re-archive’ regularly, I’ve seen it done annually or bi-annually on a large scale. It’s amazing how often even routine backups fail to deliver to expectations when tested, over weeks and months, not years - more often due to ‘pilot error’ but worryingly often enough due to media integrity/etc. Expecting to put any media down blindly for 10 years or so/more is an act of pure optimism :slight_smile: test test test, and have more than one copy …


#12

This is a huge problem due to manufacturers constantly upgrading and improving technology. Good for them but bad for us!
I have also experienced all those issues starting with the old VHS tapes transferred to DVD’s and also new video recorders not being able to read the DVD’s. Perhaps saving the videos on computers and laptops will be easier?


#13

There is a rule of backups in IT:

3 backups
2 different formats
1 offsite

That is, if you want to make sure it’s going to be there when you need it you need to have multiple backups - including one offsite (e.g. using cloud storage). You will find that cloud storage providers deliver much more redundancy and security than you could ever hope to have on your own. In particular, any hard disk you buy today will be running its own error correction, as the density of data is such that they just cannot get it right every write!

My understanding of DVDs is that they can be extremely stable if they are kept out of the sun (and away from water, and not fed after midnight, of course). I am somewhat surprised that yours failed so quickly. Spinning hard drives can fail, but remain the most reliable local storage option other than magnetic tape (yes, it’s still around). SSDs should not fail quickly, because they do not have the same design issues as RAM - but will still leak electrons over enough time.

Finally, if you want to try to recover data I keep hearing about Spinrite on a security podcast to which I subscribe (Security Now, with the program’s author). It gets some good feedback, but I have never felt the need to buy it - yet. If you do visit the site, you may even look up SQRL - a possible password replacement, if it’s ever finished.


#14

I do quite a lot of transfers from VHS and DVDs and find that quality differences can be quite large. I have transferred some VHS tapes of the same age - some pretty well perfect and others filled with static. I think we have to consider that there are different quality tapes, different quality players as well as different storage quality. I recommend that people transfer out of the old technologies as soon as possible rather than leaving old tapes and disks in storage until it is too late. I certainly agree with having multiple copies of priceless family videos. The “off-site” copy needs to be in a geographically different location whether it be “in the cloud” or another town. Whenever I see floods or bushfires on TV I wonder how many family movies and photos have been lost.