Tuesday, September 15, 2009

Photo Archiving Pt 1 - Learn along with me

There is a word that can make almost any photographer cringe... "archiving" or alternatively "backups". It just seems like one big Monday, lots of tedious repeditive work thats boring. Well all it takes is one harddrive crash and you'll realize just how important it all is. I've been lucky so far (knock on wood) in that I haven't had any HDDs fail but a few friends have recently experienced it and that was enough to light a fire under my butt.

This post may turn out to be a series, at least thats my current intent. As I go about backing up and archiving I'll share with you what I've learned in hopes that it saves you time. Everyone has different needs so what I do may not work for you.

First things first, what types of backups are available.

1) CD's and DVD's
2) Harddrives (HDD)
3) Online

I have over 400GB of photos and now that I'm slinging a 5DMKII each trigger press means a 30MB+ RAW file. CD's and DVD's aren't going to cut it except for maybe the finished edits from each shoot. Harddrives are getting cheaper by the minute, a 1TB drive is around $100 right now, this will definitely be one of the options I go with. Online backups, this is a viable option but it depends how much you shoot and how much you're willing to spend on storage space. Another downside is that its one of the slowest of the 3 methods since most service providers limit upload speeds.

Conclusion: A nice mix of all 3.
After I'm done editing a shoot I'll burn DVD copies of the top pics as well as upload them to my flickr pro account (full size of course). I'll use HDDs for the main backup possibly in a RAID 1 configuration or multiple external HDDs that all contain the same data incase one fails. We'll go more indepth into this later.

Staying Organized
This is the most important aspect of archiving, I'm no saint when it comes to this but I'm turning over a new leaf ;) Last week I had my photos scattered across 5 harddrives, some internal and some external, none in what any sane person could all an organized fashion. I've spent the last 3 or 4 days, a few hours at a time, copying them all to a 1TB external drive.

First I created folders on the 1TB drive to classify the photos in the root of the drive


From there I can make more specific folders
>Family Portraits



These are just examples, I may change or add or remove folders and sub-folders as I see fit. The point is there is rhyme and reason to the folder layout. Keep the sub-folders fairly generalized or else you'll just end up with another maze of folders to navigate through.

Importing your photos properly
Taking a little extra time when you're dumping your memory card will save you lots of time later. If you're not already using some form of photo management software its time that you started. I highly recommend Adobe Lightroom 2 but another (and free) alternative is Google's Picasa. Both will catalogue your photos and give you options when importing. Both programs have the ability to detect when you plug in a memory card and will ask you if you want to import. Choose the appropriate folder to import to and name the folder with a descriptive title like "Crooked trees with Sarah", then be sure to add keywords that will help you search out the photo later. For example my the previously mentioned import was a modelling portfolio shoot I did out at the crooked trees so I keyworded it as follows: people, modelling, fashion, portrait, female, girl, location shoot, crooked trees, trees, 40D, autumn, fall. With these tags I should easily be able to search my catalogue and find them quickly by only entering one or two of those keywords in my search. Another option during importing is to replace the generic "Image_001.jpg" filenames with something more descriptive. Again in this case I might use "Crooked Trees Sarah" as my prefix so that my files come out Crooked Trees Sara 001.jpg, Crooked Trees Sarah 002.jpg etc.

Off-site Backups
Another thing I should note is the importance of an off-site backups. You never know when disaster could strike and either your house gets broken into or you have a house fire or flood etc. This is where backup option #3 the online backup is handy but most of the time you don't have all of your photos online. When you're burning off a DVD take another 10 minutes and burn a second one off to store at your parents place or in a safety deposit box at the bank. The same can be done with a second external HDD, the only issue with this is having to go pick it up periodically and update the new files on it. It all comes down to that cliche of "don't put all your eggs in one basket".

A quick note on HDDs and RAID
I'm ashamed to admit, being a tech and all, that I didn't really keep up to date with what exactly RAID is. RAID stands for Redundant Array of Inexpensive Discs

There are various RAID configurations, here is an excerpt from Wikipedia
  • RAID 0 (striped disks) distributes data across several disks in a way that gives improved speed at any given instant. If one disk fails, however, all of the data on the array will be lost, as there is neither parity nor mirroring.
  • RAID 1 mirrors the contents of the disks, making a form of 1:1 ratio realtime backup. The contents of each disk in the array are identical to that of every other disk in the array.
  • RAID 5 (striped disks with parity) combines three or more disks in a way that protects data against loss of any one disk. The storage capacity of the array is reduced by one disk.
  • RAID 6 (striped disks with dual parity) combines four or more disks in a way that protects data against loss of any two disks.
  • RAID 10 (or 1+0) uses both striping and mirroring. "01" or "0+1" is sometimes distinguished from "10" or "1+0": a striped set of mirrored subsets and a mirrored set of striped subsets are both valid, but distinct, configurations.
The most common ones you'll probably see is RAID 0 & 1 and possibly 5. RAID 0 is useless to us and I'd actually recommend everyone stay away from it period, basically it makes 2 HDDs into one big one, the problem is that if either HDD fails ALL the data is lost!

For us either RAID 1 or RAID 5 is what we need, with RAID 1 you essentially have two mirrored HDD's that can be used to repair or recover the other HDD in the event that it fails or becomes corrupted. With RAID 5 its similar to RAID 1 except it uses 3 or more HDD's and protects against any 1 failing.

Redundancy costs money, for the HDDs it'll cost you double since you need twice the space. To have 1TB with a RAID 1 setup you'll need to buy two 1TB drives. You'll also need special hardware to manage the RAID setup, either a RAID controller in your PC or an external HDD enclosure that has a RAID controller built in. These are not cheap. As I write this the cost to put together an external HDD enclosure with two 1TB HDDs configured as RAID 1 its about $400CAN.

Based on the price tag the RAID solution may not be for you, this solution is more for the advanced hobbyist and professionals who earn money from their photography and can't afford to lose any photos.

In closing...
As I researched more and more about archiving in the digital era the more I realized that no one method is foolproof or permanent. With film the negatives can last for years, they may yellow or have colour shift but the image integrity is fairly good. With digital, when failure or corruption does occur, the results are normally catastrophic. CD's and DVD's degrade over time, the initial promises of 100years is definitely rubbish, HDD's are mechanical and fail over time and the platters lose their magnetism. Online backups can suffer server failures, though specialized online backup sites would probably have redundancy to protect from this. In the end you may want to make hardcopy prints of your most treasured photos as they will probably last the longest ;)

Hopefully this post wasn't too boring, I know it is a little lengthy and there were no pictures ;)

I'll post more as I go further with my archiving and learn more.


Jay said...

Thanks for the fire under my arse. I started doing my periodic backup after the first sentence. So simple, so quick, just forget how important it is.

I lost many files of our puppy some time ago. Somehow they got deleted; It wasn't due to a HDD crash. Undelete utilities saved about 80% of them, but man, was that a lesser stomach in your throat/nearly got in a car accident/someone close to you died feeling.

jphphotography said...

Yup it definitely is important, the funny thing is those of us that preach this the loudest are often the ones who don't follow it ourselves ;) I'll most likely be making another post on this soon, I've come across some handy programs and some interesting gear as well since I wrote this post.

Thanks for commenting!