Pixiq

One of the major topics in any digital workflow is how to store, archive and catalog your digital images. This is a complex topic and in this short blog post I definitely will not cover everything. I will speak to my experiences and what I have found best suits my methods and work. And while this blog post isn’t directly related to Lightroom we’ll come around to how it fits into my archiving and cataloging set up after we talk about hard drives and storage options.

First and foremost storing you images depends on what type of photography you do and how you need to access your images. If you are a wedding and portrait photographer it is unlikely that you’ll need instant access to older images as they were shot for a specific direct client. If you are an editorial or commercial photographer then it is likely you’ll need access to your entire image archive so that you can pull images for submissions. The first case is much easier to deal with while the second is a little harder depending on the size of your image library. 

Storing your Images

There are lots of options out there for storage. The least expensive method is to store images on DVD or BluRay discs but that is cumbersome and very time consuming to access files and move data to new storage devices if this is your only storage method. Hard drives are the mainstay of storage devices and whatever you choose make sure it is reliable and backed up.

I highly recommend that you keep at a minimum three copies of every image file. I myself have had to go to that third backup on one occasion to retrieve images when two 750 GB hard drives failed within a day of each other. I was very happy (serious understatement!) to have the third backup as I would have lost two years worth of digital images!

Most photographers, myself included back images up on multiple hard drives. I used to burn my third back up to DVD but I have so many DVD’s now that currently I am making the third back up to hard drives. At the moment I have over 5 TB of hard drives. I have four hard drives living in my Apple MacPro tower and three external drives on my desk. Two of the drives in the MacPro are image drives, one is a back up for my main tower’s hard drive and then the fourth drive is the working hard drive for my Apple MacPro. All of these drives are at least 750 GB, most are 1 or 1.5 TB drives. The internal drives are backed up to the external drives so I have a portable copy of my images. The third back up lives off-site at my bank and gets updated as I finish processing assignments. Using multiple hard drives is a very viable and reliable storage option. Of note, my 2nd backup here in the office is a RAID 5 array which we'll talk about here in a moment. 

One hard drive solution I would recommend is from MacGurus (www.macgurus.com). They build these fantastic Port Multiplier enclosures that will hold up to eight internal SATA hard drives. They connect with an SATA cable and upload and download data 7.5 times faster than Firewire 400. Internal hard drives in general are much less expensive than an external hard drive and are more reliable in my experience. The Port Multipliers also keep everything nice and tidy on the desktop as well as control the temperature of the hard drives. From a cost analysis this is also the cheapest and highest performance storage solution I have found.

A note about connection speed, it may or may not be important to you but I find that slow connections are painful and hence I really prefer SATA or eSATA connections – but I can work with Firewire 800 if I have to. SATA is 3.0 GB per second – pretty stinking fast, FireWire 400 is in the neighborhood of 400 MB/s and Firewire 800 is twice as fast as Firewire 400. USB 2.0 is supposed to be faster than Firewire 400 but on Macs this is rarely the case. I avoid USB anything like the plague. I do look forward to Apple's new Thunderbolt connection and hope to see many hard drive maufacturers come out with compatible drives for this fancy new option. At 10 GB per second it makes everything else look pretty week. We'll see how that plays out. At the moment there are very few computers that have Thunderbolt and even fewer Thunderbolt hard drive options. 

Getting down to the nitty-gritty, in terms of which brand of hard drive I would recommend I can’t say there are external drives that are better or worse than any others necessarily. Every brand has had failures. My personal preference are the internal Seagate Barracuda drives. I have found these to be the most reliable of any hard drive I have used. And I have yet (knock on wood) seen any of my Seagate HD’s fail on me.

And finally, the safest method of storing data (on one set up) is using a RAID 5 system. RAID stands for “Redundant Array of Independent Drives”. Instead of me going into a long diatribe on what RAID is and the different levels of RAID I implore you to check out Wikipedia’s excellent RAID and RAID 5 definitions.

RAID 5 is the most expensive storage solution we’ll cover here by a factor of no less than 2. Apple’s Xserve Raid (now discontinued) was the cream of the crop but at almost $6,000. That wasn’t exactly affordable for most photographers. The new kid on the block is the Drobo. This is a slick, easy to use RAID 5 backup solution and I would highly recommend that buyers go with the Firewire 800 or eSATA option. If you are working on an Apple MacPro you can set up an 8 TB RAID 5 system inside your tower if you opt to purchase four 2TB internal hard drives and the Apple RAID card. My favorite RAID 5 arrays are the G-TECH options. They have a fan built in and have speedy connections. There are many ways to set up a RAID system and I have only talked about a few options here.

What RAID 5 offers is a solution for storing large amount of data like 8 TB or more on one hard drive system. The big issue with RAID 5 is how do you back up a multi-terabyte RAID 5? With another RAID 5 is the answer you didn’t want to hear. Many will say that you don’t need to back up a RAID 5 system but if two or more drives fail then you have lost everything. A RAID 5 set up is not a backup in and of itself. It still needs to be backed up.

In the future we’ll have much more reliable CompactFlash hard drives but that might still be a few years away. The gist of it is choose a system that fits your budget, fulfills your needs and is easily scalable as your image archive grows.

Now, in regards to the two types of photographers I mentioned at the start of this article. For those of you that never really need to go back and look at images shot for direct clients (e.g. wedding and portrait photographers) a system of redundant hard drives will most likely suffice. For those of you that need access to your entire library of images then you have several options as laid out above that depend on your budget and the amount of storage you actually need.

Below is an article I wrote in my newsletter way back in 2006 that reveals the difficulties and importance of vigilant archiving strategies. This article was written after attending an all day ASMP Digital Workflow meeting here in Santa Fe that exhaustively covered many of the issues we face with digital photography.

Archival Stress (from MCP Winter 2006 Newsletter)

In an ASMP New Mexico Digital Workflow seminar I attended, Marc Mintz (http://themacosxperts.com) gave a doomsday presentation on how to store our digital images and it wasn’t pretty. At first I thought he was just trying to scare us, so he could make some money with his consulting business, but my own experiences and advice from other computer experts leads me to believe he wasn’t exaggerating at all. Each photographer chooses their own method of preserving their digital files, usually a combination of multiple external hard drives and DVD/CD’s. But the gist of Marc’s presentation is don’t trust anything. A few choice quotes by Marc including “It is not a matter of if your data will become corrupted, it is when” and “If you lose your files - you lose your business” woke us all up pretty quickly. The scary news is the latest research shows that all media, be it hard drives, DVD’s or CD’s, if loaded up and stored in a hermetically sealed safe for five years without any use have a 50/50 chance of failure. That’s with no use! If you use your hard drive often and it is on all the time you have a significant chance of failure or corrupted data after just six months! Failure rates for hard drives run about three to five percent and normally if a hard drive runs ok for the first six months it will be ok for quite some time. Hard drives usually fail in the first six weeks if they are faulty. But even if the hard drive is not faulty every hard drive is susceptible to data corruption. If that isn’t scary for the digital photographer I don’t know what is.

Marc recommends a minimum of three backups, usually two hard drives and on DVD. He also recommends replacing all hard drives every 18 months to two years just in case. In addition it is also wise to have “online” and “offline” backups - meaning you have one back up in your office and another somewhere else off site in case of fire or some other disaster. And oddly enough, he does not recommend RAID 5 systems because they are so big you have to have two of them to have a back up and that is not the most cost effective method of backing up files. So, if you haven’t gotten the point yet, be prepared to spend a lot of money keeping your digital images archived.

All of this may seem a little crazy. It did to me at first. After all, back in the “old days” just a few years ago we all stored our original film in filing cabinets - and nothing is as good as the original so if it all got burned up it all burned. All of the sudden with digital we now have backups just as good as the original and that has changed everything. Regardless, this is still planet earth and everything still obeys the third law of thermodynamics: everything tends toward chaos, meaning everything breaks down. And if you have ever had a hard drive fail, you know what a sickening feeling that can be if you didn’t have it backed up as you should have.

Archiving

Archiving your images is basically backing them up responsibly, including accurate metadata, having off-site storage and working with software that allows you to find your images.

How to archive your images is still a hot topic. Whether it is better to convert to Adobe’s DNG RAW format or leave your files in their native format is not yet clear. There are many arguments for converting your original RAW files to DNGs but for my part I choose not too convert my RAW files to DNG because I prefer to have the most flexibility in the future - i.e. the ability to use Nikon’s proprietary software to process the Nikon RAW file. If I converted my files to DNG then I would not be able to use Nikon’s software, which for better or worse I find gives me the best skin tones - at least for some images. Lightroom has closed that gap considerably now and there are very few images (I can’t remember the last time I actually opened Nikon Capture NX2) that I actually take into Nikon Capture NX2 these days.

The other hitch to archiving efficiently is to make sure your images have the pertinent metadata and keywording either as an XMP sidecar file or embedded into the jpeg, psd or tiff file. If you make sure to enter this information in your normal workflow and have good organizational skills then this is a non-issue. We already have powerful software tools on the Apple platform for finding any file on attached hard drives with the Spotlight tool. In addition, as we continue to accumulate and fill up hard drives being able to locate images that are on hard drives not attached to our computer or turned off will become of paramount importance. Luckily, Adobe Lightroom is an excellent tool for these instances and right now it is also my top pick for archiving and cataloging my images. 

Cataloging

Cataloging your images is a vast topic that involves Digital Asset Management (DAM) - basically how are you going to manage all of these digital files that are filling up hard drive after hard drive? A key part of any cataloging system is entering keywords and metadata into every image. That is the first BIG step. If you do have keywords entered then on a Mac you can use the Spotlight feature to find any image if you have all of the hard drives your images are stored on plugged into the computer and turned on. This is of course a very rudimentary method for finding images with metadata. As we continue to fill up multiple hard drives the need for a true cataloging system is critical to find images easily.

At the moment there are a few software options that do an excellent job of cataloging images: Adobe Lightroom, Apple Aperture and Capture One's  Media Pro. Of these, Lightroom and Aperture are the clear favorites and both work very well. I have a single Lightroom catalog with over 150,000 images in it, which is my entire catalog of raw digital images. Lightroom allows me to search all of my images in short order for any keyword, metadata or any other search term like date, camera, lens, filename, etc. In addition to having keywords and metadata entered into my images, they are also organized by file name and folder names that allow me to find just about anything relatively quickly.

If you are interested in developing a complete workflow of your own using Lightroom and perfecting your color management, I would recommend checking out my Lightroom Workflow e-book Adobe Photoshop Lightroom: A Professional Photographers Workflow. You can purchase that eBook for the low price of 24.95 on my website.

This e-book presents a complete workflow which includes my in-camera settings, how to determining the optimum white balance and exposure, color management, working with Lightroom and Photoshop, creating web galleries, Noise Ninja and much, much more. A sample table of contents is available for download on my website if you want to see exactly what is covered. The e-books is also up to date and covers the latest versions of Lightroom and Photoshop.

Adobe® Photoshop® Lightroom® 3 software helps you bring out the best in your photographs, whether you're perfecting one image, searching for ten, processing hundreds, or organizing thousands. Get your free 30-day trial at www.adobe.com/downloads