Benjamin Eckstein

Recently, several bloggers I follow have posted about different storage workflows and solutions, so I thought I would add my two cents.  Hopefully this will help people tasked with storing a large amount of data.  (I write this knowing full well that storage solutions are a HOT topic for many who feel strongly that THEIR solution is the best and other solutions are DEAD WRONG, so hopefully I will stay out of the line of fire.)

I bought my first tapeless camera in January 2008 (my Sony EX-1) which is when I first created my storage system.  I had been freelance for one year prior, and was shooting almost all tape-based cameras during that time.  Interestingly, when I (and many others) began using tapeless cameras a lot of naysayers talked about the security issues of storing all this data, as opposed to the “fool-proof” solution that tape-based cameras offered.  I quickly realized that, at least in my case, it was quite the opposite solution.  In my years of shooting on tape-based cameras (as a freelancer and the company I worked for prior), very rarely were these tapes cloned, or backed up.  Sure they got digitized at the time of editing, but what if we didn’t edit for a while and something happened to the tapes.  I’ve certainly had decks eat tapes before and that could be disaster.  But with the tapeless solutions, backing up is an easy part of the process.

When I bought my EX-1, I also upgraded my computer to a new MacPro, so I needed to come up with a storage system for editing, as well as archive.  I briefly looked into some of the bigger RAID systems (things like the Drobo, which I am not even sure were around then) but most were very pricey or slow.  Also while a lot of these systems offered redundancy, by storing data to separate drives, they did not allow for separating these drives easily for offsite storage (which is high priority to me).  It’s great to have all your data on 2 or 3 different “things” but if they are in the same place, you are not protected against theft, fire, or water damage.

I did make the decision on basing my system around hard drives.  There exist a lot of naysayers to this approach, especially, because of the risks that exist with hard drive failure, but they are cheap and fast.  I knew that I needed my system to be something easy to implement, or else I just was not going to keep up with it.  (While systems like data discs and tape drives have lower risk they take a long time to manage.)

So here is what I came up with:

For archive I decided on building external drives by purchasing internal hard drives and external cases.  My reason for this was that the warranty on most of the internal drives is 3-5 years, whereas most of the all-in-one external systems had a 1-year warranty on the drive.  Initially I was buying 500 GB drives, but have gone to 750GB, 1TB, and now 2TB drives.  I always buy 2 separate brands of drives (typically Hitachi, Seagate, or Samsung) and create a storage pair with one of each brand.  This helps to protect against a (unlikely) bad run of drives in the manufacturing.  I store these in inexpensive external cases by Eagle Tech (which I get for about $22 from NewEgg).  They have USB 2.0 and eSATA connections.  I just bought an eSATA card so that will help with transfer speeds, but I didn’t care too much because these were just for archive storage.  I keep one set of drives in my office and one set at home for added security.  There are similar solutions that involve bare internal drives, with a drive docking device but I really like the ability to take my drives and be able to plug them in and use them without a separate device.

When I finish a shoot all my footage goes to a pair of these drives, and when I am ready to edit I bring the footage over to my edit system.  I edit probably 60% of what I shoot, so in cases that I am not editing, after a shoot I dump the footage to a drive for the producer or client.  I still will archive all my footage on my own drives as a backup for them.  I have had a few occasions where a client called me because they lost or screwed up their footage and I could come in and save the day with my copy.  That is a major plus that didn’t easily exist when shooting on tape.  Clients REALLY like the idea that they get that security.

For editing, I am using all 4 drive bays in my MacPro.  One is for the system drive (applications), one is a “library” drive (photos, music, and the Final Cut cache), and two are configured through the MacOS in a RAID 0 configuration (which is for speed, not redundancy).  I built the machine with 2-500GB drives to make a 1TB RAID, but am about to upgrade to create a 3TB RAID.  Since this is my “active” projects, I try to offload projects to the archive drives as they get finished, but as I have more and more active and ongoing projects I have needed a bigger system for editing.  For each of these 3 drives in the MacPro (system, library, and RAID) I have an external drive as backup.  I use Synk (by Decimus Software) as my backup system.  I imagine Time Machine would work well for many, but at the time I set this up I had heard of some issues between Time Machine and Final Cut, so I avoided it.  The other benefit of Synk is that you can backup the system drive to a create a bootable backup.  That way if your computer dies for some reason, you can plug that backup into another computer and boot off of it, so that the other computer appears as your computer.  I keep my backup drives of the computer and RAID offsite as well.

I imagine that some might find some holes in the system and there is a little bit more manual labor involved than some other all-in-one systems, but this solution works well for me, is economical and (knock on wood) secure.  Between all my drives, I have close to 20TB of storage at this point, a number that not too long ago, seemed astronomical.

If you have any questions about my storage workflow, don’t hesitate to ask.

Tags: backup, data, freelance, hard drives, HD, MacPro, RAID, storage

Posted Monday, June 28th, 2010 and filed under Blog. Both comments and pings are currently closed.