2010-12-29: I’ve updated my backup strategy post here.
I’ve been making backups since I started using computers back in the early 1980s. Software at the time came on cassette tape and 5 1/4-inch floppy disks before transitioning to 3 1/2 inch floppies and CD-ROMs. When my love affair with the Mac began in 1989, I knew what it was like to have a cavernous storage system. I recall thinking to myself, “I’ll never run out of 40MB of hard drive space!” I still have old issues of MacUser magazine advertising 20MB hard drives for thousands of dollars. Now, you can get a hard drive with 50,000 times the storage capacity for a $100!
My current camera generates files that are 14MB in size. Two photos alone would almost fill up my poor Mac SE/30’s hard drive were it still spinning today. My current data set is topping 4TB these days, so my need to keep my data safe and sound is critical. I’ve personally experienced hard drive crashes before, and they are not fun to recover from without backups.
Over the years, I’ve gone through many types of backup strategies: multiple hard drives, NAS boxes, storage robots, online, and off-site storage. I’ve finally developed a comprehensive multi-pronged approach for keeping my data safe and available.
This article will be useful for those who have massive amounts of data to be backed up. If you just have a single hard drive to worry about, get a couple of external hard disks or a Time Capsule. Use Time Machine and be done with it. On the other hand, if you have to worry about backing up terabytes of data, read on!
My goal is to be able to recover from a hard drive crash with minimal to zero data loss. In addition, I want my data stored offsite in the event — knock on wood — the house burns down or is burglarized.
The 4 terabytes of data that I need backed up is organized in the following manner:
- Boot and User Data: 400GB*
- Photos: 1.5TB
- Video: 1TB
- Archived: 1.5TB
* I know that I should separate my user data from my boot volume. One thing at a time!
Over the years, I’ve assembled a smorgasbord of hardware and software to handle backups. Here’s what I’m currently using:
- Infrant ReadyNAS 600 with 4x750GB drives configured in RAID 5
- Drobo (USB) with 2x750GB and 1x300GB drives
- WiebeTech RTX400-SV SATA enclosure with 4 drive bays
- Lots of hard drives (250GB, 750GB, 1TB, and 1.5TB varieties)
Time Machine and Infrant ReadyNAS
Three and a half years ago, I bought the ReadyNAS because I had run out of space on my 250GB Photos hard drive. Initially, the Infrant held 4x250GB drives. In a RAID 5 configuration, it offered me about 700GB of protected disk space. All of my RAW photos went onto the Infrant, and I was even using it as my primary working volume. Sadly, even though it sports Gigabit Ethernet, the Infrant’s performance was nowhere near that of using an external or internal hard drive.
Today, the Infrant is loaded with 4x750GB drives and currently functions as a Time Machine destination for my Mac Pro and two MacBook Pro computers.
The Drobo holds completed photography projects and other infrequently accessed files. I don’t have it on all of the time because (1) it’s a little noisy and (2) it’s slow because of the USB interface. There is a faster Drobo today that offers two FireWire 800 ports. Sadly, I don’t know how much longer FireWire will be with us, seeing that Apple may be phasing out the high-speed peripheral port.
My previous computer, a PowerMac Quad G5, had but two internal drive bays. Since Rae and I started working on our film in early 2008, I knew we needed more fast storage. The first step was purchasing a WiebeTech RTX400-SV at the recommendation of Eric.
Unlike many other SATA hard enclosures, RTX-series does not require drive trays. You simply insert the hard drives like a floppy disk into the RTX400. The Drobo has this feature too, but the RTX400-SV uses port-multipled eSata to transfer files at internal hard drive speeds. In other words, it’s really fast!
The second step was to replace the Quad G5 with a Mac Pro. With four drive bays, the new Mac could hold my Boot, Photos, Video, and Scratch volumes. With my RAW photos now on a internal drive, instead of the ReadyNAS, working Aperture has become much faster.
The final piece of my backup strategy is software. I use SuperDuper from Shirt Pocket Software to create clones of my hard drives. The application performs incremental backups, which speeds up the cloning process tremendously. You can also schedule backups to occur automatically throughout the week.
Another similar product to SuperDuper is Carbon Copy Cloner from Bombich Software.
Here’s what happens backup-wise on a hourly, daily, weekly, and monthly basis:
- Hourly/Daily backup to Time Machine instance on Infrant ReadyNAS RAID 5 (1.TB available): Mac Pro and 2 MacBook Pros
- Weekly backup to hard drives: Boot, Photos, Video, Laptops
- Monthly rotation of weekly backups with offsite backups: Boot, Photos, Volume
- Offsite backup: Archived
Every morning, the Mac Pro wakes up and starts a SuperDuper cloning operation onto the Infrant. The initial backup for each volume was an overnight process, but each incremental backup takes between 20 minutes to an hour.
Periodically throughout the week, I fire up the WiebeTech RTX400, which holds clones of the drives in my Mac Pro, and I use SuperDuper to create an exact clone of these drives. Every month, I rotate these drives with the ones stored offsite.
My current strategy has one weakness. Because only the boot volume is being protected with Time Machine, the Photos and Video hard drives are at risk of losing more than a day’s worth of data in the event of a hard drive crash. I noted earlier that I have four separate drives in my Mac Pro acting as four separate volumes. Originally, I had configured the four drives in a RAID 0+1 configuration, following the directions on MacProJournal. I first attempted to partition and configure the four drives into a RAID 1+0 configuration, but despite hours of wrangling with Disk Utility and the command-line interface, I was unsuccessful at striping a set of mirrored drives.
With this setup, I was protected against a single drive failure in my Mac Pro. The downside was that four 1TB drives arranged in this configuration yielded only 2TB for my Boot, Photos, Video, and Scratch partitions. A few months ago, I began to run out of disk space, and as a result, I had to break the RAID. Since each volume was already cloned on a separate hard drive, it was easy to go back to a normal four-drive, four-volume setup in my Mac Pro.
To get both storage and redudancy, I might need to buy a MaxConnect and install 6-8 drives in the Mac Pro. With 2 TB drives now available for around $200, I could have up to 4-8TB of usable, protected space, which would be enough for me for the next one to two years?
Like death and taxes, increasing storage requirements seems to be another immutable law!
11 thoughts on “Backup Strategy”
Thanks, Adam – great review, exactly what I have been looking for. I need to take it one step further since my wife and I are in separate locations, most of the week.
One option that I haven’t looked too much into is online backup. I know people who use CrashPlan (http://www.crashplan.com/), Mozy, or other online storage sites.
I use S3 to backup our production server (filesystem and MySQL databases) for RedMonk, my music collection (the initial upload took 6+ days) and my laptop. Monthly costs are now in the ~$15 range.
The server is backed up using a Ruby rsync script, while the desktop uses JungleDisk.
Thanks for the writeup. We do something similar for on-site backup, but have been looking for a nice way to swap HDs for off-site.
What do you do with the drives that go off-site, btw? I’ve been wondering how to store them inexpensively but safely.
Oh, and we tried using Jungle Disk, but the initial backup will be done in 34 days. Our upstream isn’t terrible, but we’re working with print and occasional video.
I should buy some plastic containers for the bare hard drives like this:
But right now, I put them into an anti static bag and wrap with bubble wrap in the original Seagate hard drive box. I can fit 3-4 drives in one of those boxes.
I don’t have quite that extreme of a setup – but I have my storage drive mirrored onto another drive (nightly). Then I run carbonite to back up all the data (mainly raw photos). I don’t have carbonite back up jpgs (unless they were all that existed).
Then I sometimes create private albums in smugmug and upload just the jpgs to those as well. The images that I want to share are put in public galleries.
Going back in time like 1 year ago when i was searching for a backup software i remember that my strategy was to find a cheap software , and ways to make my files secure and safe . I remember that i tried a lot of backup software but i sticked to only one called Dmailer http://www.dmailer.com/dmailer-backup.html .
Their software is free to use and also you will be able to store online on their server for free up to 3gb as a free member ( they have larger storage packs at good prices if you need more storage )
[…] the end of the year, and I’ve been cleaning up my digital clutter and revisiting my backup strategy. Since I last wrote about the subject, my storage requirements have grown. My 1.5TB Photos […]
I’ve updated my backup strategy in a new post here.
[…] back about my childhood in the eighties, several things stand out, including my introduction to Apple computers, Transformers and G.I. Joe. Growing up, I had a subscription to the monthly comic book and many […]