nattering on about backups

[chhotii]

Today's excitement: yet another hard drive bites the dust? I'm doing file backup sets with Retrospect (ugh!) and I have to rotate the hard drives used. See, when a file backup set starts to fill its hard drive to near capacity, Retrospect can't trim it in any way. It just keeps writing to it until it gets drive-full errors. When this happens, rather than delete the full drive right away, thus losing the backup (!) I rotate in another drive, and put the full one in the cabinet for a while, until its backups are obsolete. These Maxtor drives-- don't like these periodic banishments to the cabinet. Just had one bite the dust, right after I got another RMA'd drive back from Seagate.

This has me all the more eager to select an alternative to Retrospect.

The beautiful thing about both Apple's Time Machine and QRecall is that they both, unlike Retrospect, elegantly and intelligently handle the situation when the destination hard drive starts to fill up. Both of these programs allow you to specify how full you want to allow the back-up hard drive to get, and, as that limit is approached, they start selectively deleting the oldest items first. So you always (theoretically, if these programs work) have as much of the newest stuff as will fit, and you never have drive-full errors, without ever having to delete anything manually. Perfect! This is why I was so excited when Time Machine came out, and when I discovered QRecall.

Each of these has other limitations, though. Time Machine is a nightmare; I've written about its issues before. I haven't tried version 10.5.5, but I'd guess that Apple still doesn't get it.

QRecall is quite slick, and far superior to Time Machine in important ways. Its log is detailed and informative. It is extremely configurable, unlike Time Machine, which is one-size-fits-nobody.

However, when I tried backing up over the network to a mounted volume on a hard drive attached to a different computer, disaster! See, if you mount a network drive in OS X, you're likely to have the drive randomly un-mounted when there is a network glitch (which we have a lot of in this facility). When this happens, QRecall's archive is left in an inconsistent and corrupted state. When I contacted the developer, he said don't try to back up to a network drive because of this problem with the current version. (Even though he had earlier recommended this approach to another user.) His recommendation was to try the beta version, which should be 99% more reliable at backups to network drives. But 99% more reliable is not reliable enough for me to implement; I had disaster after about 4 days of using QRecall, I don't want disaster after 400 days of using it either. Really, what you need is a client-server architecture, where the client sends data to the server and the server carefully places it into an archive that is local to itself. That is the architecture that Retrospect uses, and I have only ever had corrupt backup sets when the backup hardware itself (the tape drive) was failing. Periodically I get corrupt catalogs, but at least that is recoverable.

The other way to do: Let me refer to the computer with the backup hard drive attached as the server and the computer that needs to be backed up as the client. Instead of running QRecall on the client, mounting the server's hard drive, you can run QRecall on the server, mounting the client's hard drive and backing up as much data as it can see as the remote user. There are some tricky things about this approach, and the maker of QRecall doesn't recommend it. For one thing, how will the client's drive get mounted on the server? Nobody is going to manually log in to 10 other computers from the server every day. You need to write some AppleScript or some such thing that will automatically get kicked off and mount the client drive. This sounds like it will be prone to glitches, and who or what is going to monitor the success of that script? Secondly, the remote user can't read everything. You will absolutely, definitely never get bootable backups using this approach, but I can live without that. What I worry about is if some important document that someone cares about doesn't get backed up because the backup server is logging on as a user that can't see it. It seems that if the server is using, on each client, a login that is in both the "staff" and "admin" groups, by default it should be able to read all the user stuff; and whatever happens by default is probably what is going to happen, because folks here don't tend to go mucking with file owners and permissions. (Generally the machines auto-login, and the users are oblivious to the existence of "users" and "groups".) However, while that's a rule of thumb, it seems a bit to chance-y to bank on.

So, both Time Machine and QRecall really want a directly-attached hard drive to back up to. If I have 10 clients, I would need 10 backup hard drives. Here are the problems I have with doing that:
* Additional expense: The least expensive hard drive space, measured in $ per GB, is currently in about 750 GB drives. (You might find the optimal price point in the 500 GB or 1 TB drives at the moment. Whatever. It's somewhere in that order of magnitude.) I currently back up several hard drives, incrementally over months, onto drives that size. Therefore it's cheaper to buy fewer large hard drives than many small hard drives.
* Some computers are in unsecured areas, and small hard drives could easily walk. (Really I should bolt the equipment to the desks with locks, but the unsecured ones are all iMacs, too damn bulky to steal.)
* If an extra drive shows up on a user's desk, they might just treat it as extra storage. It is, but-- if they use it that way, their backup space shrinks, plus whatever they store there doesn't get backed up.

But really, the biggest problem I have: if you can only back up to an attached drive, then your backup is always too damn close. Physically close. That is, if the HVAC equipment on the floor above us starts dripping again, it will take out a computer and the backup sitting right next to it!

But this just underscores (over and over) how I should consider the off-site aspect of backups. So, let's consider off-site solutions.

BTW, I got e-mail today that Mozy Pro has a beta version for Macintosh. But they have a terrible reputation for doing restores. I think I might have found something better.

CrashPlan looks really interesting. You don't have to back up to their servers. (In fact, the makers of CrashPlan claim that they don't want you to back up to their servers, unless you really don't have any friends to back up to. But if that's true, why are their servers so inexpensive? $0.10/GB seems like a really good deal.) You can (for example) buy your boss a big hard drive, put it in his study at home, and supposedly, with enough tweaking firewalls and so on, back up to that drive from the office. Not only is this (probably) cheaper, and under in-house control, but it overcomes a major problem with off-site-- that it's so damn slow to do the initial full backup. You can back up everything to the new hard drive at the office, at much faster LAN speeds, then send it home with the boss to continue to do much smaller incrementals forever afterwards.

My big question about CrashPlan at this point: What does CrashPlan do when the space on the remote server starts to fill up? They might say it shouldn't, because you limit the number of old versions of files that are kept, and how long deleted versions are kept. But you could easily get the space requirements calculations wrong or out-of-date. Does it delete the oldest stuff, as it should? Or does it freeze up and start spewing lossage, as Retrospect does? I've been searching the Internet for half the day without finding an answer. You would think it would do something smart. But, then again, you would think that Retrospect-- one of the more expensive, "professional" options-- wouldn't do something totally retarded, either, but there you go.