One hack of a perfect (as in jack of all trades) backup solution for Ubuntu Linux (remote, flexible, instant restore, automated, reliable)

Tuesday, 14th Oct 2008 at 21:34 (administration, ideas, linux, security, ubuntu)
Tags: automation, backup, restore

This is a work in progress (and most likely will always be so)!

Here is what I have been working on and looking for to aquip myself with. I wanted to keep working without any hassle on my daily stuff just as I ever had and with changes to come. But at the same time I needed to be sure for a situation where I needed older versions of my files — would it be due to a system or hard disk brake down or a file deleted erroneously or changes need to be undone — to just have them there by no more than one command away. To say in short I wanted a time machine for my files that just works™ — also in at least 5 years time. In the case of recreation of older versions I want to be able to focus on what to restore and not how. And also with previous backups I have had e.g. corrupted archive files or unreadable part files/CDs to many times (one is even to many) or I’ve had issues because of too old of a file format (mostly proprietary formats).

Here is what I’ve been looking for feature-wise generally:

• no expenses money-wise
• robust
• using only small and freely available tools — the more system core utils the better
• version control
• snapshot system
• remote storage
• private, i.e. secure data transmission over network and reliably encrypted storage
• suitable for mobility, independent of how I’m connected
• simple yet flexible usage

for daily backups:

• automation using cron
• no need for interaction
• easy and flexible declaration of files or folders to omit from backup

and for restoring data:

• just works™ (see above)
• fast and easy look up of what versions are available at best via a GUI like Timeline with filter options
• at very best some sort of offline functionality, e.g. caching of most likely (whatever that means) required older versions

(partly) Alternative solutions I have come across on the run

• Suns’ z file system (zfs): Haven’t had enough time to get it working with Ubuntu Linux (because of license issues not packaged, only working via FUSE so far). Need’s partition setup thus lavish. Not sure about networking/mobility demands, e.g. remote snapshot location nor ease of use.
• subversion together with svk: Easy and flexible to use and automate, version control per se, distributed and offline operations (svk). Contra: Recovery relays on subversion software, i.e. no cp or mv. Basic idea is to work on a copy: checkout before you start) and have daily automated commits. Should need no interactions since I’m the only one working with my “backup projects”. See this lengthy description.
• Coda file system: distributed file system with caching. Had not enough time to try out.
• rsnapshot: Has remote feature (ssh, rsync), automation, rotation. Relies on file systems using hard links within backup folder hierarchy for “non-incremental files” and runs as root only (system wide conf file, ssh configure issue, ssh-key, …). Workaround could be to use a specific group.
• sshfs: FUSE add on to use remote directories via ssh transparently.
• croned bash backup script using tar and gzip; daily incremental and monthly save “snapshot” similar to logrotate.
• grsync: gnome GUI for rsync optimized for (incremental) backups

Update 10/2009: A few weeks ago i stumbled upon Back In Time which has astonishingly many properties of what I expect from a perfect backup solution. It basis on flyback project and TimeVault. There is a — for some people maybe a little lengthy — video on blip.tv that shows how to install and use it and how straight forward the GUI is.