What follows is a short tale about how to recover digital photos from a broken hard disk under Linux.

When I returned from a short vacation this week, one of the hard disks reported read errors for the Reiser filesystem superblock. And while I had multiple backups of all valueable data, my digital photos were only stored on this hard disk.

To add insult to injury, I noticed that I had deliberately switched off a SMART warning some months ago: for the "offline uncorrectable sector count". I had ignored this early warning sign because further SMART offline tests returned a count of zero.

After running badblocks for a while, it looked like only two small areas at the start of the disk were damaged.

Not wanting to take any more risks, I bought another 160GB hard disk, and copied the whole disk:

dd if=/dev/hdb of=/dev/hdc bs=512 count=65536 conv=noerror,sync
dd if=/dev/hdb of=/dev/hdc bs=4096 skip=8192 conv=noerror,sync

The conv parameters ensure that read errors on /dev/hdb are ignored, and that any data missing due to these errors is replaced with null bytes. Because I hoped that there were no errors after the first 32 MB, I used a larger block size for the second run, to speed up dd.

While all blocks after the first 32 MB were indeed readable, the disk grew hot, and only delivered 2.5 MB per second. This gave me 18 hours to plan the next steps.

First, I ran reiserfsck. The superblock was gone, so I had to use the --rebuild-sb parameter. Didn't help much, so --rebuild-tree was next. After that, I saw some familiar directories and filenames, but looking at the content of those files did not turn up a single of my photos.

So, do what special agents do, and use some forensic tool for evidence recovery. Reading the list from the Knoppix Security Tools Distribution, I first tried foremost, but found that it was unusable for 2.5 MB digital photos, because its memory consumption grew exponentially with the size of searched files. The next tool was photorec, a tool specifically written "to recover pictures from digital camera memory".

As with all filesystem-agnostic tools, it cannot deal with fragmented files, but it recovered almost 90% of my photos. Its usage is pretty simple, as it has a text-based UI. The recovered files are stored in a directory named recup_dir, which is created in the current working directory.

The only problem I had with photorec was that it sometimes did not find the end marker for a file type before it had copied a few hundred MB. And it seems that it looks for that start marker in areas which it already has copied to a file, because I ended up with more data than I could store.

Telling photorec to only scan a specific area of a disk is not possible, but one can use fdisk for that. Changing the partition table of an unmounted disk which you don't plan to write to does not erase any data, except for the old partition table. So I created a couple of partitions, ran photorec on one of them, sorted out the good data, and moved on to the next partition.

Now, I have two hard disks connected to two different IDE channels, housed in fan-cooled aluminium docking bays. I use software RAID level 1 to mirror my data, and I have set up multiple smaller partitions with LVM so I don't ever have to shift through the whole disk when a few thousand bytes disappear. And of course I have included my photos in my backup process.

Setting up RAID and LVM and copying an existing Fedora Core 3 installation to the new disks was pretty easy. The only tricky part was that I use LVM for the root partition (not recommended). To make this work you just have to create a new initial RAM disk with mkinitrd, as the RAM disks from the kernel RPMs do not contain the necessary LVM stuff.

Update 20 Jan 2005: The initial RAM disk is not part of the kernel RPM, but is created when you install the kernel on your machine. So if /etc/fstab already points to an LVM root partition during a kernel upgrade, there is no need to created a custom RAM disk.