recovering deleted files in osx

Fri, Mar 13, 2015

Here’s how I stupidly deleted some important data and earned my honorary 1337 hacker badge by recovering it.

The setup

I use a macbook pro running osx, but I do my developmentand communication on linux vms. Recently I was deleting some old files on my mac (because for some reason despite the fact I have a terabyte SSD, using 250GB of data seems wasteful). I came upon a 30GB virtualbox disk image that clearly wasn’t needed anymore. The file was called CentOS.vdi and I was certain I didn’t need any centos vms. I attempted to delete the file via finder but osx wouldn’t allow me because the file was in use. “Luckily” I was able to use rm from the command line and the file was instantly gone. Problem solved.

Around this time I started looking to see how big the drive file was for my “main” linux mint vm. Couldn’t find it, but strangely the vm continued to run without any problems. It was at this moment that I realized my dev vm disk file was gone. When I had created my linux mint vm I had re-used an older centos disk file.

Luckily for me, my vm wasn’t crashing. Not being a complete *nix n00b I realized that it was quite likely that virtualbox was maintaining an open file descriptor to the now unlinked file. The inode formerly known as CentOS.vdi was no longer in the filesystem namespace but it still existed as long as the vm was running. The only question was how to recover it?

I come from a linux background so I instantly thought of the proc filesystem. Unlinked files show up in there as broken symlinks. A little known fact is that you can still open() those symlinks and get to the original file. In linux it would be very straightforward to recover the file with a shell script that looks like:

cp /prod/$pid/fd/$n ~/somewhere-safe

However OSX lacks the proc filesystem so I’m out of luck there.

Using OSX debugging tools

What can I do on OSX? I can use lsof to determine which file descriptor in the process refers to my deleted file. I can use gdb (or lldb) to freeze and alter the process. In the past I’ve used tools like pyrasite, which use gdb under the hood to make python programs execute arbitrary injected code. (python programs generally have a symbol called something like PyRun_SimpleString). However virtualbox is a c/c++ program which makes injecting code more complicated than simply calling a function with a string. However I now at least have a plan of attack:

attatch the process using lldb -p
open() a new writable file using call open("filename", flags)
Take the fd referring to my deleted file and seek to the beginning
??? call something like cat or python’s shutil.copyfileobj

Preparing the process

The first few steps look something like this:

lsof -p $pid # to find the fd we want to undelete, in my case 22
lldb -p $pid
call open("/tmp/hail-mary", 0x0002)
call lseek(22, 0, 0)

Getting data out of the process

sendfile v1

Finding code already in the process to copy data from one fd to another was quite tricky. My first attempt was to use sendfile() , which actually copies data from one fd to another. All I needed to know was the size of the file, which I could determine by calling lseek(fd, 0, SEEK_END) with the appropriate arguments. Unfortunately I hit a snag here because sendfile only works if the outgoing fd is a socket. sendfile returns ENOTSOCK.

sendfile v2

I briefly tried to use the socket/bind/connect apis plus sendfile to ship the data to a local netcat process, but my ability to call functions in lldb is pretty limited. I can call functions that take string or integer arguments, but I would have needed to look up the values of several #defined constants like AF_INET (easy) and properly allocate and initialize a c structure called sockaddr_in (seemed harder). I had now ruled out netcat and sendfile.

dlopen

At this point I was out of ideas. Then inspiration struck: I merely needed to write a simple function that did the job I needed and load it as a shared library using dlopen . The function I linked in needed to be easily callable from within lldb, so I hard coded in the arguments. I ended up using the below code.

#include<stdio.h> //printf
#include <unistd.h>

int shane_copy() {
  int output = 23; // from running call open("/tmp/hail-mary", O_RDRW)
  int input = 22;  // found using lsof
  char buf[4096];
  int n;
  int copied = 0;
  while(1) {
    n=read(input, &buf, 4096);
    if (n==0) {
      return copied;
    }
    if (n<0) {
      return n;
    }
    copied +=n;
    n=write(output, &buf, n);
  }
}

We build the above statement into a shared library like so:

clang -g -shared  -o libfoo.so foo.c

Let’s load our new library using lldb. Luckily we don’t have to use dlsym and casting. In my version of lldb shane_copy() is directly callable.

print (void*) dlopen("/path/to/libfoo.so")
call shane_copy()

shane_copy() ran for a few minutes and viola, my virtual machine’s disk had been saved!

Epilogue

Now that the heat of the moment has passed I’ve thought of a few other easier ways to save my deleted file.

dd from the vm
dlopen python as outlined here

Also, it would have been a great idea to have more frequent backups and to make sure my vm only has volatile state.

Whitane Tech.