Wiping a server without rebooting and without rescue media

Hey folks,

Have you ever found yourself in a situation where you're trying to leave a hosting provider but you want to make sure that while migrating your VPS you leave nothing behind?

How sure are you that your data has been destroyed?

What do you do with a VPS though? You cannot (almost always) reboot it with a different medium.

Here's a way to do it without having to reboot your system. As you can imagine this process is very destructive so only do it when you're sure you don't want to leave anything behind.

Before we go into the process let's make note of some things:

lsblk and df -hT - we'll compare those after we're done let's also get the output of cat /proc/mounts:

lsblk:

lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 40G 0 disk 
└─sda1 8:1 0 40G 0 part /

df -hT:

df -hT
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda1 ext4 40G 677M 37G 2% /
tmpfs tmpfs 499M 0 499M 0% /dev/shm

cat /proc/mounts:

cat /proc/mounts 
rootfs / rootfs rw 0 0
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
devtmpfs /dev devtmpfs rw,relatime,size=499232k,nr_inodes=124808,mode=755 0 0
devpts /dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /dev/shm tmpfs rw,relatime 0 0
/dev/sda1 / ext4 rw,noatime,barrier=1,data=ordered 0 0
/proc/bus/usb /proc/bus/usb usbfs rw,relatime 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0

OK, OK what's the trick ? the trick is to use a ramdrive - so let's create one

1. Create a mount point for your ramdrive:

mkdir /ramdrive

2. Create the ramdrive:

mount -n -t tmpfs -o size=900M none /ramdrive

3. Change directory to your current root (/):

cd /

4. Copy everything from your `/` to `/ramdrive`:

find . -depth -xdev -print | cpio -pdv /ramdrive

5. Create a directory for the old root

mkdir -v /ramdrive/oldroot

6. Mount the virtual filesystems in the ramdrive:

mount -v --move /dev /ramdrive/dev/ 
mount -v --move /proc /ramdrive/proc/ 
mount -v --move /sys /ramdrive/sys/
# the next command is needed for Ubuntu but will fail on CentOS - this is normal
mount -v --move /run /ramdrive/run/

Special note about the --move option from the `mount` manual:

"This will cause the contents which previously appeared under olddir to be accessed under newdir. The physical location of the files is not changed."

Remember, that the directories /dev, /proc, sys and /run are pseudo filesystems - the "files" in them represent system devices.

7. Finally magic happens: move the root (/) to the new location (/ramroot)

pivot_root /ramdrive /ramdrive/oldroot

8. To make sure everything is relative to the ramdrive run /sbin/init using chroot

Note that running this command may kick you from SSH - this is normal.

For Ubuntu:

exec chroot . /sbin/init 2

For CentOS:

exec chroot . /sbin/init 3

9. Now if you SSH back in and run df you'll notice that the size of the / filesystem has changed:

df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 900M 675M 226M 75% /

See how it's exactly 900M ? that's because this is tmpfs even though it says it runs on /dev/sda1

10. Let's see what happened to `lsblk`:

lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 40G 0 disk 
└─sda1 8:1 0 40G 0 part /oldroot

11. That's an interesting difference - /dev/sda1 is no longer at / it's at /oldroot - let's try to unmount it!

umount /oldroot
umount: /oldroot: device is busy.
 (In some cases useful info about processes that use
 the device is found by lsof(8) or fuser(1))

Before you can unmount the old partition install lsof and see what processes are holding onto it:

yum -y install lsof

1. Now get a listing of processes holding onto /oldroot - to do this we use the `lsof | grep oldroot | awk '{print $1 " " $2}' | sort | uniq` command

lsof | grep oldroot | awk '{print $1 " " $2}' | sort | uniq
acpid 991
auditd 923
crond 1119
init 1
master 1105
mingetty 1132
mingetty 1134
mingetty 1136
mingetty 1138
mingetty 1140
mingetty 1142
ntpd 1026
pickup 1113
qmgr 1114
rsyslogd 945
sshd 1015
udevd 332
udevd 576

2. Okay fine - send the SIGHUP (restart) signal to all those processes:

for i in `lsof | grep oldroot | awk '{print $2}' | sort | uniq`; do echo $i; kill -SIGHUP $i; done

3. Let's check which ones still remain:

lsof | grep oldroot | awk '{print $1 " " $2}' | sort | uniq
acpid 991
auditd 923
crond 1119
init 1
master 1105
pickup 1245
qmgr 1246
rsyslogd 945
udevd 332

4. About half of them - some are straightforward enough to restart:

/etc/init.d/acpid restart
/etc/init.d/auditd restart
/etc/init.d/crond restart
/etc/init.d/rsyslog restart
/etc/init.d/acpid restart
/sbin/start_udev

5. Let's see what's left:

lsof | grep oldroot | awk '{print $1 " " $2}' | sort | uniq
init 1
master 1105
pickup 1245
qmgr 1246

6. OK - the master, pickup and qmgr are all part of postfix - these are safe to kill:

kill 1105 1245 1246

7. One more time:

lsof | grep oldroot | awk '{print $1 " " $2}' | sort | uniq
init 1

8. Now kill init - it will automatically restart :)

kill 1
lsof | grep oldroot | awk '{print $1 " " $2}' | sort | uniq | wc -l
0

done!

Proceed with the unmounting:

1. Now unmount /oldroot

umount /oldroot

2. Confirm it's gone by checking the output of `cat /proc/mounts`:

cat /proc/mounts
rootfs / rootfs rw 0 0
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
devtmpfs /dev devtmpfs rw,relatime,size=499232k,nr_inodes=124808,mode=755 0 0
devpts /dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /dev/shm tmpfs rw,relatime 0 0
/proc/bus/usb /proc/bus/usb usbfs rw,relatime 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
none / tmpfs rw,relatime,size=921600k 0 0

Yep, it's definitely gone!

Now before we wipe stuff let's see if EXT4 journaling is on:

1. To do this we use the command: dumpe2fs /dev/sda1 |egrep 'Filesystem features|journal' --color.

dumpe2fs /dev/sda1 |egrep 'Filesystem features|journal' --color
dumpe2fs 1.41.12 (17-May-2010)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Journal features: journal_incompat_revoke

2. If you see `has_journal` above, turn journaling off first:

tune2fs -O ^has_journal /dev/sda1
tune2fs 1.41.12 (17-May-2010)

3. Check again:

dumpe2fs /dev/sda1 |egrep 'Filesystem features|journal' --color
dumpe2fs 1.41.12 (17-May-2010)
Filesystem features: ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize

Yup, journaling is gone, now wipe stuff!

To shred we use.. the shred command :)

time shred -n1 -z -v /dev/sda1

Notes on the above switches:

-n1 = do only 1 pass

-z = overwrite everything with 0 (considered an additional pass)

-v = be verbose

/dev/sda1 = the hard drive we're wiping

Here's what the shred output looks like:

time shred -n1 -z -v /dev/sda1 
shred: /dev/sda1: pass 1/2 (random)...
shred: /dev/sda1: pass 1/2 (random)...615MiB/40GiB 1%
shred: /dev/sda1: pass 1/2 (random)...1.2GiB/40GiB 3%
shred: /dev/sda1: pass 1/2 (random)...1.8GiB/40GiB 4%
shred: /dev/sda1: pass 1/2 (random)...2.4GiB/40GiB 6%
shred: /dev/sda1: pass 1/2 (random)...3.0GiB/40GiB 7%
shred: /dev/sda1: pass 1/2 (random)...3.6GiB/40GiB 9%
shred: /dev/sda1: pass 1/2 (random)...4.2GiB/40GiB 10%
shred: /dev/sda1: pass 1/2 (random)...4.9GiB/40GiB 12%
shred: /dev/sda1: pass 1/2 (random)...5.5GiB/40GiB 13%
shred: /dev/sda1: pass 1/2 (random)...6.0GiB/40GiB 15%
...
...
shred: /dev/sda1: pass 2/2 (000000)...37GiB/40GiB 92%
shred: /dev/sda1: pass 2/2 (000000)...38GiB/40GiB 95%
shred: /dev/sda1: pass 2/2 (000000)...39GiB/40GiB 97%
shred: /dev/sda1: pass 2/2 (000000)...40GiB/40GiB 100%

real 11m10.869s
user 0m58.993s
sys 1m51.743s

The job is done - go ahead and reboot the system and check what happens on the console!!

Lampros - Weird Bricks

Wiping a server without rebooting and without rescue media

Before we go into the process let's make note of some things:

OK, OK what's the trick ? the trick is to use a ramdrive - so let's create one

1. Create a mount point for your ramdrive:

2. Create the ramdrive:

3. Change directory to your current root (/):

4. Copy everything from your `/` to `/ramdrive`:

5. Create a directory for the old root

6. Mount the virtual filesystems in the ramdrive:

7. Finally magic happens: move the root (/) to the new location (/ramroot)

8. To make sure everything is relative to the ramdrive run /sbin/init using chroot

9. Now if you SSH back in and run df you'll notice that the size of the / filesystem has changed:

10. Let's see what happened to `lsblk`:

11. That's an interesting difference - /dev/sda1 is no longer at / it's at /oldroot - let's try to unmount it!

Before you can unmount the old partition install lsof and see what processes are holding onto it:

1. Now get a listing of processes holding onto /oldroot - to do this we use the `lsof | grep oldroot | awk '{print $1 " " $2}' | sort | uniq` command

2. Okay fine - send the SIGHUP (restart) signal to all those processes:

3. Let's check which ones still remain:

4. About half of them - some are straightforward enough to restart:

5. Let's see what's left:

6. OK - the master, pickup and qmgr are all part of postfix - these are safe to kill:

7. One more time:

8. Now kill init - it will automatically restart :)

Proceed with the unmounting:

1. Now unmount /oldroot

2. Confirm it's gone by checking the output of `cat /proc/mounts`:

Now before we wipe stuff let's see if EXT4 journaling is on:

1. To do this we use the command: dumpe2fs /dev/sda1 |egrep 'Filesystem features|journal' --color.

2. If you see `has_journal` above, turn journaling off first:

3. Check again:

Yup, journaling is gone, now wipe stuff!

Sources:

Before we go into the process let's make note of some things:

OK, OK what's the trick ? the trick is to use a ramdrive - so let's create one

1. Create a mount point for your ramdrive:

2. Create the ramdrive:

3. Change directory to your current root (/):

4. Copy everything from your / to /ramdrive:

5. Create a directory for the old root

6. Mount the virtual filesystems in the ramdrive:

7. Finally magic happens: move the root (/) to the new location (/ramroot)

8. To make sure everything is relative to the ramdrive run /sbin/init using chroot

9. Now if you SSH back in and run df you'll notice that the size of the / filesystem has changed:

10. Let's see what happened to lsblk:

11. That's an interesting difference - /dev/sda1 is no longer at / it's at /oldroot - let's try to unmount it!

Before you can unmount the old partition install lsof and see what processes are holding onto it:

1. Now get a listing of processes holding onto /oldroot - to do this we use the lsof | grep oldroot | awk '{print $1 " " $2}' | sort | uniq command

2. Okay fine - send the SIGHUP (restart) signal to all those processes:

3. Let's check which ones still remain:

4. About half of them - some are straightforward enough to restart:

5. Let's see what's left:

6. OK - the master, pickup and qmgr are all part of postfix - these are safe to kill:

7. One more time:

8. Now kill init - it will automatically restart :)

Proceed with the unmounting:

1. Now unmount /oldroot

2. Confirm it's gone by checking the output of cat /proc/mounts:

Now before we wipe stuff let's see if EXT4 journaling is on:

1. To do this we use the command: dumpe2fs /dev/sda1 |egrep 'Filesystem features|journal' --color.

2. If you see has_journal above, turn journaling off first:

3. Check again:

Yup, journaling is gone, now wipe stuff!

Sources:

4. Copy everything from your `/` to `/ramdrive`:

10. Let's see what happened to `lsblk`:

1. Now get a listing of processes holding onto /oldroot - to do this we use the `lsof | grep oldroot | awk '{print $1 " " $2}' | sort | uniq` command

2. Confirm it's gone by checking the output of `cat /proc/mounts`:

2. If you see `has_journal` above, turn journaling off first: