Booting Linux from a USB Flash Drive

Back to Articles

I like my computers to be as quiet as possible: if a computer is emitting very little sound, that makes it easier to live with. The noise a computer makes is especially important when it's in a home theatre situation, or acting as a media server; if a HTPC is itself throwing out lots of noise, that detracts from the sound of the movie being played.

The ideal case, of course, is for there to be no moving parts: things that move or spin inevitably have friction, and that causes noise. I've tried to eliminate everything that spins from my setup: I have a CPU that doesn't need a fan on the heatsink, and a fanless power supply. However, there's one thing left that is spinning, and that's the hard disk which hosts the Linux installation. Dropping that would make my system truly silent, so I started looking into how that could be done.

The only viable choice for a non-spinning medium to host the operating system is a USB Flash drive: most motherboard BIOSes have the ability to boot from a Flash device, and there's no other medium which is easily available in the sizes required. So the choice was obvious: copy the operating system from hard disk to Flash, and boot it from there.

Unfortunately, it's not quite that easy. In order for the BIOS to understand the Flash disk and boot from it, the disk must be formatted in a very simple format: specifically, good old FAT32. It's quite unacceptable for a Linux root filesystem to be based on FAT32, so a certain process has to be run through:

• Compress the root filesystem down into a disk image
• Boot the Linux kernel from a FAT-formatted Flash disk
• Get the kernel to mount the compressed image
• Continue booting from that image

What you'll need

Each step will require its own tools to get the job done, and I'll be covering the details behind each of these tools when they're used. In the meantime, here's a short list of everything that will be employed:

SquashFS tools:

We'll be using SquashFS to compress the root filesystem into an image, and the SquashFS make-filesystem tool will be needed for that.

SysLinux:

Getting Linux to boot from a FAT-formatted disk is no mean feat, unless you use SysLinux; it makes everything so much easier.

BusyBox:

In order for Linux to boot the filesystem image, it'll need to mount, cd and pivot_root, among other things. BusyBox provides all these tools in one go, as I'll explain in more detail later.

Stage 1a: Copying the existing root

The first step is to build the disk image that will eventually act as the root filesystem. This will most likely come from a system that's already running: in my case, I'll be using the media server, which runs on Gentoo. We can't compress / as-is, since other filesystems are mounted into it, so we have to remove those mounts. A simple way of doing this is to remount / somewhere else, by binding it with mount:

# mkdir /root/bindmount
# mount -o bind / /root/bindmount

The new bindmounted root is a plain representation of what's on the disk, with no additional mounts: that means there should be no files in /proc or /sys, and the bare minimum in /dev. If there are any files in these three directories, you can safely get rid of them:

# cd /root/bindmount
# rm -rf proc/* dev/* sys/*
# cp -a /dev/console /dev/null /dev/initctl dev

The last line above copies over the three devices which are initally needed by the boot process: the character devices for the console and null, and the FIFO used by init. Every other device is filled in by the kernel when udev automatically populates /dev.

We're still operating on the root filesystem itself; it's now safe to make a copy, to which we can make any further changes:

# mkdir /root/fscopy
# cp -av /root/bindmount/* /root/fscopy

Once the copy is complete, any files in the copy's tmp and var/tmp directories can safely be cleaned out, since they're temporaries and won't be needed.

Stage 1b: Compressing the existing root

In an ideal world, you'd be able to take the filesystem copy and dump it to Flash, from which it would run. Unfortunately, there are two reasons as to why it's not that simple:

FAT32:

As I mentioned earlier, the filesystem employed by Flash drives is FAT, which isn't conducive to holding a Linux root filesystem. No problem; we simply hold an uncompressed disk image on the drive and read/write to that. That causes us to run into a second problem;

Write Limits:

Flash memory has a limit on how many times it can be written to before bits start sticking and data gets corrupted. Sure, it's a figure with six zeros, and hard drives have a similar limit, but I don't want this setup to work for a year or two; I'd like it to run for many years yet. Therefore, we need a way to store the filesystem that won't involve writing changes to Flash.

Fortunately, people more clever than I have devised a way of doing this: keep a compressed image on the Flash as a read-only starting point for the filesystem, and hold the changes in a RAM drive. I'll be setting up the RAM drive later, but the disk image can be done right now, using SquashFS.

There are two redeeming factors to SquashFS. Firstly, it has a hugely high compression ratio; the 2GB filesystem on my media server compresses down to about 500MB, which makes it ideal for the smaller USB Flash drives. Secondly, it's read in blocks (64k blocks by default), and the kernel will only cache a few blocks at a time; as a result, SquashFS doesn't take up much memory at all when it's running.

Setting up the image is a simple matter of calling the compressor:

# cd /root/fscopy
# mksquashfs * ../filesystem.squash

Stage 2a: Patching the kernel

Now, if you're lucky, you'll already have kernel support for SquashFS, and for UnionFS which we'll be using later. Those of you running Ubuntu Feisty, for example, will already have the modules, and don't need to do anything further. You can find out if you're one of the lucky people quite easily:

# modprobe squashfs
# modprobe unionfs
# mount -o loop /root/filesystem.squash /mnt

If these commands all run fine, and you can see your squashed filesystem in /mnt, then you don't need to do any more work on the kernel. If, however, you're a masochist like myself, you'll have to compile your own kernel containing SquashFS and UnionFS. Unfortunately, these eminently useful filesystems aren't in the default kernel package, so you'll have to patch the kernel tree:

# cd /usr/src
# wget http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.22.9.tar.bz2
# tar xjf linux-2.6.22.9.tar.bz2
# wget http://switch.dl.sourceforge.net/sourceforge/squashfs/squashfs3.2-r2.tar.gz
# tar xzf squashfs3.2-r2.tar.gz
# cd linux-2.6.22.9
# patch -p1 < ../squashfs3.2-r2/kernel-patches/linux-2.6.20/squashfs3.2-patch
# wget -O- http://download.filesystems.org/unionfs/unionfs-2.1/unionfs-2.1.6_for_2.6.22.9.diff.gz | gunzip | patch -p1

Of course, these packages are current as of October '07, and will change with time. At the moment, the above lines will complete successfully.

Once the patches have been applied successfully, you can build your kernel as normal, including all the drivers and modules as normal, but also including the two patches:

File systems ->
  Layered filesystems ->
    <M> Union file system
  Miscellaneous filesystems ->
    <M> SquashFS 3.2 - Squashed file system support

Stage 2b: Booting the kernel

Whether you've built your own kernel or using the one shipped with your distribution, the next step in the process is to get it booting. Since the USB Flash disk has a filesystem, it should be a simple matter of copying the kernel file over and applying some magic; that magic comes in the form of SysLinux, a small package which boots a kernel from a variety of situations. In order to use SysLinux, it first has to be installed; on Gentoo Linux, the installation would run as follows:

# emerge syslinux

SysLinux provides a boot system from which the Linux kernel can be started; it does this by adding a bootsector and system file to the FAT partition on which it's installed. Performing this installation couldn't be simpler:

# syslinux /dev/sda1

When SysLinux starts booting, it will look to a configuration file called syslinux.cfg, in order to find out what to do. You can make the configuration file complex, with multiple kernels and various options between which you can select at boot-time; in this case, we need the simplest possible configuration:

default kernel.img

This configuration will force SysLinux to look for a Linux kernel file, called kernel.img, and load it. Now all that's required is to copy the kernel over: depending on which route you took above, the kernel will either be inside the kernel source tree, or sitting in /boot. I compiled my kernel, so I ran the following:

# mount /dev/sda1 /mnt/flash
# cp /usr/src/linux-2.6.22.9/arch/i386/boot/bzImage /mnt/flash/kernel.img

Now you'll be able to boot Linux from the Flash drive. Unfortunately, you won't get very far before you hit a kernel panic; no root device was specified, so the boot falls over. To allow the rest of the operating system to boot, a temporary root filesystem is required: that's what I'll build next.

Stage 3a: Building an initial root

Now, I could just use the SquashFS image as the root disk, and let it be. There are problems with that, though: namely that the SquashFS image is read-only, and we'll need a system that can accommodate changes to stuff like the log files. Fortunately, there is a way to allow this, which is to overlay a temporary file system on the SquashFS image, and write changes to that.

In order to set this up, we need an intermediate step between the kernel and the filesystem image, which can perform the overlay and then "pivot" over to the actual root image. The kernel has a feature which allows it to do this: the initrd, or initial root disk, which can be loaded into RAM. The initrd can contain anything you like, as long as it's relatively small: as the whole disk image is loaded into RAM, it has to leave room for the root filesystem proper.

Building the initial root disk is a less complicated affair than putting together the SquashFS image, and it starts by allocating some space for the image. I've used 8MB, since it's ample room for everything we'll need:

# dd if=/dev/zero of=/root/initrd bs=1M count=8
# mke2fs /root/initrd
# mount -o loop /root/initrd /mnt/initrd

Once the initrd has been filled in with a filesystem, and mounted somewhere, we can add stuff to it. The first things that are required are a basic directory structure, and a few device nodes:

# cd /mnt/initrd
# mkdir -p bin dev etc lib/modules mnt proc sbin tmp usr/bin usr/sbin var/lib
# for a in {0,1,2,3,4,5,6,7}; do mkdir mnt/$a; done
# for a in {tty*,console,null}; do cp `find /dev -maxdepth 1 -name "$a" -type c` dev; done
# for a in {hd*,sd*,fd*}; do cp `find /dev -maxdepth 1 -name "$a" -type b` dev; done

Stage 3b: Filling in the initrd

Once the initrd has a basic structure, it needs some executable utilities in order to do anything: commands like sh, mount and cp all have to be provided. Fortunately, there's a nifty package called BusyBox which compiles all the utilities one could possibly need into one binary file. The configuration process for BusyBox is something I won't be covering in great detail, but it's very similar to the Linux kernel configuration: a system of menus is provided, from which selections can be made. Be aware that you'll have to compile BusyBox as a statically-linked binary, otherwise the initrd will require not just the executable, but the libraries on which it depends.

# wget http://busybox.net/downloads/busybox-1.7.2.tar.bz2
# tar xjf busybox-1.7.2.tar.bz2
# cd busybox-1.7.2
# make menuconfig
# make

I've provided a copy of the initrd below, for those who are less willing to run through the above process. It contains the basic directory structure, along with the device nodes and a statically compiled copy of BusyBox:

http://oopsilon.com/software/linux-initrd.gz [1.1MB]

What's missing from the initrd is a copy of the modules associated with the kernel. You can retrieve these from /lib/modules and simply copy them over to the initrd; in the example below, I'm copying the modules from the 2.6.22.9 kernel I compiled earlier:

# cp -a /lib/modules/linux-2.6.22.9 /mnt/initrd/lib/modules

Stage 3c: Mounting the real root

The final step in the initrd is to give it a purpose: at present, it's a collection of binaries and device nodes with nothing to do. We have an initial root disk and a kernel, but no way to link the two: the kernel should load and boot the initrd. This is done by giving parameters to the kernel when it boots, and that's done from the SysLinux configuration:

default kernel.img initrd=initrd.gz root=/dev/ram0 ramdisk_size=8192 rw init=/linuxrc

As can be seen in this configuration, the kernel is told to run a file called linuxrc on the initrd, as the initialisation script. I've provided a simple linuxrc on the initrd image linked above; all it does is load a shell:

#!/bin/msh
mount -t proc proc /proc
clear
exec /bin/msh

You may have guessed that this file can contain just about anything, as long as it uses commands supplied by BusyBox. We can mount the SquashFS root filesystem, overlay a temporary RAM disk on top and start up the new root, with the following script:

#!/bin/msh
echo Initial root disk loaded. Proceeding.

# Mount the proc filesystem, and the Flash disk
mount -t proc proc /proc
mount /dev/sda1 /mnt/0

# Find the SquashFS image on the Flash disk, and mount it
mount /mnt/0/newroot.sfs /mnt/1

# Mount a temporary filesystem, to use as the overlay
mount -t tmpfs -o size=100M tmpfs /mnt/2

# Perform the overlay with UnionFS, with tmpfs as read/write
# and the SquashFS as read-only
mount -t unionfs -o dirs=/mnt/2=rw:/mnt/1=ro /mnt/1 /mnt/3

# Pivot to the new root
cd /mnt/3
mkdir initrd
pivot_root . initrd

# Enter the new root, and run init
exec chroot . /sbin/init </dev/console >/dev/console 2>&1

You may have to change the device reference for the Flash disk, depending on the kernel you use: if you boot the initrd with the simple shell-exec script I provided above, and check the output of dmesg, you should be able to see where the kernel has loaded from.

Stage 4: Finalising and testing

With the new linuxrc, the initrd is complete, and can now be unmounted and compressed:

# umount /mnt/initrd
# gzip -9 /root/initrd
# cp /root/initrd.gz /mnt/flash
# umount /mnt/flash

And that should be that. Throw the Flash disk into a spare computer, and watch it boot: it should look just like your hard disk's boot process. If it doesn't, the cause may be one of a few things:

Kernel panic: No root or no init

Remember that in order to use initial root disks, the support must be compiled in (not just built as a module) into the kernel; furthermore, the options must be specified on the SysLinux configuration line.

The initrd shows a message, but nothing else happens

Ensure that the correct device name is being used by linuxrc for the Flash disk; if the script can't find the Flash disk, things will fall over.

init fails to read /dev/initctl

The SYSV init script, as used by most Linux distributions, uses a FIFO called /dev/initctl to communicate and change runlevels; if this node doesn't exist on the SquashFS root, init will fail.

If you have any more obscure errors, feel free to get in touch with either myself or your local Linux support channel; also, please let me know if you manage to get this setup working. The procedure above was quite smooth for me, but in the eternal clause of the technical tutorial, your mileage may vary.

Copyright Imran Nazar <tf@oopsilon.com>, 2007