Dev/Embedded/Rootfs/BusyBox

Introduction

BusyBox is "The Swiss Army Knife of Embedded Linux".
BusyBox is a "multi-call" binary which combines tiny versions of many common UNIX utilities into a single small executable.

A simple description and a full list of embeddable commands and available options can be found here.

Most information can be found in the FAQ and all around the Busybow website, but here is a short "How-To".

Get the sources

Either use git (below) or download a version from here : Download BusyBox sources

git clone git://git.busybox.net/busybox

You can then extract a specific version if required.

Configure

BusyBox uses a menu equivalent to the Linux kernel's one which is accessible using :

make menuconfig

There's also a "defconfig" and an "allnoconfig" target.
Tune to your needs and move on to the next steps :)

Some notes and warnings :

• The cross-compiling toolchain prefix can be set under "Settings"
• Note that some configuration options will greatly impact your system architecture, and some may need specific development libraries to be installed.
• Keep in mind that some tools may depend on specific kernel features (and includes) and that you may need to install your own kernel hearders and reference them instead of those comming with the libc.
  This is the case for the "tc" network applet which is included in the "defconfig".
• Adding headers or additionnal external libraries is done in the "Settings" part of the configuration, under "Additional LDFLAGS" and "Additional LDLIBS".
• As of 2025-01-03 selinux has deprecated "security_context_t" and "matchpathcon()" (at least) and you'll need to update the selinux related code if you need it.

Compile

Still in the kernel path, cross-compilation is made using make with CROSS_COMPILE option set. For example for the armhf architecture with GNU libc, use

make CROSS_COMPILE=arm-linux-gnueabihf-

Most of the time, using BusyBox is a choice due to storage size constraints, and thus using uClibc instead of the GNU libc is a good idea.
Most information about using uClibc in a cross-compilation toolchain can be found on uClibc website.
You will then need to use the cross-compilation prefix of your toolchain to build BusyBox.

An the end of the compilation the build system displays the "additionnal" required libraries (of course, the libc is always required), for example, with an almost default configuration (make defconfig) :

Final link with: m resolv

In this case, you'll need to include the libm and libresolv.
With a stripped down busybox, you may well end up with :

Final link with: <none>

But not that in this case you'll still need the libc !

Anyway, there are other ways to know which libs are required in your final system.

Install BusyBox

As stated previously Busybox is a "multi-call" binary.
This means that there is a single binary called "busybox", and in order to use other commands the usual way one must create symlinks to this binary with the name of each command included in the busybox binary.
Hopefully, this task is automated by the BusyBox build system :).
Run :

make install

and you will get a directory called "_install" containing the base "bin" directories with all the links for the commands you configured.
Not that you must remove the old "_install" directory before running the new "make install".
Of course this does not make for a full system, but you already have all the binaries (or almost all depending on your needs).

Before moving on to the next part, let's make a short point on how to copy this "_install" directory somewhere usefull.
If you stick to a simple "cp" with no arguments, then you will get many full copies of the busybox binary instead of all the symbolic links, which is obviously not what we want.
We have three options to copy this "the right way" :

• Use "cp -a" (--archive) : The easiest way
• Use rsync (with the right set of options)
• Use tar (make an archive with "tar -c", and then extract somewhere else with "tar -x").

I'm using the "cp -a" solution most of the time, unless I want to keep a backup somewhere (and use the tar solution then).

Let's now move to the next parts of the base system.

Populate the base system

Busybox installation process only installs the busybox binary and the symbolic links for all the selected utilities (commands).

Your system will need some more files so you can use it as you would use any other Linux based system.
Some of these files are mandatory, while others depend on your application and the kind of system you want to build. Most of the time when you start with busybox as the heart of your system you do not plan for a multi-user desktop box, but rather aim for a small, single purpose embedded system, so you won't need too many stuff to get it running.
Usually, what you need can be split in a few categories :

• directories
• libraries
• bootup script
• configuration files
• application binaries
• application related files

directories

You will not need a full set of directories from the FHS but this can be a good place to get information if you do not know what you need.

• The default busybox installation process has created four "*bin" directories for you, which is a good starting point (though you can merge them to get two if you want to follow the usermerge way, or even go further and get a single one bin directory). : /bin - /sbin - /usr/bin - /usr/sbin
• Then you will usually want somewhere to mount proc, sysfs ans devtmpfs : /proc - /sys - /dev
• You should also stick to the standard and place your text configuration under /etc
• Then you will need a place for your libraries. This can be a single place, or many, it's up to you (it's actually a real mess on some distributions, with multiarch support, 32 and 64 bits support, usrmerge, and maybe other funny stuff getting in the way). Maybe you should consider keeping it as simple as possible (refer to the next section to make your choices about this part). On a Devuan or Debian distribution you usually get those three at least : /lib - /usr/lib - /usr/lib/*arch-triplet* - ...
• A place to put "shared ressources" (architecture-independent data, but you're building an embedded system, so this may not have much sense put this way, simply consider it as a place for data used by the tools you'll need) : /usr/share
• A place for "variable files". As stated on the FHS page : "files whose content is expected to continually change during normal operation of the system, such as logs and spool files". These are files which you want to keep accross reboots : /var/log - /var/spool - ...
• A place for "run-time" files, which you do not want to keep accross reboots : /run
• A place for temporary files. Could be either one of the following : /tmp - /run/tmp
• A user home directory (or many if you're not building a single user, single task system) : /root - /home/....
• Somewhere you can mount removable stuff ? : /mnt
• If you need them, a place for modules and firmwares : /lib/modules - /lib/firmware
• And last but not least, some place specific for your application if you do not want it merged to the base system. Can be any or all of these : /opt - /srv - /usr/local ...

Of course, this may seem to be a lot, and it can be reduced to something simpler, depending on your needs !
Here is a very simple set of directories for a configuration without /usr :

mkdir -p dev etc lib proc root run sys tmp var/{log,spool}

And another with many more directories (see below for the "triplet" you'll need) :

triplet=
mkdir -p dev etc mnt proc root run sys tmp usr/{lib/{${triplet},firmware,modules},share} var/{cache,lib,log,spool}
ln -s usr/lib lib
ln -s run/tmp tmp
ln -s ../run/tmp var/tmp

As some in the second example, many can also be links to other ones ... once again it's up to you and your needs !

Libraries

First, the answer about the "triplet"

An inclomplete way to answer is that it is composed of three fields which identify the target for which you're building your system, though it is somehow much more complicated (it can be only two fields, or four), and there is not "one" answer. It is more of a convention which is not shared by all of the interested parties. You can get some information on osdev.org Wiki, on debian.org Wiki, or in GNU autoconf manual (also here).
Most of the time it looks like "arch"-"os"-"abi", though the names are not these ones.
It's usually what "-dumpmachine" or "-print-multiarch" option to the gcc from your toolchain will print, which is also the cross-compilation toolchain prefix.
As I'm using Devuan/Debian's cross-toolchains I stick to the convention adopted by the Debian project.

Which are the required libraries ?

In order to know which libraries are required on your system you get a few option.

• Makefiles and build process output
• Use ldd
• Use "readelf -d"

The first option is some kind of theoretical approach. Have a look at the makefiles or the build output for the libraries used for the linking stage of the build process, for all your binaries. This could be a pretty hard task, and is not my choice.

The second option, which I'm using most of the time (because I never remember the last one, which is the best one) is to use ldd on a target sharing the architecture (the triplet mentionned above) of your target system.
ldd is a command (script) which lists what shared libraries are used by given dynamically-linked executables (read comments in /bin/ldd for more information).
The main limitation is that you need to have a target running a full-blown distribution at hand.
On the other hand, it lists more than the libraries linked, and tells you whether it found them on the system, and which file is used (save for "linux-vdso.so.1" which is not a file but a "virtual dynamic shared object" provided by the Linux kernel : look at the vdso manpage : man 7 vdso).
Of course, you'll need to run this for all your binaries, but that's something you can perform with a succession of commands and filters (or a script), and get a list with single entries for the required libraries.
Here is an example output of ldd :

$ ldd /bin/bash
   linux-vdso.so.1 (0x00007fbb70314000)
   libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007fbb70170000)
   libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fbb6ff7a000)
   /lib64/ld-linux-x86-64.so.2 (0x00007fbb70316000)

The last option, directly available on your host, is to use the readelf tool from the binutils part of your cross-toolchain.
Once again, you'll need to run it on all your binaries.
Here is the interesting part of the output of "readelf -d" for the bash binary on my host system :

$ readelf -d /bin/bash
   Tag               Type                Name/Value
 0x0000000000000001 (NEEDED)     Shared library: [libtinfo.so.6]
 0x0000000000000001 (NEEDED)     Shared library: [libc.so.6]

Where do they go ?

Now that you know which libraries you need, you'll need to figure which files to put in which directory for each library.

/etc/ld.so.conf and ldconfig libc.so

bootup script

• inittab
• rc script(s)

Create an initramfs image for the rootfs

Creating an initramfs (a real one, not an initrd) is pretty simple and can be done in two different ways.

• From the command line, on your own :

cd test && find . > ../liste_cpio
cat ../liste_cpio | cpio -H newc -o > ../initramfs.cpio
gzip -9 initramfs.cpio
mv initramfs.cpio.gz initramfs.img

• Using the kernel build system.

TO BE CONTINUED