Dev/Embedded/Rootfs/BusyBox/Example2

If you want to use BusyBox as the heart of an almost complete but light system, with multi-user, many services (possibly including SSH for remote access) you will need some more stuff than the pretty small configuration from the previous example.

This example explains how to create a more complex Linux based system using BusyBox as a base for the userland part of the system.

Note :
THIS IS CURENTLY *VERY* INCOMPLETE.
I simply created the "second example" page in order to "store" the bits which were going too far for the first "simple example".

Complex system configuration

FIXME : Add link to example complex config

TO BE COMPLETED

Notes and warnings :

• Keep in mind that some tools may depend on specific kernel features (and includes) and that you may need to install your own kernel hearders and reference them instead of those comming with the libc.
  This is the case for the "tc" network applet which is included in the "defconfig".
• Adding headers or additionnal external libraries is done in the "Settings" part of the configuration, under "Additional LDFLAGS" and "Additional LDLIBS".
• As of 2025-01-03 selinux has deprecated "security_context_t" and "matchpathcon()" (at least) and you'll need to update the selinux related code if you need it.

Directories

You will not need a full set of directories from the FHS but this can be a good place to get information if you do not know what you need.

• The default busybox installation process has created four "*bin" directories for you, which is a good starting point (though you can merge them to get two if you want to follow the usermerge way, or even go further and get a single one bin directory). : /bin - /sbin - /usr/bin - /usr/sbin
• Then you will usually want somewhere to mount proc, sysfs ans devtmpfs : /proc - /sys - /dev
• You should also stick to the standard and place your text configuration under /etc
• Then you will need a place for your libraries. This can be a single place, or many, it's up to you (it's actually a real mess on some distributions, with multiarch support, 32 and 64 bits support, usrmerge, and maybe other funny stuff getting in the way). Maybe you should consider keeping it as simple as possible (refer to the next section to make your choices about this part). On a Devuan or Debian distribution you usually get those three at least : /lib - /usr/lib - /usr/lib/*arch-triplet* - ...
• A place to put "shared ressources" (architecture-independent data, but you're building an embedded system, so this may not have much sense put this way, simply consider it as a place for data used by the tools you'll need) : /usr/share
• A place for "variable files". As stated on the FHS page : "files whose content is expected to continually change during normal operation of the system, such as logs and spool files". These are files which you want to keep accross reboots : /var/log - /var/spool - ...
• A place for "run-time" files, which you do not want to keep accross reboots : /run
• A place for temporary files. Could be either one of the following : /tmp - /run/tmp
• A user home directory (or many if you're not building a single user, single task system) : /root - /home/....
• Somewhere you can mount removable stuff ? : /mnt
• If you need them, a place for modules and firmwares : /lib/modules - /lib/firmware
• And last but not least, some place specific for your application if you do not want it merged to the base system. Can be any or all of these : /opt - /srv - /usr/local ...

Of course, this may seem to be a lot, and it can be reduced to something simpler, depending on your needs !

And another with many more directories (see below for the "triplet" you'll need) :

triplet=
mkdir -p dev etc mnt proc root run sys tmp usr/{lib/{${triplet},firmware,modules},share} var/{cache,lib,log,spool}
ln -s usr/lib lib
ln -s run/tmp tmp
ln -s ../run/tmp var/tmp

As some in the second example, many can also be links to other ones ... once again it's up to you and your needs !

Libraries

If you chose to use shared libraries (shared object files) when you compiled your binaries then you need to include them in your final system image.

This step is not very difficult in itself, but not as simple as busybox compilation and installation.

First, the answer about the "triplet"

I mentioned this "triplet" twice in the above section.

An inclomplete way to answer is that it is composed of three fields which identify the target for which you're building your system, though it is somehow much more complicated (it can be only two fields, or four), and there is not "one" answer. It is more of a convention which is not shared by all of the interested parties. You can get some information on osdev.org Wiki, on debian.org Wiki, or in GNU autoconf manual (also here).

Most of the time it looks like "arch"-"os"-"abi", though the names are not these ones.
It's usually what "-dumpmachine" or "-print-multiarch" option to the gcc from your toolchain will print, which is also the cross-compilation toolchain prefix.

As I'm using Devuan/Debian's cross-toolchains I stick to the convention adopted by the Debian project.

Which are the required libraries ?

In order to know which libraries are required on your system you get a few option.

• Makefiles and build process output
• Use ldd
• Use "readelf -d"

The first option is some kind of theoretical approach. Have a look at the makefiles or the build output for the libraries used for the linking stage of the build process, for all your binaries. This could be a pretty hard task, and is not my choice.

The second option, which I'm using most of the time (because I never remember the last one, which is the best one) is to use ldd on a target sharing the architecture (the triplet mentionned above) of your target system.
ldd is a command (script) which lists what shared libraries are used by given dynamically-linked executables (read comments in /bin/ldd for more information).
The main limitation is that you need to have a target running a full-blown distribution at hand.
On the other hand, it lists more than the libraries linked, and tells you whether it found them on the system, and which file is used (save for "linux-vdso.so.1" which is not a file but a "virtual dynamic shared object" provided by the Linux kernel : look at the vdso manpage : man 7 vdso).
Of course, you'll need to run this for all your binaries, but that's something you can perform with a succession of commands and filters (or a script), and get a list with single entries for the required libraries.
Here is an example output of ldd :

$ ldd /bin/bash
   linux-vdso.so.1 (0x00007fbb70314000)
   libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007fbb70170000)
   libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fbb6ff7a000)
   /lib64/ld-linux-x86-64.so.2 (0x00007fbb70316000)

The last option, directly available on your host, is to use the readelf tool from the binutils part of your cross-toolchain.
The main limitation for this third option is that it does not tell you about the dynamic linker/loader (ld-linux-*.so)
Once again, you'll need to run it on all your binaries.
Here is the interesting part of the output of "readelf -d" for the bash binary on my host system :

$ readelf -d /bin/bash
   Tag               Type                Name/Value
 0x0000000000000001 (NEEDED)     Shared library: [libtinfo.so.6]
 0x0000000000000001 (NEEDED)     Shared library: [libc.so.6]

The minimal list

I cannot give you a full list, but if you chose to have dynamically linked binaries (which is usually a good idea if your system has more than one or two binaries) you will always need these two :

• The dynamic linker/loader (name depending on the target architecture/triplet) : ld-linux-*.so.*
• the libc

Where do they go ?

Now that you know which libraries you need, you'll need to figure which files to put in which directory for each library.

Unless you specified an absolute or relative pathname during the linking stage of the compilation of your binaries, there is no "fixed" location for each library file, save maybe the dynamic linker/loader (ld-linux-*.so.*), which usually goes under "/lib", and the dynamic linker will end up looking for libraries in some "default" path which usually includes "/lib".

So the easy way is to create a single "/lib" directory especially if you chose to desactivate the use of "/usr" in busybox configuration.

If you want to build a base system closer to what you find in common distributions and use the "/usr", here are some information you should be aware of :

The dynamic linker will end up looking for libraries in some "default" path, which varies depending on it's configuration and the architecture (usually "/lib" and then "/usr/lib", or '/lib64" and then "/usr/lib64").

Even if the linker/loader will first look at the "cache" file ("/etc/ld.so.cache") which is generated using "ldconfig" it is not of much use in our case because this "ldconfig" is a binary which knows only of one architecture, so (as for ldd) this command must be run on a target of the same architecture with a complete running system.

(but is under "/usr/lib/**triplet**/" in a Debian, whith a link from "/usr/lib64/")

UNDER REDACTION

libc.so

bootup script

• inittab
• rc script(s)

TO BE COMPLETED

configuration files

TO BE COMPLETED

application binaries

TO BE COMPLETED

application related files

TO BE COMPLETED