Dev/Embedded/FromScratch/LinuxKernel

Introduction

The Linux kernel will be the heart of your system.
It will have two functions :

• Handle the hardware (and provide an abstraction of the hardware to the userspace)
• Handle the userspace (schedule the processes, check permissions, ...)

Some technical stuff

The Linux kernel is a modular monolithic kernel, which means that some services are included in the main kernel image and cannot be changed at runtinme, but some part (the modules) can be loaded (and unloaded) dynamically, at runtime.
Note that the Linux kernel can be compiled without module support, with all the drivers included in the main image.

To be completed

You can find all technical documentation about the Linux kernel directly from kernel.org or in the "doc/" directory of the kernel sources.

Grab the sources

For the Linux kernel, you usually have two (sometimes three) choices :

• Get the official "vanilla" kernel from kernel.org
• Get the custom kernel provided by the board manufacturer
• Get a custom kernel from a community project when there's one available.

I try to use official kernels as much as possible, but sometimes you are stuck to a custom kernel (most of the time outdated).
If you have some time available, then you can try to port the patches from the custom kernel to the official one and have them included in the upstream official kernel, which will always be greatly appreciated, though the hardest part of this is not the technical part (effective porting of the patches) but getting them accepted upstream.

When getting the source from kernel.org you can use either git, or download an archive from kernel.org (which will lack git history but be smaller, both for download and for disk space)

git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

Otherwise, grab the source from whatever alternative source you identified as suitable for your board.

If you cloned a git repository, you may want to checkout a specific tag or revision in order to be in sync with a version known to run fine on your hardware.

It is also time to add any patch you received on top of your source tree in order to add any specific support which is not included in the cloned or extracted kernel sources.

Kernel Configuration

Preamble : external build

It's usually a god idea to build the Linux kernel in a separate Build directory in order to keep the sources clean and allow you to run multiple builds using a single source tree, saving disk space and making it easier to spot what got compiled.

Let's suppose you have your sources in the "linux" directory then to setup the Build environment you should run the following commands (set the arch variable to the right value for your hardware) :

mkdir -p Build/linux
cd linux
arch=
make ARCH=${arch} -j8 O=../Build/linux/ allnoconfig

Note that "allnoconfig" can be replaced by "alldefconfig".

You now have a kernel "Build" directory, and should run all further commands from this build directory in order to keep the sources clean.

The .config file

The Linux Kernel configuration is stored in a file named ".config" at the root of the Build directory.

This is a text file, so it is easy to share it and store it, and look at what's included in a given kernel by looking at this file.

It is sometimes possible to use a "default" configuration provided with the kernel sources, though for for most ARM board you will not find a configuration to fit your board from the official kernel sources, only one matching your processor familly. The reasons behind this are that there are too many socs out there, and much much more boards using them, even if you only count the commercial ones, and also that the "hardware" related part of a configuration is only a fraction of a Kernel configuration, which is really specific to each user ans use case (the "get everything as modules" approach used by distributions does not fit embedded systems).

So the usual way is for the board designer to provide an example configuration which should include all the support of your hardware. For the parts which are not hardware related (IPCs, network protocols, debugging, file-systems, ...) you'll get whatever comes with the hardware configuration and you'll need to spend some time tunning the configuration to your needs.

As for the technical/practical part of how to do this : copy the config file you received from the board manufacturer as ".config" at the root of the build directory, overwriting the one generated during the previous step, and run either "oldconfig" or "olddefconfig". If you are using the same kernel version as the one the configuration file came from, then the configuration is simply checked, but if you're using a newer kernel version, then the "oldconfig" case will prompt for a choice for each new configuration option, while the "olddefconfig" case will use the default choice for each new configuration option.

cp vendor_provided_config Build/linux/.config
cd Build/linux/
make ARCH=${arch} -j8 oldconfig

You can then check and tune the configuration for your needs by running "menuconfig" (or "xconfig" for a Qt frontend, or gconfig for a GTK+ configuration frontend) :

make ARCH=${arch} -j8 menuconfig

The device tree

Why a device tree

While for the x86 and x86_64 architecture (and so any computer running such a processor) the config file holds the whole configuration of the Linux Kernel, embedded targets running ARM processors (and any architecture which has a "dts" directory within the Linux kernel sources) need a second configuration file which describes the available hardware and the board specific hardware configuration.
You can get the list of concerned architectures by running this command :

find arch/ -type d -name "dts"

The origin of this requirement is from somewhere between 2004 and 2005 (initial commit of the device tree compiler is 2005-06-08).
Systems running on x86* (and a few other) based hardware use discovery mechanisms like ACPI or PCI bus enumeration to detect available hardware at runtime and as such can run a generic kernel image. But most other architectures use SoC (System on Chip) processors which do not provide such discovery mechanisms to detect integrated buses or available modules, and most of these SoCs support multiple configurations for each of their pins, connected to different modules, which can be used (or not) by a given board. Thus these cannot use a generic kernel image and need a specific kernel image.
As more and more SoCs became available, and many more boards using them, Linux kernel developers used to provide support for each board as a new set of C files holding copy of the required drivers with the specific configurations for the board, which introduced a bunch of duplicated code.
Linux Torvald put an end to this by refusing any patch which added a new board support and asked the developers to find a way to support these board specific configurations in a better way.
The solution came from a "Flattened Device Tree" data format which is derived from the OpenFirmware specifications, first used for PowerPC boards, and later on for ARM systems and other architectures.

You may find additional information here.

What are device trees used for ?

The goal of the device tree is to provide board specific configuration.
While the kernel configuration (".config") activates code (drivers) to support a specific module within a SoC (let's say a serial module), the driver will support all modes this serial module can use, and the SoC may have many instances of this serial module, not necessarily running in the same mode for a given board.
The role of the device tree is to describe how the Soc must be configured, which means, for our example, how many serial modules will be activated, in which mode, where each module's configuration registers will be found, which pins they use, and so on.
The device tree will also provide information on the devices available on the busses lacking discovery mechanisms (I2C, SPI, UARTs, ...) so that the corresponding drivers can be loaded at runtime if available.
Thus, the kernel image will include all the code required for the SoC support (can be reduced to the used interfaces of course), while the device tree holds the SoC configuration for each interface.

Once again, you will find more information on the device tree pages of eLinux Wiki.

Technical part

All the technical information about the device tree concepts can be found on the Device Tree Usage page of elinux.org as well as the Linux kernel specific information.

You will also get information from the pages dedicated to the device tree in the kernel documentation.

Practical part

• Device tree files location :

The device tree for each supported board is located somewhere under arch/<arch name>/boot/dts/<vendor> (the <vendor> part is not present for all architectures). It is often split in some generic "SoC" support parts (*.dtsi files) included in a final "board" (*.dts) file.

• Create a new device tree :

Usually, you do not modify existing dts files which refer to an existing board (much less dtsi files), but rather copy an existing one (with a new name refering to your board or project) and add it to the list of device trees to be compiled within the Makefile for the device trees for your SoC.

• Sharing device tree files :

Sharing device tree files is easy when you specify the kernel version used with them and stick to this version, but when the kernel version changes too much the content of the included device tree parts and the drivers interfaces may also become incompatible with your device tree file and you then have to update the device tree content to the new kernel version.

When you receive a specific device tree file, you add it to the dts directory (or rather the sub-directory corresponding to the SoC vendor) and to the list of target dtb in the corresponding Makefile.

• Compilation of device tree files :

Compilation of device tree files is automatic when compiling the Linux kernel, but you may want to re-run the compilation, which is done using the following command if you want to avoid the overhead of checking the whole kernel for modifications : compiler_prefix= arch= make ARCH=${arch} -j8 CROSS_COMPILE=${compiler_prefix} dtbs

Kernel Compilation

The compilation part is rather simple, provided that you have all the required dependencies installed.
From the Build directory created in the configure step :

compiler_prefix=
arch=
make ARCH=${arch} -j8 CROSS_COMPILE=${compiler_prefix}

And that's it !

Compilation results

Linux Kernel

Compilation result is in "arch/${arch}/boot/"
The image name depends on your architecture and target. Most of the time it is set according to your configuration and there's no additional step required, but in some specific situations you may need to run an additional tool to build the final image for your bootloader.
This will be detailled on each board specific page (tough I actually have no boards/configurations where this is required, I've had to do it for some custom boards).

This image file (your kernel binary) will then need to be loaded by your bootloader, so the next steps with this depends on your boatloarder, your board, and the chosen boot procedure.

Device tree

The binary version of the device-tree is to be found under "arch/${arch}/boot/dts/".
There's usually more than one compiled even with only one SoC selected in the configuration, so you'll have to pick the right one for your board.

This binary device tree file will also need to be loaded by your bootloader, so it usually goes somewhere close to the Linux kernel image in about the same way, which also depends on your boatloarder, your board, and the chosen boot procedure.

Modules

If you have activated the modules support in your kernel configuration (which is usually the default) and selected some services or drivers to be compiled as modules, then you will need to install them.

But usually, when building the kernel for an embedded system, your kernel sources are not on the target machine, so the default "make modules_install" cannot be used (it would install your target's modules on your host development system). Instead you'll need to specify the modules installation path by setting "INSTALL_MOD_PATH" :

make ARCH=${arch} -j8 CROSS_COMPILE=${compiler_prefix} modules_install INSTALL_MOD_PATH=../results/

If the compilation has been performed using the above command lines you should have a directory "Build/results/lib/modules/***" (*** = name of the kernel you just compiled).

You must copy the whole directory (not only the content) on the SD card, most probably in the partition holding the root filesystem (rootfs), in the /lib/modules/ directory without changing the name.
It could go to another partition, but then you'll need to make sure that all the drivers to access this partition are built inside the main kernel image.