Build system is a set of makefiles and utilities organized within one directory which is mounted into the source tree of the developed product.

The main purpose of the build system is automating all stages of software solution development, from arrangement of source code from third-party developers usage to publication of own deliverable distributive.

By no means we have in mind to present our work as some kind of out-of-the-box product that can be used even by unexperienced users; we assume that the user has some basic knowledge and experience. For this reason, the information here is presented in the form of articles and not as detailed instructions or manuals. We expect that, while reading the material, the reader will compare our solutions to the solutions he or she has implemented in their everyday work.

Introduction

Build system is a set of Makefiles and scenarios written on Bash and Perl programming languages. The following listing illustrates the short tree of the build system. The heart of the system is the file core.mk containing all principal building rules that allow simplifying user Makefiles creation.


 .
 ├── Makefile
 ├── _kxLab.pm
 ├── apply_patches
 ├── build-config.mk.template
 ├── build_packages_list
 ├── build_pkg_requires
 ├── build_requires
 ├── build_requires_tree
 ├── build_src_requires
 ├── canonical-build
 ├── constants.mk
 ├── core.mk
 ├── dist_clean
 ├── downloads_clean
 ├── global_clean
 ├── install_pkgs
 ├── install_targets
 ├── rootfs_clean
 ├── target-setup.mk
 ├── tree-build-system.mk
 ├── tree-src.mk
 ├── tree.mk
 ├── 3pp
 │   └── fakeroot
 │       └── 0.18
 │           └── Makefile
 ├── pkgtool
 │   ├── change-refs
 │   ├── check-db-integrity.in
 │   ├── check-package.in
 │   ├── check-requires.in
 │   ├── install-package.in
 │   ├── install-pkglist
 │   ├── make-package
 │   ├── make-pkglist
 │   ├── pkginfo
 │   ├── pkglog
 │   └── remove-package.in
 ├── html
 │   └── requires_tree_html.template
 └── scripts
     ├── fill_align
     └── fill_to_size

From the whole set of files, only the files constants.mk and core.mk are of interest to a regular build system's user. They are included in user Makefiles using the include directive.

The build system tree is mounted into the build-system directory of the common source code tree from which programming packages are built for different devices. The common tree can look, for example, as shown below:


 .
 ├── .svn
 ├── app
 ├── base
 ├── boot
 ├── build-system
 ├── dev
 ├── doc
 ├── hal
 ├── libs
 ├── net
 ├── products
 ├── secure
 ├── share
 ├── sources
 ├── .svnignore
 └── Makefile

The build system has no assumptions regarding the structure of the common source code tree apart from a couple of directories. Let's look at these directories in details.

Directory sources

In the sources directory Makefiles are created that are used to download source programming packages for consequent building. The structure of the Makefiles in this directory is fundamentally different from the other directories, because these Makefiles are needed only to download archives from a FTP server, and not to build the application's components.

Directory dist

The dist directory is temporary. It is created automatically and used to store the build results. We will have a closer look at the structure of this directory after we describe several build system features.

Hello, World

It is well-known that the best way to learn a system is to use it in practice. Because of that, before we start to describe the build system in details, we want to use it to write some simple programs and Makefiles.

The build system's purpose is to simplify writing user Makefiles that help to build software solutions for several different devices simultaneously using the corresponding cross-compilers. Because of that, the first lines of any Makefile shall contain the list of devices this Makefile is intended for.

COMPONENT_TARGETS  = $(HARDWARE_BUILD)
COMPONENT_TARGETS += $(HARDWARE_AT91S)

Here, we have described two targets: first, it's the developer's machine HARDWARE_BUILD, on which the finished application can be tested, and second, a board based on the Atmel AT91SAM7S micro controller (HARDWARE_AT91S). The names of the target devices in use can be seen in the file build-system/constants.mk. And, of course, we need to have the definitions from constants.mk in our Makefile. For this, we will use the include directive.

COMPONENT_TARGETS  = $(HARDWARE_BUILD)
COMPONENT_TARGETS += $(HARDWARE_AT91S)

include ../../build-system/constants.mk

It is important to note that to include files belonging to the build system, the user must use relative paths.

Since we are working on a computer with x86_64 architecture, it would be nice to also build a 32-bit version of our application. To do this, we need to define another target device variant, this time using the FLAVOUR feature of the build system. This possibility is provided mainly to help the developer distinguish between several target device modifications within the same product line built on the same central processor, but having some differences in periphery. Of course, the way we create here a 32-bit application can hardly be applied in the production environment, but we will still take it to demonstrate the system's possibilities. So, let's define another modification of the application:

COMPONENT_TARGETS  = $(HARDWARE_BUILD)
COMPONENT_TARGETS += $(HARDWARE_AT91S)

include ../../build-system/constants.mk

BUILD_FLAVOURS = hello-32compat

To assure parallel operation, the build system will create the directory structure shown in the following listing. This structure is created automatically, and the user should use these directories to eliminate mutual influence of applications during building.

 .
 ├── .at91sam7s-newlib
 │   └── at91s
 ├── .build-machine
 │   └── build
 │       └── hello-32compat
 ├── Makefile
 └── main.c

To free the user from specifying full paths to the created directories, the build system automatically creates the TARGET_BUILD_DIR variable. The value of this variable changes depending on the target for which our program main.c is currently being built. This way, when the program is built for the target HARDWARE_AT91S, the value of the TARGET_BUILD_DIR variable equals .at91sam7s-newlib/at91s. During building for the developer's machine HARDWARE_BUILD, the TARGET_BUILD_DIR variable has the value .build-machine/build. And, finally, when the 32-bit version of the program is being built, the TARGET_BUILD_DIR variable has the value .build-machine/build/hello-32compat. It is worth noting, that the path in TARGET_BUILD_DIR is relative to the current directory where our Makefile is located. The user can get the absolute value of the current path using the $(CURDIR) variable provided by the Make utility.

Let's continue creating our Makefile. Our "Hello, World" program consists of the only file main.c, so the list of source files is short:

bin_srcs = main.c

SRCS = $(bin_srcs)

SRCS is the key variable in the build system. For all files contained in this list, a cross-compiler will be called according to the filename extension to get the corresponding object file.

To put the resulting object files into the TARGET_BUILD_DIR directory that we mentioned before, we need to define the list of these object files:

bin_objs = $(addprefix $(TARGET_BUILD_DIR)/,$(bin_srcs:.c=.o))

Needless to say, the name of the resulting application must also be defined:

bin_target = $(TARGET_BUILD_DIR)/main

Since we use the compiler installed on the developer's machine to build the application for 32-bit Intel architecture, and this compiler is able to generate code for two different architectures, we have to redefine the compiler controls when the x86_32 modification of our program is built:

ifeq ($(FLAVOUR),hello-32compat)
ARCH_FLAGS := -m32 -march=i486 -mtune=i686
CFLAGS     += -DFLAVOUR=32
LDFLAGS    += $(ARCH_FLAGS)
endif

Now we can define the last lines of our Makefile, where, using the include directive, we will include the core of our build systembuild-system /core.mk – into our file:

Here is the full content of the Makefile:

COMPONENT_TARGETS  = $(HARDWARE_BUILD)
COMPONENT_TARGETS += $(HARDWARE_AT91S)

include ../../build-system/constants.mk

BUILD_FLAVOURS = hello-32compat

bin_srcs = main.c

SRCS = $(bin_srcs)

bin_objs = $(addprefix $(TARGET_BUILD_DIR)/,$(bin_srcs:.c=.o))

bin_target = $(TARGET_BUILD_DIR)/main

ifeq ($(FLAVOUR),hello-32compat)
ARCH_FLAGS := -m32 -march=i486 -mtune=i686
CFLAGS     += -DFLAVOUR=32
LDFLAGS    += $(ARCH_FLAGS)
endif

BUILD_TARGETS = $(bin_target)

include ../../build-system/core.mk

$(bin_target): $(bin_objs)
       $(LINK)

So, our Makefile is ready. The BUILD_TARGETS variable needs to be mentioned separately. The list contained in BUILD_TARGETS is critical. For the targets specified in this list the build system doesn't perform any actions, but it can't proceed to execute the rest of the tasks until these targets are built. The main job of building the object code is performed using the $(LINK) call. The $(LINK) command is essential and is described in the general rules section of the build system. These rules are intended for building user programs contained directly in the source code tree, as opposed to the third-party programs which, usually, arrive for building in the form of compressed source packages.

To build our program following the $(LINK) call, the system will automatically, depending on the contents of the $(SRCS) list, choose the matching compiler and linker, call the command represented by the value of the $(LINK) variable, wait until it is finished, which means processing all targets from the BUILD_TARGETS, and quit.

Now, by calling make, we will get a set of files that is better represented as a tree:

 .
 ├── .at91sam7s-newlib
 │   └── at91s
 │       ├── main
 │       ├── main.d
 │       ├── main.linkmap
 │       └── main.o
 ├── .build-machine
 │   └── build
 │       ├── hello-32compat
 │       │   ├── main
 │       │   ├── main.d
 │       │   ├── main.linkmap
 │       │   └── main.o
 │       ├── main
 │       ├── main.d
 │       ├── main.linkmap
 │       └── main.o
 ├── .makefile
 ├── .src_requires
 ├── .src_requires_depend
 ├── Makefile
 └── main.c

If our source program main.c looks like this:

#include <stdlib.h>
#include <stdio.h>

int main()
{
#if FLAVOUR == 32
   printf( "\nHello, World! (x86_32 object built for running on x86_64 machine)\n\n" );
#else
   printf( "\nHello, World!\n\n" );
#endif
   return( 0 );
}

then calling main in the .build-machine/build/hello-32compat directory will produce the following result:

$ cd .build-machine/build/hello-32compat
$ ./main

Hello, World! (x86_32 object built for running on x86_64 machine)


$

In the following sections we will study in more details the various aspects of using the build system, but even now it can be said with certainty that the best way to learn a system is its practical usage. In addition, the user can always avoid developing his/her own Makefiles from scratch by simply copying one of the many existing Makefiles created by other developers to his/her directory and modifying it in accordance with his/her own goals.

Targets and Directories

In the previous section we mentioned such concepts as the temporary build directory TARGET_BUILD_DIR and the list of primary build targets BUILD_TARGETS, in relation to the Make utility. Let's investigate these concepts in more details.

Temporary Build Directories

First and foremost, the build system is designed for the situation when the same source code can be built for several target devices. In other words, the same Makefile can be used multiple times to build the same program for different devices. And if we assume that the program is built in parallel, i.e. simultaneously for all targets, then it is important to make sure that, at least, the intermediary object files and resulting applications are put during building to different directories.

To differentiate between the intermediary build results destined for different devices, the build system automatically creates temporary directories. The naming convention is simple. Since the build system uses only three concepts, namely, TOOLCHAIN, HARDWARE and FLAVOUR, where FLAVOUR is optional, the name for a temporary directory is created using the following formula:

TARGET_BUILD_DIR = .$(TOOLCHAIN)/$(HARDWARE)$(if $(FLAVOUR),/$(FLAVOUR)))

Of course, the TOOLCHAIN, HARDWARE and FLAVOUR variables that at any time have values corresponding to the current target are always available to the user. By analyzing these variables' values in his/her own Makefile, the user can see, for example, which toolchain is used currently for building. Is it also important to note that the directory name is formed relatively to the current directory in which the user's Makefile is stored, and its name starts with a period ('.'). From the perspective of Unix-like systems, directories and files that have names starting with a period are hidden. Using hidden directories allows maintaining a cleaner list of user files and, in addition, it prevents potential errors during clean-up (using the make clean command) when, normally, it is enough to delete hidden files and directories.

The build system's users should make it a rule that it is best to create all intermediary targets in the TARGET_BUILD_DIR directory. This prevents many problems, especially during parallel building.

Reserved Temporary Files

It is important to mention that the build system creates its own temporary files in the TARGET_BUILD_DIR directory. User isn't allowed to create or modify temporary files reserved by the build system.

$(TARGET_BUILD_DIR)/.dist
$(TARGET_BUILD_DIR)/.rootfs
$(TARGET_BUILD_DIR)/.requires
$(TARGET_BUILD_DIR)/.requires_depend

In addition, in the current directory where the user's Makefile is located, the build system can create files describing the current package's dependencies on the source archives.

$(CURDIR)/.src_requires
$(CURDIR)/.src_requires_depend

User has to remember that all aforementioned temporary files belong to the build system, and in no circumstances he or she is allowed to create files having the same names by themselves.

Main Targets

From the perspective of the Make utility, the build system has three lists of main targets.

The most important and necessary list that must be defined in the user Makefile is the list of build targets BUILD_TARGETS. If this list is empty, the build system will not perform any actions. In general, the main target must be a package creation that, in turn, can be a deliverable intended for installation to the root filesystem of the target device. Here it should be mentioned that the build system only tries to reach the targets specified in the BUILD_TARGETS list and doesn't perform any other actions.

The other two lists are optional and are related to installation of the ready packages. These are the list of products PRODUCT_TARGETS and the list of targets intended for installation to the temporary root system of a device, namely, ROOTFS_TARGETS.

Before studying these lists in details, we should tell about one more feature of the build system.

Destination Directories

For temporary storage of the build results, the dist directory is created at the source tree root. This directory contains subdirectories of the following three types:

  1. Directories for temporary storage of the development environment.
  2. Directories for temporary storage of the ready packages.
  3. Directories for temporary storage of the root filesystem images with installed packages having the form that they will have in the target filesystem.

Let's study the structure and purpose of the directories in more details.

The first type includes directories that have names defined by the TARGET_DEST_DIR variable.

Target Destination Directory

The TARGET_DEST_DIR directory is used for temporary storage of the target platform's development environment. In this directory all applications and header files should be installed that are needed for software building. The full name of this directory is formed as follows:

TARGET_DEST_DIR = $(TOP_BUILD_DIR_ABS)/dist/.$(TOOLCHAIN)/$(HARDWARE)

Here the TOP_BUILD_DIR_ABS variable contains the full path to the root of the source tree. This path is detected by the build system automatically, which allows storing local copies of the repository branches anywhere in the filesystem without any additional set-up by the user.

There are two ways to install the build result in the TARGET_DEST_DIR directory. The first way is to use the cp call, for example:

$ cp -a $(TARGET_BUILD_DIR)/build/libintl.h $(TARGET_DEST_DIR)/usr/include

In case the approach with the cp call was chosen, the user has to explicitly create the target directory:

$ mkdir -p $(TARGET_DEST_DIR)/usr/include

before copying the files.

The best way to copy files into the TARGET_DEST_DIR directories is to use the install_targets utility of the build system. The install_targets utility is a simple scenario written on Perl. It is advisable to specify the full path to the install_targets utility in user Makefiles. For this, the BUILDSYSTEM variable pointing to the build-system directory of the current source tree is provided by the build system. This way, to copy the file '$(TARGET_BUILD_DIR)/build/libintl.h' to the temporary storage, it is sufficient to write:

       $(BUILDSYSTEM)/install_targets        \
         $(TARGET_BUILD_DIR)/build/libintl.h \
         $(TARGET_DEST_DIR)/usr/include $(HARDWARE)

The install_targets utility allows installing multiple files by reproducing the source directory tree in TARGET_DEST_DIR. The following is a more complicated example of using the install_targets utility.

       @cd $(TARGET_BUILD_DIR) && \
         CWD=$(CURDIR)            \
           $(BUILDSYSTEM)/install_targets --preserve-source-dir=true \
             $$(find * -path 'lib*' -type f -print) $(TARGET_DEST_DIR) $(HARDWARE)

Here all files from directories having names starting with lib are being copied. These can be, for example, the directories '$(TARGET_BUILD_DIR)/lib' and '$(TARGET_BUILD_DIR)/libexec'.

It is important to note that using install_targets allows the user to avoid creating target directories manually before copying files.

There is an even simpler way to install files into the temporary development environment which is using the install-into-devenv function. In the section named User's Makefile you will find an example of using this build system's function.

Let's get back to our "Hello, World" example and add a line into the Makefile to describe installation of our application to the $(TARGET_DEST_DIR) directory:

$(bin_target): $(bin_objs)
       @$(LINK)
       @CWD=$(CURDIR) $(BUILDSYSTEM)/install_targets $(bin_target) \
                                        $(TARGET_DEST_DIR)/bin $(HARDWARE)

This way, the execution will result in the following directory tree:

 .
 └── dist
     ├── .at91sam7s-newlib
     │   └── at91s
     │       └── bin
     │           └── main
     └── .build-machine
         └── build
             └── bin
                 └── main

Here it is important to explain that if we are talking about automatic installation, the build system doesn't support device modifications (FLAVOURS). This is because there is no need to build the full set of target software for every modification. Usually, only a limited set of applications and device drivers depend on a certain modification. Because of this, the user should make his/her own decision regarding which exactly modification of the application shall be installed in the system. This kind of decisions should be made during distributive preparation from the common set of packages.

Target Products Directory

The PRODUCTS_DEST_DIR directory is used for storing already built packages of software applications. Packages built with the goal of subsequent distribution are installed into this directory according to group names. For each package group a separate subdirectory is created in the PRODUCTS_DEST_DIR directory. The name for this directory is formed automatically according to the current target.

PRODUCTS_DEST_DIR = $(TOP_BUILD_DIR_ABS)/dist/products/$(TOOLCHAIN)/$(HARDWARE)

The user doesn't have to create this directory explicitly. It is enough to define the PRODUCT_TARGETS list, and the build system will automatically create the required subdirectories and store the target files in them.

The build system automatically creates the needed directories and installs there the files presented in the PRODUCT_TARGETS list. Usually a package is accompanied by two files containing the hash sum and textual description of the package. The build system provides features that simplify product list creation in user Makefiles. In practice, a PRODUCT_TARGETS list creation looks as follows:

pkg_basename    = $(NAME)-$(VERSION)-$(ARCH)-$(DISTRO_NAME)-$(DISTRO_VERSION)

pkg_archive     = $(TARGET_BUILD_DIR)/$(GROUP)/$(pkg_basename).$(pkg_arch_suffix)
pkg_signature   = $(call sign-name,$(pkg_archive))
pkg_description = $(call desc-name,$(pkg_archive))
products        = $(call pkg-files,$(pkg_archive))

PRODUCT_TARGETS = $(products)

The key components here are:

The package group name $(GROUP) is defined by the user.

  1. variable pkg_arch_suffix that defines the extension of the package name. Currently, the extension is txz;
  2. function sign-name that replaces the $(pkg_arch_suffix) suffix with sha256;
  3. function desc-name that replaces the $(pkg_arch_suffix) suffix with txt;
  4. function pkg-files that creates the full list of files to be installed into PRODUCTS_DEST_DIR.

You will find more detailed description of packages and package groups in the section that tells about the utilities provided by the package manager pkgtool.

Target Root File System Directory

The ROOTFS_DEST_DIR directory contains the target root filesystem. The name of this directory is formed following this rule:

ROOTFS_DEST_DIR = $(TOP_BUILD_DIR_ABS)/dist/rootfs/$(TOOLCHAIN)/$(HARDWARE)

Packages specified in the ROOTFS_TARGETS list are installed into this directory automatically. The only thing the user has to do is creating this list, for example, like this:

ROOTFS_DEST_DIR = $(pkg_archive)

Here the pkg_archive variable contains the relative filename for the package.

Service Targets

To simplify search for information, the build system provides special service targets. For example, to output the list of all valid targets, the developer can call

$ make help

This command will show on the screen the list of all valid targets and, in addition, will display the list of devices that are supported by the Makefile in the current directory.

Requires Tree

After all build targets are processed, it is possible to create a requires tree in JSON format reflecting the interdependencies between the packages and directories of the source tree. This creates an HTML document accompanied by a JSON data structure which is located in a separate file with '.json' extension.

To create such tree, run the following command:

$ make requires_tree

If the target device is not specified, then this command will build trees for all device types supported by the Makefile in the current directory.

Figure 1 illustrates the requires tree bind-9.10.1. We built it using the command

$ HARDWARE=omap5uevm make requires_tree

Running this command creates three files in the TARGET_BUILD_DIR directory: omap5uevm.html, omap5uevm.json and omap5uevm.min.json. The last is a compressed copy of the JSON file.

In the source code tree, all prepared files are temporary. For this reason they are created in the temporary TARGET_BUILD_DIR directory to be used later, for example, to create a repository for deliverable packages. This is the common approach which is used as an additional way to avoid unwanted commits to the repository.

Fig.1. Requires Tree

Requires trees, like the one shown on figure 1, are very useful both for the developer, who can use them to optimize dependencies between packages and debug package information, and for the delivery engineer, who can use the resulting trees during distributive preparation and writing accompanying documents for the product.

It is important to note that the requires_tree target should be called only after the main targets are already built in the directory for which the requires tree is prepared. Moreover, calling

$ make requires_tree

is only possible manually. In other words, no calls to the requires_tree target must be included in the user Makefile in the hope that after building is finished, the requires tree will automatically be prepared for the packages created in the current directory.

To prevent the attempts to automatically build trees after the product build is finished, before the requires_tree target is called, the system checks whether the $(TARGET_BUILD_DIR)/.requires file exists. It is done using the GNU Make $(wildcard ...) function, which makes it necessary for the user to run the Make utility again from the command line.

Packages List

The packages_list target is used to create the list of packages intended for installation on the target device.

To get this list, run the following command:

$ make packages_list

If the target device is not specified, then this command will build lists for all device types supported by the Makefile in the current directory.

Of course, using the HARDWARE variable, it is possible to specify the target device's name:

$ HARDWARE=omap5uevm make packages_list

In the TARGET_BUILD_DIR directory, this command creates the omap5uevm.pkglist file containing the list of packages. The order of packages in this list represents the order in which they are to be installed. This file is used by the install-pkglist utility which is included in the pkgtool toolset.

In the same manner as with creating the requires tree for packages, the list of packages can only be created if the $(TARGET_BUILD_DIR)/.requires file exists and all main targets in the Makefile of the current directory are reached.

Devices Table

To be able to create filesystem images using such utilities as genext2fs or populatefs, the developer has to know the list of device files that will be created in the /dev directory. Some devices' files are required at the initial system boot stage, when the virtual filesystem /dev has not been unrolled yet. In addition, device files may be required at loader's installation; for example, LILO, which is often used on Intel machines, requires that device files are in the /dev directory even if LILO is installed not on a working machine, but in chroot-environment.

The genext2fs and populatefs utilities take the device table in the form of a file similar to the following:

# device table

# <name>         <type>   <mode>   <uid>   <gid>   <major>   <minor>   <start>   <inc>   <count>
/dev/console     c        0600     0       0       5         1
/dev/fb0         c        0660     0       18      29        0
/dev/fb1         c        0660     0       18      29        1

/dev/initctl     p        0600     0       0
/dev/kmem        c        0640     0       9       1         2
/dev/kmsg        c        0644     0       0       1         11
/dev/kvm         c        0600     0       0       10        232
/dev/mem         c        0640     0       9       1         1
/dev/mmcblk0     b        0660     0       0       179       0
/dev/mmcblk0p1   b        0660     0       0       179       1
/dev/mmcblk0p2   b        0660     0       0       179       2
/dev/mmcblk0p3   b        0660     0       0       179       3
/dev/mmcblk0p4   b        0660     0       0       179       4

/dev/mtd0        c        0660     0       0       90        0

/dev/null        c        0666     0       0       1         3
/dev/ppp         c        0660     0       16      108       0
/dev/ram0        b        0640     0       6       1         0

/dev/random      c        0666     0       0       1         8
/dev/sda         b        0660     0       6       8         0
/dev/sda1        b        0660     0       6       8         1
/dev/sda2        b        0660     0       6       8         2
/dev/sda3        b        0660     0       6       8         3
/dev/sda4        b        0660     0       6       8         4

/dev/tty         c        0666     0       5       5         0
/dev/ttyS0       c        0660     0       16      4         64
/dev/ttyS1       c        0660     0       16      4         65

/dev/urandom     c        0666     0       0       1         9
/dev/zero        c        0666     0       0       1         5
/var/log/wtmp    f        0644     0       22
/var/run/utmp    f        0644     0       22

The table can contain either special devices' files or common files and directories.

Let's see now what the build system does when the following user command needs to be executed:

$ make devices_table

Looking through all packages from the $(HARDWARE).pkglist list in the $(PRODUCTS_DEST_DIR) directory, the build system identifies:

  • device files;
  • files having irregular attributes, for example, files with the suid, sgid or sticky bits set;
  • files belonging not only to the super-user, but also to special user groups, such as /var/log/wtmp, that has uid:gid equal to root:utmp;

and creates the corresponding records in the output file $(TARGET_BUILD_DIR)/.DEVTABLE .

In addition to the packages from the $(HARDWARE).pkglist list, the build system will check the existence of the $(PRODUCTS_DEST_DIR)/base/init-devices-*.txz package, and if it exists, the build system will add to the resulting list the files from this package that meet the requirements listed above.

The init-devices-*.txz package is a special package not intended for automatic installation mode that is used in two cases only:

  1. To prepare the list of devices for the /dev directory which is used during root filesystem's image creation using the populatefs utility.
  2. To install the system in interactive mode to the specified directory or to the specified media partition on behalf of super-user.

The goal of this approach is that development can be done by non-privileged users, and super-user privileges are only utilized when it is required.

To illustrate how packages containing special devices' files are created on behalf of a non-privileged user, the file base/init-devices/Makefile from the Radix.pro platform's repository can be considered.

Root Ext4 FS Image

As we mentioned before, during system build, all packages are installed into the temporary directory $(ROOTFS_DEST_DIR) that represents some image of the target device's root filesystem's contents. And, while this directory is suitable for mounting to the target machine via the NFS protocol, it still cannot be used to create filesystem on a hard drive or another media. This is because the packages were built and installed on behalf of a non-privileged user, so all files in the $(ROOTFS_DEST_DIR) directory have incorrect access rights and don't belong to the super-user. Needless to say that this image also lacks the required device files in the /dev directory, that, again, can only be created on behalf of the super-user.

To automate root filesystem creation meeting all the requirements of Linux environment, the build system provides possibility to create an Ext4 image of a filesystem in a common file which can be written to the target media using the dd command.

To create an image of the root filesystem, the user should issue the following command:

$ make ext4fs_image

Of course, if the goal is to create the filesystem for only one target device, for example the MIPS Creator CI20 board, it is possible to explicitly specify the name of the target machine:

$ HARDWARE=ci20 make ext4fs_image

If the device table (the .DEVTABLE file) is missing in the $(TARGET_BUILD_DIR) directory when the ext4fs_image is being created, this table is created automatically, because the ext4fs_image target depends on the devices_table target.

The size of the filesystem can be specified using the size argument, for example:

$ make ext4fs_image size=14.49G

The size can be defined in bytes, kilobytes, megabytes and gigabytes by adding the corresponding suffix: K, M or G with no dividing space and occupying only one character in uppercase. Of course, when the size is specified in bytes, no suffix needs to be used.

It is worth noting that the target filesystem's size cannot be less than the size of the $(ROOTFS_DEST_DIR) directory's contents plus 40% and, if the size requested by user is less than the allowed limit, then the filesystem will be created without taking into account the user requirements.

In addition to the root filesystem image, an MBR (Master Boot Records) boot record will be created containing the partition table for the target media, where the first 446 bytes are filled with zeros. This way, the result of running the following command:

$ HARDWARE=ci20 make ext4fs_image

will be two files in the $(TARGET_BUILD_DIR) directory: $(HARDWARE).ext4fs and $(HARDWARE).SD.MBR .

Products Release

The products_release target is intended for installing the $(HARDWARE).ext4fs, $(HARDWARE).SD.MBR and $(HARDWARE).pkglist files into the $(PRODUCTS_DEST_DIR) directory. Running the command

$ make products_release

allows automating the procedures of preparing the full set of resources required to install the product to the target device. This way, the command

$ HARDWARE=ci20 make products_release

will produce for the MIPS Creator CI20 device not only the set of deliverable packages prepared during build, but also the list of packages in the order reflecting the inter-package dependencies. In addition, since the $(PRODUCTS_DEST_DIR) directory will contain the image of the target filesystem, the installation program will be able to suggest to the user to simply copy the filesystem to the target drive instead of installing the packages one by one.

Considering the specified target device's name, the full names of these files relative to the root of the built repository branch will be as follows:

dist/products/jz47xx-glibc/ci20/ci20.SD.MBR
dist/products/jz47xx-glibc/ci20/ci20.ext4fs
dist/products/jz47xx-glibc/ci20/ci20.pkglist

The .SD.MBR suffix was selected for a reason. When the build system creates the target media's partition tables, it takes into account that most of the time SDHC cards will be used for which the Linux default is 4 reading heads and 16 sectors per track. At the same time, this assumption in no way limits the user's choice to use any other media.

In addition, the beginning of the filesystem on the target media is supposed to be offset by 1,048,576 bytes or 2048 sectors.

To write to the SDHC card linked to the /dev/mmcblk0 descriptor, it is enough to issue two commands:

# dd if=ci20.SD.MBR of=/dev/mmcblk0  bs=512 count=1
# dd if=ci20.ext4fs of=/dev/mmcblk0 obs=512k seek=2

The remaining card memory can be allocated for other partitions using the fdisk command.

In addition to the files explained above, the build system checks if the $(PRODUCTS_DEST_DIR) directory contains the $(HARDWARE).boot-records file that can hold the U-Boot loader.

If, at the time when the following command is issued:

$ make products_release

the file $(HARDWARE).boot-records exists, the build system puts it in the beginning of the partition table, and for the user the process of writing files to the SD card is boiled down to the following commands:

# cat ci20.boot-records ci20.ext4fs > SDHC.img
# dd if=SDHC.img of=/dev/mmcblk0 obs=512k

We will discuss boot records creation in the sections dedicated to various supported target devices. For now, the user is welcome to study this process on his/her own by examining the file boot/u-boot/ci20/2013.10-20150826/Makefile. A bit more complicated procedure including creation of FAT32 boot sector is defined in the file boot/u-boot/omap543x/2013.04-20140216/Makefile.

In conclusion, it should be said that the .ext4fs, .SD.MBR and .boot-records suffixes are critical for the build system.

Naming Convention

Toolchain names, device and device modification names must not include the underscore character ('_'). The only exceptions are toolchain names containing the x86_64 abbreviation, for example, x86_64-eglibc. Such names are processed by the build system correctly.

Toolchains

Toolchain is a set of application packages needed to compile and generate executable code from the source code. From this point onward we will discuss program sets intended for cross-compiling and code generation for various target platforms. These programs are used on personal computers (usually x86_64) and are created within the GNU project. Usually, a GNU Toolchain consists of a GCC cross-compiler, object code linker (included in the binutils package) and libraries intended for usage on the target machine. The main library required for every application is the standard C language library (the absence of this library makes every application unusable). This library must be a part of the toolchain and, in addition, must be installed in the target system. Here it is important to say that the target system must contain the same library that is linked to the applications created using this toolchain. There are two ways to ensure such compliance:

  1. Copying the library from the toolchain to the target system.
  2. Building a new library and using it in building all applications instead of the library present in the toolchain.

The latter is preferable in the context of ensuring integrity of the target system and its stable operation. Most embedded systems developers choose the first way which can be followed only in situations where the set and functionality of the target libraries included in the toolchain fully satisfy all software requirements. But if this is not the case, then developers have either to build a new toolchain containing the needed library set, or opt for the second way.

The second way implies the need to point the compiler to the directory where to search for libraries with which it shall link the applications it builds. This functionality is embedded in the GCC compiler and can be invoked either when the compiler itself is built or when it is used. In the first case the library directory is specified during configuration phase using the configure option of the scenario:

$ ./configure --with-sysroot=DIR

and in the second case the --sysroot directive shall be used:

$ gcc -c --sysroot=DIR -o main main.c

Suppose our toolchains are located in the /opt/toolchain directory. Then the full name for cross-gcc can be, for example, as follows:

/opt/toolchain/arm-OMAP543X-linux-eglibc/1.0.7/bin/arm-omap543x-linux-gnueabihf-gcc

In turn, path to the system directory which is the target machine's root filesystem's image and contains libraries will be like this:

/opt/toolchain/arm-OMAP543X-linux-eglibc/1.0.7/arm-omap543x-linux-gnueabihf/sys-root

Here it is important to note that the path /opt/toolchain/arm-OMAP543X-linux-eglibc/1.0.7 is arbitral, meaning that is was selected arbitrarily for storing the toolchain on the developer's machine. The directory name in blue type, arm-omap543x-linux-gnueabihf, is defined automatically and represents the option that was used to specify the target processor architecture at toolchain creation during the configuration phase:

$ ./configure ... --target=arm-omap543x-linux-gnueabihf ...

And, finally, the directory name in red type was selected by the build's author using the option:

$ ./configure \
     --with-sysroot=$(TOOLCHAIN_PATH)/$(TOOLCHAIN_VERSION)/$(TARGET)/sys-root

The information presented above is needed to move over to the next section where we will discuss how to connect new toolchains to the build system.

Connection of the New Toolchain to Build System

To connect a new toolchain to the build system, it is needed to add several lines to the file build-system/constants.mk. Let's study this process in details using an example.

In the beginning of build-system/constants.mk file the path to all toolchains used by the build system is defined:

TOOLCHAINS_BASE_PATH = /opt/toolchain

All connected toolchains must be residing in the same directory. Of course, to distinguish toolchains by targets and versions, users should employ relevant subdirectories. To provide information about actual toolchain location, the build system offers a set of variables that we will discuss later.

Since we are adding a new toolchain, we should probably already know the codenames of the target devices for which we are developing our software. Let's says it's the OMAP5 uEVM (Pandaboard 2) and DRA7XX EVM evm boards from Texas Instruments. This means that we need to define two new variables specifying the names of our machines:

HARDWARE_OMAP5UEVM = omap5uevm
HARDWARE_DRA7XXEVM = dra7xxevm

In addition, we should assign unique numerical identifiers that we will utilize during software development for our target devices.

OMAP5UEVM_ID_STD = 70
DRA7XXEVM_ID_STD = 71

Here it's important to note that when defining the names and values of the variables, we must ensure exact compliance to the following requirements:

The name of the target device must be equal to the 'HARDWARE_...' variable's suffix in lower case. In other words, the name parts highlighted in red must be equal and, translated to the lower case, must represent the device name highlighted in blue:

HARDWARE_OMAP5UEVM = omap5uevm
OMAP5UEVM_ID_STD = 70

Now it is possible to describe the toolchain itself that was assigned the codename omap543x-eglibc:

TOOLCHAIN_OMAP543X_EGLIBC    = omap543x-eglibc

OMAP543X_EGLIBC_ARCH         = arm-omap543x-linux-gnueabihf
OMAP543X_EGLIBC_VERSION      = 1.0.7
OMAP543X_EGLIBC_DIR          = arm-OMAP543X-linux-eglibc
OMAP543X_EGLIBC_PATH         = $(TOOLCHAINS_BASE_PATH)/$(OMAP543X_EGLIBC_DIR)

OMAP543X_EGLIBC_ARCH_DEFS    = -D__OMAP543X__=1
OMAP543X_EGLIBC_ARCH_FLAGS   = -march=armv7-a -mtune=cortex-a15     \
                               -mfloat-abi=hard  -mfpu=neon-vfpv4   \
                               -mabi=aapcs-linux -fomit-frame-pointer

OMAP543X_EGLIBC_SYSROOT      = sys-root
OMAP543X_EGLIBC_DEST_SYSROOT = yes

OMAP543X_EGLIBC_HARDWARE_VARIANTS := $(HARDWARE_OMAP5UEVM) \
                                     $(HARDWARE_DRA7XXEVM)

Note that here the same variable naming convention is applied that we followed during target device name definition, but with one additional rule: conversion to lower case changes the underscore character '_' into dash '-', and vice versa. Speaking the Linux commands language, this transition from lower case to upper case can be carried out like this:

$ OMAP543X_EGLIBC=`echo omap543x-eglibc | tr '[a-z-]' '[A-Z_]'`

Let's take a closer looks at the variables and their values. First of all, the user-defined toolchain's code name is omap543x-eglibc. This name must be short but meaningful. The current name tells us that this toolchain is used to build software for the device line based on SoCs OMAP5430 and OMAP5432 from Texas Instruments. In addition, target software is built using EGLIBC [www.eglibc.org] – a revised version of GNU Glibc intended especially for embedded devices. And, finally, the fact that the target system is an operating system based on Linux kernel can be understood from the context.

The OMAP543X_EGLIBC_VERSION variable having the value of 1.0.7 defines the toolchain version and, at the same time, the directory name where the toolchain is located. The build system uses this variable to define the cross-compiler's full name or, in other words, the path to the cross-compiler. The user doesn't need to use this variable explicitly. Variables OMAP543X_EGLIBC_DIR, OMAP543X_EGLIBC_PATH are defined by the user in accordance to the actual location of the toolchain on the developer's machine or build server. The variable OMAP543X_EGLIBC_SYSROOT defines the name of the final subdirectory of the system root directory path that was specified at toolchain creating using the configuration scenario's --with-sysroot option. The value of the OMAP543X_EGLIBC_ARCH variable must be equal to the --target directive's value that was used to create the cross-compiler. In our case, the target architecture is defined by the string arm-omap543x-linux-gnueabihf.

The following figure explains how these variables are used.

TOOLCHAINS_BASE_PATH OMAP543X_EGLIBC_SYSROOT
OMAP543X_EGLIBC_VERSION
OMAP543X_EGLIBC_DIR
OMAP543X_EGLIBC_ARCH
/opt/toolchain /arm-OMAP543X-linux-eglibc /1.0.7 /arm-omap543x-linux-gnueabihf /sys-root
TOOLCHAIN_PATH TARGET
Fig.2. Toolchain's System Root

On figure 2, variables defined by the user in the build-system/constants.mk file are highlighted in blue. Variables TOOLCHAIN_PATH, TARGET are detected by the build system automatically in accordance to the current target and can be used in user Makefiles. Later we will discuss in more details how to use these variables.

Variables OMAP543X_EGLIBC_ARCH_DEFS, OMAP543X_EGLIBC_ARCH_FLAGS are used to give additional directives to the cross-compiler when it is used as part of the build system. Thus, the variable OMAP543X_EGLIBC_ARCH_FLAGS allows specifying the architecture of the target device's processor. Note that directives set using the OMAP543X_EGLIBC_ARCH_FLAGS variable are the same as those that were used to build the cross-compiler itself.

The OMAP543X_EGLIBC_DEST_SYSROOT variable should be explained separately. If its value is yes, then cross-compiler will be called with the --sysroot=DIR directive, where the directory that DIR represents is not the toolchain's system directory, but so-called TARGET_DEST_DIR path. It defines the location of the target directory that is used as a temporary system environment when any application is built and that represents the future target system. This way, software configuration and building for the target will be done using the header and object files representing the current target system's state, and the built programs will be linked to these files.

Of course, such reassignment of the system directory makes sense only for building user applications and is not required for building kernel, kernel modules and system boot loaders, because such system software is built without the use of other libraries. To build target software with variable OMAP543X_EGLIBC_DEST_SYSROOT set to yes, but without the --sysroot=DIR directive, the user can cancel such reassignment in a single Makefile by defining there the USE_TARGET_DEST_DIR_SYSROOT variable with a value different from yes. For example,

USE_TARGET_DEST_DIR_SYSROOT = no

Of course, it should be done before build system's files are included, so that this value can be read at the right time.

If the OMAP543X_EGLIBC_DEST_SYSROOT variable has the value of no,

OMAP543X_EGLIBC_DEST_SYSROOT = no

then the --sysroot=DIR directive is not used. This behavior is typical for building software for systems based on microcontrollers using libraries from the toolchain, or without any system libraries at all.

And, finally, the OMAP543X_EGLIBC_HARDWARE_VARIANTS variable is used to define the list of devices for which software will be built using this cross-compiler.

We will conclude this section with the full list of variables that we defined in the build-system/constants.mk file for connecting a new toolchain:

HARDWARE_OMAP5UEVM = omap5uevm
HARDWARE_DRA7XXEVM = dra7xxevm

OMAP5UEVM_ID_STD = 70
DRA7XXEVM_ID_STD = 71

TOOLCHAIN_OMAP543X_EGLIBC    = omap543x-eglibc

OMAP543X_EGLIBC_ARCH         = arm-omap543x-linux-gnueabihf
OMAP543X_EGLIBC_VERSION      = 1.0.7
OMAP543X_EGLIBC_DIR          = arm-OMAP543X-linux-eglibc
OMAP543X_EGLIBC_PATH         = $(TOOLCHAINS_BASE_PATH)/$(OMAP543X_EGLIBC_DIR)

OMAP543X_EGLIBC_ARCH_DEFS    = -D__OMAP543X__=1
OMAP543X_EGLIBC_ARCH_FLAGS   = -march=armv7-a -mtune=cortex-a15     \
                               -mfloat-abi=hard  -mfpu=neon-vfpv4   \
                               -mabi=aapcs-linux -fomit-frame-pointer

OMAP543X_EGLIBC_SYSROOT      = sys-root
OMAP543X_EGLIBC_DEST_SYSROOT = yes

OMAP543X_EGLIBC_HARDWARE_VARIANTS := $(HARDWARE_OMAP5UEVM) \
                                     $(HARDWARE_DRA7XXEVM)

Using Hardware IDs in source code

While developing own programs, sometimes the need arises to modify a certain part of the code in accordance with the architecture of one or another target device. Suppose the main part of the program is independent and can be executed on all devices that we defined above in the build-system/constants.mk file. But some amount of code is device-dependent and, therefore, the engineer needs to know for which device the program is currently being built. For this, the build system provides macro definitions that are created automatically and can be fed to the compiler. So, the first macro definition that is created for all devices is the __HARDWARE__ variable. This variable contains the value of the current device identifier for which the code is currently being compiled. In addition, for all devices declared in the build-system/constants.mk file, variables are created having the names equivalent to the device names, but in upper case. This way, in our case:

HARDWARE_OMAP5UEVM = omap5uevm
HARDWARE_DRA7XXEVM = dra7xxevm

OMAP5UEVM_ID_STD = 70
DRA7XXEVM_ID_STD = 71

three variables will be defined, and the compiler will be called with the following directives:

$ gcc -D__HARDWARE__=70 -DOMAP5UEVM=70 -DDRA7XXEVM=71

for the omap5uevm device, and with the following directives:

$ gcc -D__HARDWARE__=71 -DOMAP5UEVM=70 -DDRA7XXEVM=71

for the dra7xxevm device.

Using these variables, the engineer can implement tests in their own code, for example, like in the following listing.

/* HW independent code */

#if __HARDWARE__ == OMAP5UEVM

/* OMAP5UEVM specific code */

#elif __HARDWARE__ == DRA7XXEVM

/* DRA7XXEVM specific code */

#endif

Using the OMAP543X_EGLIBC_ARCH_DEFS variable, the build system's user can define any variables. But it is important to remember that user-defined variables must not have the same names as predefined macros created by the compiler itself or by other built components.

To get the list of predefined constants, the user can run the following command:

$ gcc -dM -E -x c /dev/null

Here, the -dM directive starts printing the predefined macro definitions, the -E directive triggers printing to the standard output stream and, finally, -x c selects the C language.

CCACHE

The build system uses the CCACHE(1) variable to shorten the build time. In the build-system/constants.mk file the variable

CACHED_CC_OUTPUT = /opt/extra/ccache

is defined holding the working directory of the ccache utility as the value. This directory contains the cashed output of cross-compilers that are involved in targets building.

To output the performance statistics of the build system, the user should run the make ccache_stats command in any directory of the tree.

To set the maximum cache size, the following command shall be used:

$ CACHE_DIR=/opt/extra/ccache ccache -M 64G

where the size can be specified in kilobytes, megabytes and gigabytes, using the K, M and G suffixes, correspondingly.

Versioning

Build system version are numbered using the common approach, where the version number consists of three values: major.minor.maintenance.

In the Download section where we discuss getting the build system source code, different ways of downloading certain product versions are explained. Usually, users are interested in fixed revisions labeled with tags.

Before using a certain build system version, the user can check which devices are supported there and which toolchain versions are involved. To do this, it's enough to examine the file build-system/constants.mk, for example, the constants.mk file corresponding to the tags/build-system-1.1.5 tag.

On the other hand, if we are talking about integrating the build system in the software development process, then sometimes it is not enough to simply use a fixed version. Therefore, a more deep understanding of the build system's repository evolution is required.

Continuous Branches

Common development process implies continuous evolution of a trunk and ensuring its stability. Usually, development itself is done in separate branches. After some new functionality has been developed, the changes are uploaded to the trunk. To fix stable product revisions, tags are created, which are copies of the trunk as of certain points of time. The following diagram (figure 3) illustrates how tags are created in a trunk until the time point Tn, provided that the standard product configuration management policy is applied.

Browser is currently unable to display SVG images.
Fig.3. Continuous Branches

Suppose that at the point of time Tn the decision was made to include new versions of toolchains in the build system or to implement new devices support. After this, the product minor version shall be increased.

If such changes go directly to the trunk, then the old system's functionality becomes history and all products depending on the current set of toolchains and supported devices either need to be rebuilt or the developer should continue to maintain them on a fixed revision of the build system trunk. The first option is unacceptable because it can lead to a very large amount of changes in all product lines and, therefore, calls for difficult decisions on a higher level. The second option implies inability to improve the build system within the current configuration, which limits maintenance possibilities for product lines based on an old set of toolchains and supported devices.

The correct solution here would be to create a new branch, build-system-1.1.x, that will preserve the current build system functionality and ensure possibility to further improve it on the fixed set of toolchains and supported devices.

Until the build-system-1.2.x branch is created, build-system-1.1.x remains a common branch, which means that all modification made within this branch can be uploaded to the trunk, and tags fixing the trunk state can be created as usual.

At the point of time Tn+1 the branch build-system-1.1.x will be transferred to so called CONTINUOUS state that has a special feature: no changes implemented in this branch after the state change can be uploaded to the trunk. Tags fixing the development states will now be created as direct copies of the build-system-1.1.x branch. On figure 3 it is illustrated by the tag 1.1.7.

The branch build-system-1.2.x created at the point of time Tn+1 will also go to the CONTINUOUS state in future, but only when the need arises to upload new key changes to the trunk. For example, when the build-system-1.3.x or build-system-2.x.x branch is created. Here the x variables illustrate variability of the corresponding build system version number's component.