Sources

Создание дистрибутивов связано с использованием большого количества исходных пакетов сторонних производителей. Хранятся исходные пакеты, в глобальной сети, на множестве ресурсов, таких, например, как sourceforge.net, github.com и savannah.gnu.org. Большие проекты, такие как GNU, X.org, KDE, Gnome и другие, имеют собственные сетевые ресурсы.

В основном, исходные пакеты предоставляются в виде архивов, однако, часто возникает ситуация, когда необходимо воспользоваться определенной версией исходных текстов, взятых напрямую из репозитория стороннего производителя.

Однако главной проблемой, для создателя дистрибутива, является не поиск исходных пакетов в глобальной сети, а скорость их загрузки перед началом сборки.

На нашем сервере организовано зеркалирование ключевых Git-репозиториев и хранение архивов исходных пакетов для того, чтобы ускорить загрузку исходных пакетов перед началом сборки Radix.pro платформы. Зеркала Git-репозиториев расположены по адресу http://git.radix.pro.

Архивы сторонних производителей, необходимых для сборки платформы, можно найти на FTP-сервере в каталоге ftp://ftp.radix.pro/sources.

Фактически, каталог ftp://ftp.radix.pro/sources представляет собой клон репозитория http://git.radix.pro/sources.git, который обновляется по мере необходимости и доступен для клонирования посредством, команды

$ git clone http://git.radix.pro/sources.git

Создание зеркалированного хранения исходных пакетов позволяет весьма существенно сократить время загрузки исходных пакетов перед сборкой. Например, если FTP-сервер находится в собственной локальной сети, то загрузка всех, необходимых платформе, исходных архивов занимает около пяти минут. С другой стороны, если создать клон репозитория http://git.radix.pro/sources.git и попытаться выполнить загрузку пакетов из глобальной сети путем выполнения команд

$ git clone http://git.radix.pro/sources.git sources
$ cd sources
$ make

то такая загрузка будет длиться несколько суток.

Система сборки по умолчанию использует FTP-сервер ftp://ftp.radix.pro. Однако пользователи могут самостоятельно организовать зеркалированное хранение исходных пакетов на собственном сервере и изменить значение переменной DOWNLOAD_SERVER в файле 'build-system/constants.mk'.

Toolchains

Система сборки поддерживаемых toolchain-ов хранится в SVN-репозитории http://svn.radix.pro/svn/toolchains и доступна для клонирования посредством команды

$ svn co http://svn.radix.pro/svn/toolchains/trunk

Релизы, фиксированные тегами, можно получить на нашем FTP-сервере ftp://ftp.radix.pro/toolchains/x86_64 в каталоге, соответствующем номеру версии.

В разделе Build System in Practice, приведен пример загрузки toolchain-а для платы Firefly-RK3288.

Build System

The repository of the build system sources is placed at http://svn.radix.pro/svn/build-system.

The latest stable release of the build system is available in the ftp://ftp.radix.pro/radix/build-system/releases directory.

Another way to obtain the latest version of the build system is checking out an appropriate tag of repository using following command

$ svn co http://svn.radix.pro/svn/build-system/tags/build-system-1.2.2 build-system

In the section Build System in Practice there is an example of downloading the archive from the FTP-server.

Git-mirror Creation

In the section Versioning we talked about the build system integration into your own process and also discussed some issues related to CM (Configuration Management) policy. Given the review of CM policies, users can integrate the build system in their own projects, which are supported by a variety of source code management systems.

As an example, let us consider the creation of Git-mirror of the Radix.pro build system repository. Such mirroring can provide better integration of the build system into user's projects and also helps to automate the process of obtaining updates.

To create a Git-mirror we have to run following command

$ git svn clone \
     --no-metadata           \
     --branches=/branches    \
     --tags=/tags            \
     --trunk=/trunk          \
     --preserve-empty-dirs   \
     --placeholder-filename=README \
     http://svn.radix.pro/svn/build-system \
     -s build-system

After creating the mirror we have to copy needed tags to directory which is specifically provided by Git system. During this process we can to remov unnecessary tags. This can be done using the following commands

$ cd build-system/.git/refs/remotes/origin/tags
$ cp build-system-?.?.? ../../../tags

To bring the tag names to the form that is most acceptable to the Git version control system you can make use the following script writen in Perl:

#!/usr/bin/perl

use FindBin;
use lib $FindBin::Bin;

use strict;
use warnings FATAL => 'all';

use IO::Handle;
use File::Basename;
use File::Copy qw(copy);

my $program = basename( $0 );

my $dirname = ".";

opendir( DIR, "$dirname" ) or
    die( "Copy Tags: Could not open directory: $dirname: $!" );
  my @tags = grep { -f "$_" } readdir( DIR );
closedir DIR;

foreach my $tag (@tags)
{
  if( $tag =~ /^build\-system\-([0-9]+\.[0-9]+\.[0-9]+)/ )
  {
    my $target = $1;
    print "Copy: $tag --> ../../../$target\n";
    copy( $tag, "../../../tags/$target" );
  }

  if( ( $tag !~ /^build\-system\-([0-9]+\.[0-9]+\.[0-9]+)/ ) and ( $tag !~ /$program/) )
  {
    print "Bad tag name: $tag -- Have to be removed.\n";
    unlink $tag;
  }
}

This script copies the original tags into files which names are equal to digital part of the original tag name.

To use this script, it must be copied to the directory build-system/.git/refs/remotes/origin/tags:

$ cp copy-tags.pl build-system/.git/refs/remotes/origin/tags/
$ cd build-system/.git/refs/remotes/origin/tags/
$ ./copy-tags.pl
$ rm -f ./copy-tags.pl

In addition after creating a mirror it is desirable to remove the branches which are mainly located in the deadwood directory of the SVN repository because they will not needed in the further work. Since in the Git understanding the branch is a file with hash-sum placed in a suitable directory this deletion of unnecessary branches can be done, for example, as follows.

$ cd build-system/.git/refs/remotes/origin
$ rm -f BS-external deadwood genext4fs

You can also remove unneded branches and tags in the build-system/.git/logs/refs/remotes/origin directory.

Now we can show the list of branches

$ git branch -a
* master
  remotes/origin/build-system-1.1.x
  remotes/origin/build-system-1.2.x
  remotes/origin/tags/build-system-0.0.1
  remotes/origin/tags/build-system-1.0.0

     .
     .
     .

  remotes/origin/tags/build-system-1.1.5
  remotes/origin/tags/build-system-1.1.6
  remotes/origin/tags/build-system-1.1.7
  remotes/origin/trunk

as well as to refer to the tags

$ git tag
0.0.1
1.0.0

  .
  .
  .

1.1.5
1.1.6
1.1.7

$

Note here that we leave SVN tags in the directory build-system/.git/refs/remotes/origin/tags and they presented as ordinary branches of the origin SVN repository.

In order to monitor only needed original SVN branches we can edit the build-system/.git/config file which at this moment looks like as shown in the following listing.

[core]
       repositoryformatversion = 0
       filemode = true
       bare = false
       logallrefupdates = true
[svn-remote "svn"]
       noMetadata = 1
       preserve-empty-dirs = true
       placeholder-filename = README
       url = http://svn.radix.pro/svn/build-system
       fetch = trunk:refs/remotes/origin/trunk
       branches = branches/*:refs/remotes/origin/*
       tags = tags/*:refs/remotes/origin/tags/*
       added-placeholder = doc/README
       added-placeholder = branches/BS-external/3pp/sources/packages/dialog/patches/README
       added-placeholder = pkgtool/README
       added-placeholder = 3pp/README
       added-placeholder = branches/deadwood/genext4fs/3pp/sources/packages/dialog/patches/README
       added-placeholder = branches/deadwood/BS-external/3pp/sources/packages/dialog/patches/README

Let's remove lines related to unneeded branches:

       added-placeholder = branches/BS-external/3pp/sources/packages/dialog/patches/README
       added-placeholder = branches/deadwood/genext4fs/3pp/sources/packages/dialog/patches/README
       added-placeholder = branches/deadwood/BS-external/3pp/sources/packages/dialog/patches/README

and also comment out the lines that responsible for updating the trunk and other branches:

#      fetch = trunk:refs/remotes/origin/trunk
#      branches = branches/*:refs/remotes/origin/*

leaving only updating the tags:

       tags = tags/*:refs/remotes/origin/tags/*

Thus we get the following contents of the build-system/.git/config file.

[core]
       repositoryformatversion = 0
       filemode = true
       bare = false
       logallrefupdates = true
[svn-remote "svn"]
       noMetadata = 1
       preserve-empty-dirs = true
       placeholder-filename = README
       url = http://svn.radix.pro/svn/build-system
#      fetch = trunk:refs/remotes/origin/trunk
#      branches = branches/*:refs/remotes/origin/*
       tags = tags/*:refs/remotes/origin/tags/*
       added-placeholder = doc/README
       added-placeholder = pkgtool/README
       added-placeholder = 3pp/README

Now we can add only branches which we will monitor:

[svn-remote "trunk"]
       url = http://svn.radix.pro/svn/build-system/trunk
       fetch = :refs/remotes/origin/trunk
[svn-remote "build-system-1.1.x"]
       url = http://svn.radix.pro/svn/build-system/branches/build-system-1.1.x
       fetch = :refs/remotes/origin/build-system-1.1.x
[svn-remote "build-system-1.2.x"]
       url = http://svn.radix.pro/svn/build-system/branches/build-system-1.2.x
       fetch = :refs/remotes/origin/build-system-1.2.x

Note that it is possible to maintain, if necessary, the directory structure that correspond the SVN capabilities in the refs/remotes/origin directory, for example, in the following manner:

[svn-remote "build-system-1.0.x"]
       url = http://svn.radix.pro/svn/build-system/branches/deadwood/build-system-1.0.x
       fetch = :refs/remotes/origin/deadwood/build-system-1.0.x

Thus, the resulting Git repository configuration file will look like this:

[core]
       repositoryformatversion = 0
       filemode = true
       bare = false
       logallrefupdates = true
[svn-remote "svn"]
       noMetadata = 1
       preserve-empty-dirs = true
       placeholder-filename = README
       url = http://svn.radix.pro/svn/build-system
       tags = tags/*:refs/remotes/origin/tags/*
       added-placeholder = doc/README
       added-placeholder = pkgtool/README
       added-placeholder = 3pp/README
[svn-remote "trunk"]
       url = http://svn.radix.pro/svn/build-system/trunk
       fetch = :refs/remotes/origin/trunk
[svn-remote "build-system-1.1.x"]
       url = http://svn.radix.pro/svn/build-system/branches/build-system-1.1.x
       fetch = :refs/remotes/origin/build-system-1.1.x
[svn-remote "build-system-1.2.x"]
       url = http://svn.radix.pro/svn/build-system/branches/build-system-1.2.x
       fetch = :refs/remotes/origin/build-system-1.2.x

To add new SVN branch for tracking you can make use direct editing the build-system/.git/config file or Git commands. For example, to add the build-system/branches/build-system-1.3.x branch from origin SVN repository to the Git repository you can add the folloving lines into build-system/.git/config file:

[svn-remote "build-system-1.3.x"]
       url = http://svn.radix.pro/svn/build-system/branches/build-system-1.3.x
       fetch = :refs/remotes/origin/build-system-1.3.x

or run following commands:

$ git config --add svn-remote.build-system-1.3.x.url \
                   http://svn.radix.pro/svn/build-system/branches/build-system-1.3.x
$ git config --add svn-remote.build-system-1.3.x.fetch :refs/remotes/origin/build-system-1.3.x

To work with created Git repository, we have to create local branches. This can be done with the following commands:

$ git checkout -b build-system-1.1.x origin/build-system-1.1.x
$ git checkout -b build-system-1.2.x origin/build-system-1.2.x

To obtain the recent changes of the SVN repository of the build system we have to run git-fetch command, for example, as follows.

$ cd build-system
$ git svn fetch -q --all

To update the status of the local branches according to the entered changes from the SVN repository, it is necessary to use git-merge command:

$ git svn fetch -q --all

$ git checkout build-system-1.1.x
$ git merge origin/build-system-1.1.x

$ git checkout build-system-1.2.x
$ git merge origin/build-system-1.2.x

$ git checkout master
$ git merge origin/trunk

If we use this Git repository to create a mirror, which will be a so-called bare repository containing only copies of the local branches then all our already performed actions can be issued as a separate script which can be run by cron-daemon for periodically updating our Git repository which in turn becomes a intermediate mirror. The bare repository mentioned above we can create with the command:

$ git clone --mirror build-system build-system.git

After creating this mirror we also can to remove unnecessary references to origin SVN branches by deleting following lines from the build-system.git/packed-refs file:

93611bc9e765a5878795792e463302dad6addd2e refs/remotes/origin/build-system-1.1.x
1336179d6627d1492470b35c07054b8ad7c1cb99 refs/remotes/origin/build-system-1.2.x
bb06af52607d55098b1bded3b2fa6d88078d1ff8 refs/remotes/origin/tags/build-system-0.0.1
b60ca9986f939a14bc6575879d7dd2460a8edc58 refs/remotes/origin/tags/build-system-1.0.0

     .
     .
     .

3cb4bddcd0069da5c45391608cdcc035cf596ace refs/remotes/origin/tags/build-system-1.1.5
9f9981f58890a7a8d61131bc32e58bf6518ca155 refs/remotes/origin/tags/build-system-1.1.6
49c8ed507ed3421f85e922b67b167d804e370518 refs/remotes/origin/tags/build-system-1.1.7
eebc0d630b8b0859c33af6965468ce52900e3023 refs/remotes/origin/trunk

Thus, by updating the contents of the repository build-system.git through intermediate (repository build-system) mirror, you can either to organize the work using only local branches of the Git repository, or even to move your development from SVN to Git with preserving the history.

In the future using the Git-mirror the user will need to take care of connecting the build system mirror to their own projects and, most importantly, think about selecting relevant branches or tags according to the information from the Continuous Branches section.