Subversion Repositories DevTools

Notes on the blat package transfer system

-----------------------------------------

Reason for its creation

-----------------------

Need to transfer packages from dpkg_archive to remote sites in a timely manner.

Rsync was considered but it has several problems:

1) Does not handle symlinks in a suitable manner

2) Works with all the files in the repository. Experience has

   shown that this can be very slow

3) Still requires significant scripting in order to be useful

Blat can make several assumptions about the package system.

Blat will:

    Support multiple transfer target destinations

    Allow for rapid detection of new packages that need to be transferred

    Allow for multiple Releases to be synchronized

    Allow for all (not-closed) releases in a Project to be synchronized

    Easily configured - and can be configured on the fly

    Atomically transfer packages

    Transfer a PackageList for future cleanup operations

    Logging and debug facilities

Overview of Blat

---------------

There are two main components in Blat

    Daemon supervisor

        Responsible for start and restarting configured daemons

    Transfer Daemons

        Responsible for the package sync operations for one target

        Multiple Daemons ( targets ) are supported

Each Blat Daemon performs three main operations

    1) Fast package transfer

    2) Repository synchronization

    3) PackageList creation

Fast package transfer

===============================

This is mechanism whereby Blat will detect the need to transfer a newly built

package to the target system.

It works by monitoring a directory of tags. It is the responsibility of Release

Manager to populate the directory.

The responsiveness of the detection can be configured, but a period of 5

seconds is suggested.

Repository synchronization

===============================

The daemon will request a list of packages that are present on the target and

determine the list of packages that should be on the target. Discrepancies will

be transferred to the target. Excess packages are left on the target.

Blat will request the target to create and transfer a list of packages.

This is done by invoking a small program on the target to perform the work.

Blat will interrogate the Release Manager database for Releases to be processed

and packages in those Releases.

A package will be transferred to the target if:

    * The package is required, but not present on the target

    * The time-stamps of the descpkg files differ

Package transfer may be delayed if the source package is writable, unless it

has been writable for longer than a configured time period.

The frequency of the Repository synchronization can be configured. A time of

several hours is suggested.

PackageList creation

===============================

Blat will create and send to the target a list of package-version that are

in the current set. This list may be used to clean out the package archive,

but this functionality has not yet been implemented.

Host System Requirements

========================

1) Unix

   It has been designed for a Unix environment - not Windows

2) Perl

   Blat is written in Perl

3) Java

   Required for the Database interface

4) Shell

   Start and stop scripts are in shell

5) Utilities

    ssh

    gtar

    gzip

Target System Requirements

========================

1) Unix

   It has been designed for a Unix environment - not Windows

2) Perl

3) Shell

   Blat will execute a number of scripts on the target inn order

   to control the process. These are in shell and perl

4) Utilities

    ssh

    gtar

    gunzip

5) User with write access to the dpkg_archive

6) Link for the users home directory to the package archive

   This link is called 'dpkg_archive'

Shared requirements

===================

Blat uses ssh for the transfer process. It uses an 'identity' file to allow

passwordless authentication with the target. The public part of the identify

file must be appended to the target users .ssh/authorized_keys file.

The private part of the identity file is held by the Blat Daemon.

Design assumptions

================================================================================

Blat is designed to transfer dpkg_archive packages in one direction.

Blat makes assumptions on the structure of a package

    - They contain a descpkg file

    - They are read-only when fully released

    - The contents of packages does not change

    - It is not necessary to check every file in the package

The Blat master is designed to run in a single directory tree.

The config file should be in a 'config' directory under the location

of the blat master program.

Installation :: Target System

=============================

1) Create or aquire a user that has write access to the packaeg archive

2) Create or aquire a passwordless identity file and associated public key

   of the identity file. One set is available in the 'ssh' subdirectory.

   Append the public part of the idenity file (id_rsa_pkg_admin.pub) to

   ~/.ssh/authorized_keys

3) Create a link from the users home directory to dpkg_archive

   The must be called dpkg_archive

4) Transfer the blat receiver scripts to a directory accessable to the

   transfer user. ie: ~/bin

   The reqired receiver files are:

        get_plist.pl

        receive_file

        receive_package

        delete_package

   Ensure the programs are execuatable by the transfer user.

Installation :: Host System

=============================

This section really deals with the configuration of a new target.

1) Create a new config file in Blat's config directory - with a .conf

   suffix. This is best done by cloning an existing entry.

   Note: The blat master will automatically spawn a daemon as soon

   as a new config file is seen. Its best to create the file elsewhere

   and copy it to the directory when ready.

   Note: The Blat daemon will detect changes to its own config file and

   re-read it on the fly.

Useful Tricks

=============

kill -usr1 pid-of-daemon

    Will force the daemon to perform a repository sync check.

kill -hup pid-of-daemon

    Will force the daemon to roll its own log files

Debug verbosity is controlled via the 'verbose' config item

The pkg.xxxx config items are very special.

If the named package-versio is a symlink, then both the

link and the liink the package addresses will be transferred.

The link MUST address another version of the same package.

This is intended to support the 'jats2_current' link.

When a new version of JATS is released, then the new package

will be transferred, as will the new link.

Config items that control a time period allow the following sufixes:

    s - Seconds. Same as no suffix

    m - Minutes

    h - Hours

    d - Days

Multiple are allowed. ie: 1h10h

Config items that control a file size in blocks allow the following suffixes:

    k - Kilobytes (Same as no suffix)

    b - Blocks    (Same as no suffix)

    m - Megabytes

    g - Gigabytes

ToDo

======================

1) Purging packages on target

   Can set deletePackages to delete excess packages when the release sync

   is performed.

2) Better handling of soft-links for core_devl

   Works, but its prone to error

   There is no test to ensure the link exists. If the link

   is deleted, then it won't be recreated.