Subversion Repositories DevTools

Notes on the blat package transfer system

-----------------------------------------

Reason for its creation

-----------------------

Need to transfer packages from dpkg_archive to remote sites in a timely manner.

Rsync was considered but it has several problems:

1) Does not handle symlinks in a suitable manner

2) Works with all the files in the repository. Experience has

   shown that this can be very slow

3) Still requires significant scripting in order to be useful

Blat can make several assumptions about the package system.

Blat will:

    Support multiple transfer target destinations

    Allow for rapid detection of new packages that need to be transferred

    Allow for multiple Releases to be synchronized

    Allow for all (not-closed) releases in a Project to be synchronized

    Easily configured - and can be configured on the fly

    Atomically transfer packages

    Transfer a PackageList for future cleanup operations

    Logging and debug facilities

Overview of Blat

---------------

There are three main components in Blat

    Daemon supervisor

        Responsible for start and restarting configured daemons

    Transfer Daemons

        Responsible for the package sync operations for one target

        Multiple Daemons ( targets ) are supported

        Multiple Daemon types are supported

            dpkg_archive sync (original)

            s3Sync (AWS S3 bucket sync for CI/CD)

    On Target utilities

        A set of scripts that support Blat

        These are transferred to the target machine.

Each Blat Daemon performs three main operations

    1) Fast package transfer

    2) Repository synchronization

    3) PackageList creation

    4) Package aging (Optional)

Each Blat target can perform the following:

    1) Package aging

    2) dpkg_archive content indexing

Fast package transfer

===============================

This is mechanism whereby Blat will detect the need to transfer a newly built

package to the target system.

It works by monitoring a directory of tags. It is the responsibility of Release

Manager to populate the directory.

The responsiveness of the detection can be configured, but a period of 5

seconds is suggested.

Repository synchronization

===============================

The daemon will request a list of packages that are present on the target and

determine the list of packages that should be on the target. Discrepancies will

be transferred to the target. Excess packages are left on the target.

Blat will request the target to create and transfer a list of packages.

This is done by invoking a small program on the target to perform the work.

Blat will interrogate the Release Manager database for Releases to be processed

and packages in those Releases.

A package will be transferred to the target if:

    * The package is required, but not present on the target

    * The time-stamps of the descpkg files differ

Package transfer may be delayed if the source package is writable, unless it

has been writable for longer than a configured time period.

The frequency of the Repository synchronization can be configured. A time of

several hours is suggested.

PackageList creation

===============================

Blat will create and send to the target a list of package-version that are

in the current set. This list may be used to clean out the package archive,

but this functionality has not yet been implemented.

Package aging

=============

Blat can be configured to delete packages that are no longer a part of the

current package-version set. There are 4 methods:

1) None

   Packages will never be deleted by Blat on the target.

   The target file system will need to be managed to prevent it filling up.

2) Immediate

   Packages will be deleted as soon as they are not a part of the current

   package-version set.

3) Aged by blat master

   Packages will be marked for deletion and the blat master will delete

   the packages after a configured number of days.

4) Aged by blat target

   Packages will be marked for deletion and the blat target will delete

   the packages after a configured number of days. This operation requires

   that a cron job be configured on the target machine.

dpkg_archive content indexing

=============================

Blat provides a utility that can be run by the transfer target, as a cron job,

that will maintain a list of files and folders in the package archive.

This list greatly simplifies the process of locating a file in the archive.

The user simply greps the package list, rather than search the directory tree.

The file list is in a file .../dpkg_archive/.dpkg_archive/dpkg_archive_list.txt

S3 Bucket Delivery

===============================

Blat has been extended to provide CI/CD support via an S3 bucket

The s3Sync task will maintain a single S3 bucket with ZIP files of 

packages from  Releases that support S3Sync

Host System Requirements

========================

1) Unix

   It has been designed for a Unix environment - not Windows

2) Perl

   Blat is written in Perl

3) Java

   Required for the Database interface

4) Shell

   Start and stop scripts are in shell

5) Utilities

    ssh

    gtar

    gzip

    aws cli (for s3Sync)

Target System Requirements - dpkg_archive sync

========================

1) Unix

   It has been designed for a Unix environment - not Windows

2) Perl

3) Shell

   Blat will execute a number of scripts on the target in order

   to control the process. These are in Shell and Perl

4) Utilities

    ssh

    gtar

    gunzip

5) User with write access to the dpkg_archive - (pkgadmin)

6) Link for the users home directory to the package archive

   This link is called 'dpkg_archive'

Shared requirements

===================

Blat uses ssh for the transfer process. It uses an 'identity' file to allow

passwordless authentication with the target. The public part of the identify

file must be appended to the target users .ssh/authorized_keys file.

The private part of the identity file is held by the Blat Daemon.

Design assumptions

================================================================================

Blat is designed to transfer dpkg_archive packages in one direction.

Blat makes assumptions on the structure of a package

    - They contain a descpkg file

    - They are read-only when fully released

    - The contents of packages does not change

    - It is not necessary to check every file in the package

The Blat master is designed to run in a single directory tree.

The config file should be in a 'config' directory under the location

of the blat master program.

Installation :: Target System

=============================

1) Create or acquire a user that has write access to the package archive

2) Create or acquire a passwordless identity file and associated public key

   of the identity file. One set is available in the 'ssh' subdirectory.

   Append the public part of the identity file (id_rsa_pkg_admin.pub) to

   ~/.ssh/authorized_keys

   I suggest using 'ssh-copy-id'.

3) Create a link from the users home directory to dpkg_archive

   The must be called dpkg_archive

4) Transfer the blat receiver scripts to a directory accessible to the

   transfer user. ie: ~/bin

   The required receiver files are:

        get_plist.pl

        receive_file

        receive_package

        delete_package

        pkg_mon.pl

        pkg_purge.pl

   Ensure the programs are executable by the transfer user.

   Only get_plist.pl is really needed. The others will be transferred

   when detected missing. 

5) Set up cron jobs (optional)

   Will be used to maintain package information

   Suggest crontab entry - may vary for each installation

Installation :: Host System

=============================

This section really deals with the configuration of a new target.

1) Create a new config file in Blat's config directory - with a .conf

   suffix. This is best done by cloning an existing entry.

   Note: The blat master will automatically spawn a daemon as soon

   as a new config file is seen. Its best to create the file elsewhere

   and copy it to the directory when ready.

   Note: The Blat daemon will detect changes to its own config file and

   re-read it on the fly.

Useful Tricks

=============

kill -usr1 pid-of-daemon

    Will force the daemon to perform a repository sync check.

kill -hup pid-of-daemon

    Will force the daemon to roll its own log files

kill pid-of-daemon

    Will force the daemon to exit. It will be restarted.

Remove the daemons pid file

    Will force the daemon to exit. It will be restarted.

    Useful for debugging on a live system

kill -usr1 pid_of_master

    Will signal -usr1 to all daemons

    Will force all daemons to perform a repository sync check.

kill -hup pid_of_master

    Will signal -hup to all daemons

    Will force all daemon to roll their own log files

kill pid_of_master

    Will shut down system gracefully by sending kill to all

    children.

ssh-to <name or ip address>

    Will ssh to the target machine as the pkgadmin user

ssh-copy-id -i ssh/id_rsa_pkg_admin pkgadmin@<name or ip address>

    Will copy the ssh identity file to the target machine

    You will need the password of the 'pkgadmin' user as configured on the target machine

Debug verbosity is controlled via the 'verbose' config item

The pkg.xxxx config items are very special.

If the named package-version is a symlink, then both the

link and the package addresses will be transferred.

The link MUST address another version of the same package.

This is intended to support the 'jats2_current' link.

When a new version of JATS is released, then the new package

will be transferred, as well the new link.

Config items that control a time period allow the following sufixes:

    s - Seconds. Same as no suffix

    m - Minutes

    h - Hours

    d - Days

Multiple are allowed. ie: 1h10h

Config items that control a file size in blocks allow the following suffixes:

    k - Kilobytes (Same as no suffix)

    b - Blocks    (Same as no suffix)

    m - Megabytes

    g - Gigabytes

ToDo

======================

1) Better handling of soft-links for core_devl

   Works, but its prone to error

   There is no test to ensure the link exists. If the link

   is deleted, then it won't be recreated.