Translation synchronisation
The need for a synchronization process
The files which are really used for building &d-i; packages are hosted
in the git repository used for the development of &d-i;
packages. Each &d-i; package sources is hosted in a subdirectory of
the packages/ directory, with a very classical
Debian package source tree organization, including the
debian/po directory where the package's
translations are hosted.
These files need to be synchronised with the so-called
master files from the packages/po directory, since
translators only work on these master files. This
synchronization must work both ways with changes on the English original strings
going from individual packages to the master files and translation changes/updates
from the master files going to individual packages:
When &d-i; packages templates are changed, the individual
debian/po/templates.pot file will
change. These changes have to be moved to files in
packages/po;
When translations are updated in the
packages/po directory, these changes have
to be moved back into individual
debian/po/*.po files for each package. This
does not need to be done at each translation update, but must be
done before building/uploading packages.
The task of keeping track of the correct synchronisation is one of the
most important tasks of the &i18n-coords;. It is handled by the
scripts/l10n/l10n-sync script.
Localization files synchronisation process
Synchronization process
The l10n-sync script is run periodically under the "D-I role"
account by a cron job
on dillon.debian.org:
0 20 * * * cd $DI; mr -q up ; nice -n 19 $DI/scripts/l10n/l10n-sync --levels=5 --online --commit $DI
Its runs are monitored by the &i18n-coords;.
It will update the general
template.pot and all PO files from individual
templates.pot files from all &d-i; packages.
While running, this script will also run
debconf-updatepo for each &d-i; package and will
commit back the regenerated templates.pot files (package maintainers
often forget running debconf-udpatepo when
commiting changes to English templates).
During normal development, this synchronization process is run once a
day. During release preparations, the frequency might be increased to
speed up the full cycle.
The general process is:
Initial step: synchronize the whole repository;
Individual packages update step. For each
&d-i; package:
synchronize the local copy with the &d-i; GIT;
run debconf-updatepo;
commit back the files in debian/po.
Master templates file update step:
merge all templates.pot files to
packages/po/template.pot.
Master PO files update step. For each PO
file in packages/po:
synchronize with &d-i; GIT;
if the script is run with the --merge option,
merge translations from the reference PO file (the reference file is
given priority for identical strings);
update with
packages/po/template.pot. This step
uses the msgmerge command. The resulting PO file
formatting details will depend on the version of this
utility. For that reasons, the synchronization script should
always be run from machines using the same Debian release
version (these days it permanently runs on dillon anyway);
commit back the changed files to &d-i; GIT.
Individual packages PO files update
step. For each &d-i; package:
synchronize the local copy with the &d-i; GIT (in case some
update occurred in the meantime);
update debian/po/*.po files with master
files;
commit back the changes to &d-i; GIT;
This system minimizes race conditions which could trigger conflicts.
The conflict windows have been minimised as much as possible in the
l10n-sync script. However, in order to limit the
number of commits made by the script, files are not commited as soon
as they are modified. So, the
script is sometimes likely to trigger conflicts. For that reason, it
will stop working on a GIT copy where GIT conflict files are
present. This explains why this script must always be monitored even
when it is scheduled to be executed periodically.
Synchronization script use
The l10n synchronisation script is kept in &d-i; repository in the
scripts/l10n/ directory and is named
l10n-sync.
It is a shell script (which may contain some bashisms until a
skilled shell programmer cleans it out).
The script uses some command-line switches which may affect its behaviour.
Some of these switches are mostly present for historical reasons and are kept
because they may have some new use in the future:
--debug will trigger more output by the
script. Otherwise, the script reports about its actions but
commands output (such as git,
debconf-updatepo,
msgmerge) will be redirected to
/dev/null;
--online triggers git
pull commands before working on files. Except for
testing, there is no reason for not using
this switch.
--commit allows the script to
commit files back to the &d-i;
repository. Otherwise, the modified files are kept in the
local GIT copy. Of course, this switch should be used in
production. Omitting it is useful only for testing.
the --debconf-updatepo defines the
command for calling the debconf-updatepo
utility. It allows using a specially crafted
debconf-updatepo switch, often the
--skip-merge switch which only updates
templates.pot files and not PO files
(using this generally speeds up the synchronization script);
the --merge switch allows merging
master files from another branch. It is used, for instance, on
the sarge branch for merging translations coming from trunk.
The script needs the location of the local copy of the &d-i;
repository as an argument. It makes some simple checks about the
copy.
When using this script with commits, the &d-i; copy must be as clean as
possible. It should not be used for development tasks. The script
checks for possible GIT conflict files and aborts if it finds some.
Prospective languages (see )
are handled in a special way: for all such languages, the translations
are not copied in the individual packages directories.
The PROSPECTIVE list
When languages are on early stages of translation after they have been added
through the new language process (), they are
temporarily listed in a file named packages/po/PROSPECTIVE.
Languages listed in that file are excluded from synchronization, and,
therefore, PO files are not created in individual packages. As a consequence,
activating a language means that the language code is
removed from the PROSPECTUVE file.
List of handled packages
A special file, named packages/po/packages_list,
lists the &d-i; packages which are handled by the
l10n-sync script and are therefore
included in the master files.
The &d-i; packages maintainers must request for their packages to be
included in this file, when they estimate that their package is ready
for translation. Only &i18n-coords; can integrate packages there, after
checking that the strings have been reviewed.
Synchronization script control
Some mechanisms have been implemented to add more safety and remote
control possibilities to the synchronisation script.
First of all, before doing any work, the script opens a special file
kept in the GIT repository and named
packages/po/run-l10n-sync. The "run=" line in
this file mentions whether synchronisation should happen or not. If
the file contains "run=0", then the script will exit without taking
any action.
This mechanism gives all &d-i; developers with commit access to the
GIT repository a very simple way to disable the
synchronization script actions. For this, developers just need to
change the file and commit the new version.
While the script is disabled, runs can still be enforced by using the
-force swith, which will ignore the
packages/po/run-l10n-sync file.
While it is running, the script also creates a file named
.l10n-sync.lock at the root of the local copy of
&d-i; repository. The file is removed only after successful runs of
the script. When this file already exists, the script does not run and
exits with an error message.
This mechanism prevents running the synchronization script twice on
the same &d-i; repository checkout copy.