Rule Promotion and the sa-update Backend

'sa-update' updates are created on the zone, nightly, by a cron job. Here are some details as to how this works.

At 0830 UTC, the build/mkupdates/run_nightly script is run from svn trunk. This runs the script build/mkupdates/listpromotable, which queries the rule-QA web app for the previous day's rule accuracy figures, and determines which rules from the rules source dirs (rulesrc/core and rulesrc/sandbox/*) are to be promoted to the active ruleset, as described in RuleLifeCycle.

Promotion Criteria

You can see the previous day's accuracy figures at this URL. Promotable rules show up in dark text on the RuleQaApp, non-promoted-by-default rules show up in light grey text.

The promotion criteria are detailed on some other wiki page, RulesProjPromotion perhaps? (TODO).

Anyway, using those figures, and some other criteria:

A proposed new list of 'active' rules is created in the file 'rules/active.list'.

The Active List

The active list, in the form of the file rules/active.list is checked in to SVN trunk, and is used by build/mkrules to determine which rules are active.

More info on these rule states can be found at RuleLifeCycle.

build/mkrules attempts to keep the contents of those files consistent. In other words, meta subrules for a rule in the active list will always likewise be copied to the active file -- even if they are sandbox T_ rules. It will also ignore rules that are inside an ifplugin block for a plugin that is not available.

rules/70_sandbox.cf and rules/70_inactive.cf are used for development, and for mass-checks, but will not be installed during make install and are not packaged in sa-update tarballs. Rules in rules/70_sandbox.cf are the only ones used in the "bbmass" preflight mass-checks.

The sa-update Tarball

build/mkrules is run, using the active list to create rules/72_active.cf, rules/70_sandbox.cf and rules/70_inactive.cf. The latter two files are then discarded, and spamassassin --lint run to ensure the ruleset is still in a consistent state. If lint fails, the update process is abandoned.

A make install is run, to a temporary directory, and the resulting ruleset is packaged using tar, linted again from that dir, and then signed and checksummed.

The DNS zone is updated using the SVN revision number, and the tarball copied into place.

Administrivia

The cron jobs run as the user updatesd. The crontab looks like:

30 8 * * * bash /home/updatesd/svn/spamassassin/build/mkupdates/run_nightly > /var/www/buildbot.spamassassin.org/updatesd/mkupdates.txt 2>&1
50 8 * * * bash /home/updatesd/svn/spamassassin/build/mkupdates/run_part2 > /var/www/buildbot.spamassassin.org/updatesd/mkupdatespt2.txt 2>&1

SA-Update Mirrors

sa-update uses the MIRRORED.BY file to provide distributed update capabilities. The setup for a mirror is described in SaUpdateMirrorSetup.

The Other End

See RuleUpdates for details on running "sa-update".

SaUpdateBackend (last edited 2013-02-11 17:23:13 by KevinMcGrail)