...
- info on how a new rule compares to the full *existing* ruleset
- overlaps between rules, using "hit-frequencies -o"
- collated results across all users' corpora, which can be broken down to view each user separately or all together
- checked rules and their results are kept in a version-control history, so benefits of VC are available
- ongoing visibility of hit-rates of the existing ruleset, against fresh corpora
The ruleqa CGI is now in the SpamAssassin zone, so this is still visible, even though the automc stuff is disabled. Here it is: http://buildbot.spamassassin.org/ruleqa/
...
Someway of scheduling a small run during the development day would be useful, rather than waiting for the nightly. An email of users completed results would be nice to see too.
JustinMason: I think the more immediate, email-based, system is better done using List-Driven Mass-Checks as below; this is good for slow-but-comprehensive daily tests.
List-Driven Mass-Checks
Loren outlined the system used in SARE:
- rule developer sends mail to mailing list
- various other participants run scripts that automatically extract certain attachments posted to the list
- turn those into rules files
- lint them
- run a mass-check immediately with just the rules in that file
- post results including hit freqs and false positives matches
- masscheck requester asks for false positive verification based on report
For active rule development, this is obviously quite useful! If you can't run mass-check locally for whatever reason, it offers a way to do this using other people's corpora in almost-real-time.
JustinMason: 'I'd like to see if there's a way to combine the two (that is, nightly and list-driven mass-checks) somehow, so that new SVN commits that update sandbox rules, are immediately mass-checked alone. However, I can't see a way to do that reliably from SVN commits alone, because (for example) meta rules may depend on other rules that were not changed as part of the same commit. So I think the "email with attached rules file" is still a better model.'
'LOAFER': There are eval rules to consider too.
JustinMason: I think we have to do those as plugins, via the sandboxes.
Here's the current proposal:
...
the "preflight" system provides the "small run" option – see below!
Preflight Mass-Checks
...
.