...
- rule developer sends mail to mailing list
- various other participants run scripts that automatically extract certain attachments posted to the list
- turn those into rules files
- lint them
- run a mass-check immediately with just the rules in that file
- post results including hit freqs and false positives matches
- masscheck requester asks for false positive verification based on report
For active rule development, this is obviously quite useful! If you can't run mass-check locally for whatever reason, it offers a way to do this using other people's corpora in almost-real-time.
...
- Good web UI for "builds in progress"; you can monitor progress as it happens
- Designed to do "build whenever necessary, but not more than necessary", ie it solves the load issues caused by continuous integration
- Every mass-check output and every 'freqs' output will get a HTTP permalink, which allows side discussion to "point at" test results
- Integrated with SVN for version control and history tracking, so we can easily find test results that correspond to a mailed-in rules file
- Allows us to run mass-checks securely, in a buildbot slave running in a segmented chroot jail
In terms of results of mail-in rule checking; there's no automated check-ins into the "sandboxes" or "core" from this. Instead rules considered suitable for use are manually checked into the "sandbox" area by one of the committers who has privs to do that. With luck, they'll go into the core based on the automated testing described in RulesProjStreamlining. This is a rule-QA system, not an alternative to "svn commit".
...
JustinMason: In my opinion, we should go with this for now anyway. The use of SVN makes it hard to make it privately-viewable only, but at the same time offers some of the biggest advantages – such as keeping archive copies of the submitted rules after the mass-check has completed, providing network transparency, and ensuring that submitted rules get mass-checked (eventually).