...
JustinMason: I think the more immediate, email-based, system is better done using List-Driven Mass-Checks the faster preflight system, as below; this one is good for slow-but-comprehensive daily tests.
List-Driven Mass-Checks
Loren outlined the system used in SARE:
- rule developer sends mail to mailing list
- various other participants run scripts that automatically extract certain attachments posted to the list
- turn those into rules files
- lint them
- run a mass-check immediately with just the rules in that file
- post results including hit freqs and false positives matches
- masscheck requester asks for false positive verification based on report
For active rule development, this is obviously quite useful! If you can't run mass-check locally for whatever reason, it offers a way to do this using other people's corpora in almost-real-time.
JustinMason: 'I'd like to see if there's a way to combine the two (that is, nightly and list-driven mass-checks) somehow, so that new SVN commits that update sandbox rules, are immediately mass-checked alone. However, I can't see a way to do that reliably from SVN commits alone, because (for example) meta rules may depend on other rules that were not changed as part of the same commit. So I think the "email with attached rules file" is still a better model.'
'LOAFER': There are eval rules to consider too.
JustinMason: I think we have to do those as plugins, via the sandboxes.
Here's the current proposal:
...
Preflight Mass-Checks
...
.