The main system used to measure spam-filtering accuracy in SpamAssassin is the "FP%/FN% percentages" system.
It's quite simple. First, you scan a corpus of hand-classified mail (see HandClassifiedCorpora) to get 4 figures:
nspam = number of known-to-be-spam messages in the corpus nham = number of known-to-be-ham (nonspam) messages in the corpus fp = number of ham messages incorrectly marked as spam fn = number of spam messages incorrectly marked as ham
Next, perform this calculation:
FP% = (fp / nham) * 100 FN% = (fn / nspam) * 100
and you have two numbers that simply, concisely, and comprehensibly describe the accuracy and performance of the filter.
For example, let's say we do a test as follows:
nspam = 1000 nham = 1500 fp = 2 fn = 30
the FP% and FN% work out as (2 / 1500) * 100 = 0.1333% and (30 / 1000) * 100 = 3.0% respectively.
The 'STATISTICS.txt' files distributed with SpamAssassin versions since about 2.30 include this data, measuring the ruleset's accuracy against a validation ruleset:
# SUMMARY for threshold 5.0: # Correctly non-spam: 29443 99.97% # Correctly spam: 27220 97.53% # False positives: 9 0.03% # False negatives: 688 2.47% # TCR(l=50): 24.523726 SpamRecall: 97.535% SpamPrec: 99.967%
As you can see, FP% and FN% get pride of place in the measurement scheme.
See also MeasuringAccuracy for other methods.