Summary of the changes in 3.2.0

\[DRAFT worksheet for [bug 5382|http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5382], which is now closed; see SVN build/announcements/

3.2.0-rc2.txt for the finished results\]

Please feel free to edit away, some entries are too verbose, some aren't verbose enough. Also feel free to reorder entries so that the most "important" changes are listed at the top of each section.

The List

Changes to the core code:

• bug 3109: short-circuiting of 'definite ham' or 'definite spam' messages based on individual short-circuit rules using the 'shortcircuit' setting, by Dallas Engelken <dallase /at/ nmgi.com>.
  
• bug 5305: implement msa_networks for ISPs to specify their Mail Submission Agents, and extend network trust accordingly.
  
• bug 4636: Add support for charset normalization, so rules can be written in UTF-8 to match text in other charsets.
  
• sa-compile: compilation of SpamAssassin rules into a fast parallel-matching DFA, implemented in native code.
  
• tflags multiple: allow rules to match multiple times.
• IPv6 Support
  
• spamd: TELL commands disabled by default, use --allow-tell to enable. (Already in 3.1)
  
• bug 4589: allow M::SA::Message to use IO::File objects to read in message (same as GLOB). Also add in check to ignore unknown reference types.
  
• bug 4517: rule instrumentation plugin hooks, from John Gardiner Myers <jgmyers /at/ proofpoint.com>
  
• The Great Rules Directory reorg
  
• Rule QA Stuff
  
• bug 4363: if a message uses CRLF for line endings, we should use it as well, otherwise stay with LF as usual
  
• spamc: Add -K option to ping spamd
  
• ; important for Windows users.
  
• bug Bug 4515: content preview omits was omitting first paragraph when no Subject: header was present.
  
• Received Header header parsing updates/fixes/additions
  
• The tflags multiple thing
  
• bug 4700: certain privileged configuration settings can inject code, due to a bad fix for bug 3846. Back that out
  
• add two features to core rule-parsing code; 1. optional behaviour to recurse through subdirs looking for .cf/.pre's, to support rules compilers working on rulesrc dir. 2. call back into invoking code on lint failure, so rule compiler can detect which rules exactly fail the lint check
  
• Bump HTML::Parser minimum version to prevent errors.
  
• sa-update stuffs
  
• Bug 4636: Add support for charset normalization
  
• Bug 4636: Require non-buggy HTML::Parser for charset normalization
  
• .
  

• Wiki Markup
  trusted_networks/internal_networks fixes/stuffs \[TODO: this needs to mention the new 127.0.0.1 behaviour!\] \\

• Wiki Markup
  sa-update stuffs \[TODO: need something more specific here\] \\

• Bugs 4606, 4609: Adjust MIME parsing limits
  
• trusted_networks/internal_networks fixes/stuffs
  
bug 3109: simple short-circuiting of 'definite ham' or 'definite spam' messages based on individual short-circuit rules using the 'shortcircuit' setting, by Dallas Engelken <dallase /at/ nmgi.com>
• for nested multipart/* and message/rfc822 MIME parts.
  
• bug 3466: do the bayes expiry after results have been passed back to the client from spamd, helps avoid client timeouts, etc.
  
• Removed dependency on Text::Wrap CPAN module.
  

Spamc / spamd:

• bug 4603: Mail::SpamAssassin::Spamd::Apache2 – mod_perl2 module, implementing spamd as a mod_perl module, contributed as a Google Summer of Code project by Radoslaw Zielinski
  
• decided to add a public function to set the rendered information instead of expecting plugins to nastily muck with our internal data... bad juju.
  
• bug 5127: allow mimeheader :raw rules to match newlines and folded-header whitespace in MIME header strings
  
• .
  
• spamc: Add '-K' option to ping spamdMove rule functionality and checking into separate Check plugin.
  
• bug 3991: spamd can now listen on UNIX domain, TCP, and SSL sockets simultaneously. Command-line semantics extended slightly, although fully backwards compatibly; add the --ssl-port switch to allow TCP and SSL listening at the same time
  
• reduce memory footprint by about 750KB by: deleting the source for compiled rulesets; deleting stuff used to parse config; compacting the descriptions hash into a single string, for more RAM -efficient but slower lookups
  
• DomainKeys/DKIM stuffs
  
• ArchiveIterator/mass-check cleanups
  
• sa-compile stuffs
  
• .
  
• more complete IPv6 support.
  
• spamc: add '-z' switch, which compresses mails to be scanned using zlib compression; very useful for long-distance use of spamc over the internet.
  
• bug 5296: spamc '--headers' switch, which scans messages and transmits back just rewritten headers. This is more bandwidth-efficient than the normal mode of scanning, but only works for 'report_safe 0'.
• Bump spamc/spamd's protocol version to 1.4, to reflect new HEADERS verb used for '--headers'.
  

Mail::SpamAssassin modules and API:

• bug 4517: rule instrumentation plugin hooks, to measure performance, from John Gardiner Myers <jgmyers /at/ proofpoint.com>.
  
• add two features to core rule-parsing code; 1. optional behaviour to recurse through subdirs looking for .cf/.pre's, to support rules compilers working on rulesrc dir. 2. call back into invoking code on lint failure, so rule compiler can detect which rules exactly fail the lint check.
  
• bug 5206: detect duplicate rules, and silently merge them internally for greater efficiency. This results in about 100-120KB RAM usage saving in current svn trunk's ruleset, detecting lots of duplicate rules – so is well worth doing. also, change t/priorities.t so it doesn't accidentally confuse itself with duplicate rules
  Break out of EvalTests into various plugins.
  
• bug 5236: Support Mail::SPF replacement for Mail::SPF::Query
  bug 5243: add Plugin::register_method_priority() API, allowing plugins to control the relative ordering of plugin callbacks relative to other plugins' implementations
  
• bug 3466: do the bayes expiry after results have been passed back to the client from spamd, helps avoid client timeouts, etc.
  
• mass-check client/server mode
  
• Removed Text::Wrap dependency
  
• .
  
• Reduced memory footprint.
  

Plugins:

• bug 5236: Support Mail::SPF replacement for Mail::SPF::Query.
  

• Wiki Markup
  [DomainKeys]/DKIM stuffs \[TODO: need something more specific here\] \\

• bug 5127: allow mimeheader :raw rules to match newlines and folded-header whitespace in MIME header stringsadd spamc '-z' switch, which compresses mails to be scanned using zlib compression; very useful for long-distance use of spamc over the internet.
  
• bug 4770: add ASN.pm plugin, contributed by Matthias Leisi <matthias at leisi.net>
  
• bug 5296: add spamc --headers switch, which scans messages and transmits back just rewritten headers. This is more bandwidth-efficient than the normal mode of scanning, but only works for 'report_safe 0'. Bump spamc/spamd's protocol version to 1.4, to reflect new HEADERS verb.
  
• bug 5305: implement msa_networks for detecting MSAs and extending trust accordingly
  
• bug 4271: move ImageInfo into 3.2.0 core ruleset
  
• 5271: move ImageInfo ruleset into 3.2.0 core rules, thanks to Dallas Engelken <dallase /at/ nmgi.com>.
  
• VBounce ruleset and plugin: detect spurious bounce messages sent by broken mail systems in response to spam or viruses. (Based on Tim Jackson's "bogus-virus-warnings.cf" ruleset.)
  
• Move rule functionality and checking into separate Check plugin, allowing third parties to implement alternative scanner core algorithms.
  
• core EvalTests code moved into various plugins.
  

Removed

• Rule QA Stuff: removed, it's not part of the distributed code
  
• The Great Rules Directory reorg: removed, it's only visible in SVN
  
• bug 4700: certain privileged configuration settings can inject code, due to a bad fix for bug 3846. Back that out: already in 3.1.x
  
• decided to add a public function to set the rendered information instead of expecting plugins to nastily muck with our internal data... bad juju: an implementation detail I think, not too important
  
• Bump HTML::Parser minimum version to prevent errors, and to support charset normalization: I don't think we need to call this out when it's dealt with in the INSTALL file
  
• ArchiveIterator/mass-check cleanups: too vague, can fall under "misc bug fixes"
  
• mass-check client/server mode: mass-check isn't distributed in the release filesVBounce Plugin