Feeding back mail for the Bayesian learner via forwarded mail
This is a form of SiteWideBayesFeedback.
For MUAs (Like Netscape/Mozilla) that do a good job with keeping orignal headers intact, (almost) all you need to do is forward the email to the feedback account and strip off the header added by the forward ( provided that you forward inline. I'll try to update bayes_fixup.pl for forwarding as attachment at a later date). This can be done by calling a filter from the ~/.procmailrc file of the learner accounts. (I apologize for putting these scripts in the Wiki, but I have no publically accessable location to post them, If someone who does has that capability, and could just replace them with links, I'd appreciate it)
I am not sure how sa-learn will interpret a signature when you forward email inline, so you should probably delete your .sig before forwarding the message.
I call spamc from /etc/procmailrc, but I make sure that it does't filter mail to is_spam and not_spam
/etc/procmailrc
# Don't filter mail to is_spam and not_spam
#
# Since we are running sitewide, it could cause a serious bottleneck if
# we were to use a lockfile here. instead, we limit spamd to 20 child
# processes in /etc/sysconfig/spamassassin
#
#:0fw: spamassassin.lock
:0fw
* !^To.*spam@mycompany.com
* < 256000
| spamc
~is_spam/.procmailc ( I also have a ~not_spam/.procmailrc that is identical )
# filter spam feedback
:0fw: bayes_fixup.lock
* < 256000
| /usr/local/adm/bin/bayes_fixup.pl
bayes_fixup.pl is:
#!/usr/bin/perl
#
# This filter is designed to pull off the forwarding headers for mail
# forwarded to is_spam or not_spam from an MUA that includes all
# headers. ( as opposed to outlook, which does not include all
# headers, and thus must be resent instaed of forwarded. )
#
# In a forwarded message from Netscape/Mozilla, you will have:
#
# From ...
# ...
# From: (matches envelope from)
# ...
# one or more blank lines
# -------- Original Message --------
# From: (a date code for the forwading MUA)
# The original Headers
#
# You will not have:
# Sender:
#
# Not sure if the Netscape stuff is valid for HTML mode.
#
# Brian R. Jones 01/30/04 scumpuppy_@_earthlink_._net
#
use strict;
my ($count,$endheader,$sender,$unknown);
my $fwdmarker = "-------- Original Message --------";
my @message = <STDIN>;
#
# Determine if sender is Outlook, Netscape/Mozilla or unknown.
# If Netscape, set a marker for the end of the headers that are added
# by the forwarding.
#
for( $count = 0, $endheader = 0, $sender = 0, $unknown = 0; ; $count++ ) {
$_ = $message[$count];
/^Sender:/o and last; # It's a resent message from Outlook, skip
/^\s*$/o and do { # end of headers marked with one or more
$endheader = 1; # blank lines
next;
};
next unless $endheader;
/^$fwdmarker/o or $unknown = 1;
last;
}
#
# If it's Netscape, delete the forwarding header, and clean up the
# original. I'm also converting the 'From:' to the 'Envelope From'
# which may not be legitimate. It may be better to use the forward
# header 'Envelope From'. Unfortunately, there is no way to capture
# the original 'Envelope From'. :(
#
if ( $endheader && ! $unknown ) { # forwarded from known mailer
splice(@message, 0 , ++$count);
$message[0] =~ s/^From:/From/;
for ( @message ) { # Stupid Netscape collapse continuation lines,
# so we need to put `em back in case sa-learn
# doesn't understand `em.
/^[\w\-]+:/ and next; # Valid header
/^\t/ and next; # Valid Continuation line
/^From/ and next; # Newly created Envelope From
/^\s*$/ and last; # End of Headers
$_ = "\t" . $_; # Malformed continuation line. Add tab.
}
} elsif ( $unknown ) { # unknown, toss it.
exit 1;
}
print @message;
So all of the above handles delivery of a nearly (except for the 'envelope From') untainted message to the spam (is_spam) and ham (not_spam) accounts on the server. Note that these messages live where sendmail sends them. Next you need to run sa-learn on them, and sa-learn requires they first be split into individual messages. To do that, I call another script (learn_spam.pl) from cron. Since I'm using a Redhat Linux box I do it like this:
/etc/cron.daily/learnspam ( When you are testing, remove the redirect to /dev/null and cron will automatically email you (assuming you are root) the output from learn_spam.pl):
#!/bin/bash # # run sa-learn on mail sent to the is_spam and not_spam accounts # /usr/local/adm/sbin/learn_spam.pl > /dev/null
/usr/local/adm/sbin/learn_spam.pl:
#!/usr/bin/perl -I/usr/local/lib
#
# run sa-learn on is_spam and not_spam to update spamassassin
#
# brj 01/27/04
use strict;
use Cwd;
require "splitmail.pl";
my $spamfile = "/var/mail/is_spam";
my $hamfile = "/var/mail/not_spam";
my $tmpdir = "/var/tmp/split";
#my $learn_spam = "sa-learn --spam --showdots --dir $tmpdir";
#my $learn_ham = "sa-learn --ham --showdots --dir $tmpdir";
my $learn_spam = "sa-learn --spam --dir $tmpdir";
my $learn_ham = "sa-learn --ham --dir $tmpdir";
my $startdir = cwd();
sub init {
if ( ! -d $tmpdir ) {
mkdir $tmpdir;
} else {
if ( chdir($tmpdir) ) {
unlink <*>;
}
chdir($startdir);
}
}
sub learn {
my $infile = shift;
my $command = shift;
if ( -r $infile ) {
splitmail($infile,$tmpdir);
system("$command");
if ( chdir($tmpdir) ) {
unlink <*>;
}
chdir($startdir);
}
}
sub cleanup {
unlink $spamfile, $hamfile;
rmdir $tmpdir;
}
init();
learn( $spamfile, $learn_spam );
learn( $hamfile, $learn_ham );
cleanup();
Since I have several other apps that also require splitting a mail file I wrote splitmail (or maybe I borrowed it from someone else, I'm not sure) as a library.
/usr/local/lib/splitmail.pl:
#!/usr/bin/perl
#
# splits a file containing multiple messages into individual files
#
use strict;
sub splitmail {
my $infile = shift;
my $outdir = shift;
my $count = 0;
open(INFILE, "< $infile") or die "Can't open $infile: $!\n";
while(<INFILE>) {
/^From / and do {
close(OUTFILE) if $count;
open(OUTFILE, "> $outdir/$count") or die "Can't open $outdir/$count: $!\n";
$count++;
};
print OUTFILE $_;
}
close(OUTFILE);
}
1;
Alternately, you can use this wrapper for sa-learn and call it from a .qmail file for on-the-fly split-and-learn-via-forward.
/usr/bin/learn_spam:
#!/usr/bin/perl
#
# run sa-learn on STDIN ... easy to use with .qmail files:
#
# .qmail-spamtrap:
# | learn_spam --spam --username=alias | cat - > /dev/null
# .qmail-qqqhamreport:
# | learn_spam --ham --username=alias | cat - > /dev/null
#
# 3/16/2005 -- cgg007 at yahoo.com
#
use strict;
sub learn {
my $message = shift;
my $pipe = shift;
open LEARN, $pipe;
print LEARN $message;
close LEARN;
}
my $learn_cmd = "| bayes_fixup.pl | sa-learn " . join(" ",@ARGV);
my $count = 0;
my $message = '';
while (<STDIN>) {
/^From/ and do {
if ($count) {
learn($message,$learn_cmd);
$message = '';
}
$count++;
};
$message .= $_;
}
learn($message,$learn_cmd);