My take on MailCleaner + Exchange + IMAP Bayes Training

Users tips and ideas

Moderators: FlorianB, olivier, mentor, bourgeois

uncltom
Posts: 435
Joined: Tue Aug 26, 2008 3:01 am
How did you hear about Mailcleaner: I dont remember probably google?
Location: Spokane, WA

My take on MailCleaner + Exchange + IMAP Bayes Training

Postby uncltom » Wed Apr 15, 2009 10:24 am

This is my personal twist to the whole mailcleaner imap training thing...
I added a few things and consolidated a new feature from the original author.

Thomas Nelson
Perfection PC
Spokane, WA

Code: Select all

#!/usr/bin/perl

# Imap Interface to SpamAssassin Learn     v0.03B
# ------------------------------------     -----
#
# Connects to an imap server, and filters the messages from the INBOX
# and SpamTrap (unless otherwise told) through sa-learn
#
#  usage:
#    imap-sa-learn.pl <-hamfolder HAM> <-spamfolder SPAM>
#
#  Other options:
#    -skips nnn      skips over the first nnn messages in the folder(s)
#    -deletespam   after learning from a spam message, delete it
#    -delete-spam   (as above)
#    -dangerous-delete-ham   after learning from a ham (real email),
#                delete it. Most people don't want this...
#    -dangerous-delete-all   after learning from any message, delete
#                it, spam or ham
#
# Uses Mail::IMAPClient and SpamAssassin (sa-learn)
#
# Needs a version of SpamAssassin with the Bayesian filtering support,
# i.e. 2.50 or later
#
#      Nick Burch <nick@tirian.magd.ox.ac.uk>
#           25/06/2003
#
# Changes:
# Added SSL connection to work with Exchange 2007 from Nick's work.
# Added bogofilter to the script so it creates a new bogofilter and spamassassin filter.
# Added a seperate section for spam and ham. (we have seperate e-mail accounts for spam and ham.)
# Restored the debugging feature to IMAPClient. 
# Added several break points that give some debuggin information if there is an error.
#
# Thomas Nelson <tom AT perfectionpc.com>
#        15/04/2009
use Mail::IMAPClient;
use IO::Socket::SSL;

# Define our server and credentials here
# your details go below
my $spamusername = 'spam_account';
my $spampassword = 'spam_password';
my $spamserver = 'spam_server';

my $hamusername = 'ham_account';
my $hampassword = 'ham_password';
my $hamserver = 'ham_server';

# Define where to find messages
my $defspamfolder = 'INBOX';
my $defhamfolder = 'INBOX';
my $deletespam = 1;
my $deleteham = 1;
my $default = 1;

my $skips = 0;

# Output:  Normal (1), Debugging (2), or silent(0)?
my $debug = 0;

my @spams;
my @hams;

while(my $arg = shift) {
   if($arg eq "-spamfolder") {
     my $spam = shift;
     push @spams,$spam;
     print "Using spam folder $spam\n";
     $default = 0;
   }
   if($arg eq "-hamfolder") {
     my $ham = shift;
     push @hams,$ham;
     print "Using normal (ham) folder $ham\n";
     $default = 0;
   }
   if($arg eq "-deletespam" || $arg eq "-deletespams" || $arg eq "-delete-spam" || $arg eq "-delete-spams") {
     $deletespam = 1;
   }
   if($arg eq "-dangerous-delete-ham" || $arg eq "-dangerous-delete-hams") {
     $deleteham = 1;
   }
   if($arg eq "-dangerous-delete-all") {
     $deletespam = 1;
     $deleteham = 1;
   }
   if($arg eq "-skips" || $arg eq "-skip") {
     $skips = shift;
   }
   if($arg eq "-?" || $arg eq "-h") {
     print "Usage:\n";
     print "  imap-sa-learn.pl [-spamfolder f]* [-hamfolder f]*\n\n";
     print "with no argumnets, uses default folders\n";
     print "(a few other options exist, see the header of the program)\n";
     exit;
   }
}

if($default) {
   push @hams,$defhamfolder;
   push @spams,$defspamfolder;
}

my %folders;
$folders{'spam'} = \@spams;
$folders{'ham'} = \@hams;


# Open an SSL session to the IMAP server
# Handles the SSL setup, and gives us back a socket
my $ssl=new IO::Socket::SSL("$spamserver:imaps");
die ("Error connecting - $@") unless defined $ssl;
$ssl->autoflush(1);

# Connect to the IMAP server in peek (i.e. don't set read flag) mode
my $imap = Mail::IMAPClient->new(
             Debug    => $debug,
             Socket   => $ssl,
             Server   => $spamserver,
             User     => $spamusername,
             Password => $spampassword,
             Peek     => 1) or die "Cannot connect to $spamserver as $spamusername: $@";



   $type = 'spam';
   foreach my $folder (@{$folders{$type}}) {
      print "\nLooking in $type folder $folder\n";

      # Pick the folder
      $imap->select($folder) or die "Cannot connect to $folder on $spamserver: $@";

      # Enable peek mode
      $imap->Peek(1);

      # Fetch messages
      my @mails = ($imap->seen(),$imap->unseen);

      my $count = 0;

      foreach my $id (@mails) {
         $count++;
         if($count < $skips) { next; }

         print " Learning on $type message $id\n";
         my $mail = $imap->message_string($id) or die "Cannot fetch $id in $folder: $@";
         open SA, "| sa-learn -p /usr/mailcleaner/etc/mailscanner/spam.assassin.prefs.conf --siteconfigpath /usr/mailcleaner/share/spamassassin --no-sync --$type --single";
         print SA $mail;
         close SA;

     # Bogofilter section
         open BF, "| /usr/bogofilter/bin/bogofilter -s";
         print BF $mail;
         close BF;


         if($type eq "spam" && $deletespam) {
            # If you want to move the message rather than deleting it,
            # uncomment the line below, change the folder, but _don't_
            # remove the delete line!
            #$imap->append('TrashBin', $mail );

            print "Deleting Spam Message $id\n";
            $imap->delete_message($id);
         }
     
      if($deletespam) {
         # Only expunge now, rather than on every message
         $imap->expunge();
      }
   }
}

$imap->close;
$ssl->close;

# Open an SSL session to the IMAP server
# Handles the SSL setup, and gives us back a socket
my $ssl=new IO::Socket::SSL("$hamserver:imaps");
die ("Error connecting - $@") unless defined $ssl;
$ssl->autoflush(1);

# Connect to the IMAP server in peek (i.e. don't set read flag) mode
my $imap = Mail::IMAPClient->new(
             Debug    => $debug,
             Socket   => $ssl,
             Server   => $hamserver,
             User     => $hamusername,
             Password => $hampassword,
             Peek     => 1) or die "Cannot connect to $hamserver as $hamusername: $@";

   $type = 'ham';
   foreach my $folder (@{$folders{$type}}) {
      print "\nLooking in $type folder $folder\n";

      # Pick the folder
      $imap->select($folder) or die "Cannot connect to $folder as $hamusername: $@";

      # Enable peek mode
      $imap->Peek(1);

      # Fetch messages
      my @mails = ($imap->seen(),$imap->unseen);

      my $count = 0;

      foreach my $id (@mails) {
         $count++;
         if($count < $skips) { next; }

         print " Learning on $type message $id\n";
         my $mail = $imap->message_string($id) or die "Cannot fetch $id in $folder: $@";
         open SA, "| sa-learn -p /usr/mailcleaner/etc/mailscanner/spam.assassin.prefs.conf --siteconfigpath /usr/mailcleaner/share/spamassassin --no-sync --$type --single";
         print SA $mail;
         close SA;

     # Bogofilter section
         open BF, "| /usr/bogofilter/bin/bogofilter -n";
         print BF $mail;
         close BF;

         if($deleteham) {
            print "Deleting Ham (normal email) Message $id\n";
            $imap->delete_message($id)  or die "Cannot delete $id in $folder: $@";
         }

      if($deleteham) {
         # Only expunge now, rather than on every message
         $imap->expunge() or die "Cannot empty trash in $hamusername: $@";
      }
   }
}

print "\nNow rebuilding the Bayesian filters\n";
`sa-learn -p /usr/mailcleaner/etc/mailscanner/spam.assassin.prefs.conf --siteconfigpath /usr/mailcleaner/share/spamassassin --sync`;

$imap->close;
$ssl->close;
exit;
higadmin
Posts: 30
Joined: Mon Nov 14, 2011 3:59 pm
How did you hear about Mailcleaner: Google, Lots of Google

Re: My take on MailCleaner + Exchange + IMAP Bayes Training

Postby higadmin » Wed Feb 15, 2012 9:34 pm

I had to add this:



# Connect to the IMAP server in peek (i.e. don't set read flag) mode
my $imap = Mail::IMAPClient->new(
Debug => $debug,
Socket => $ssl,
Server => $spamserver,
User => $spamusername,
Password => $spampassword,
IgnoreSizeErrors => 1,
Peek => 1) or die "Cannot connect to $spamserver as $spamusername: $@";
jdixon
Posts: 44
Joined: Wed Feb 29, 2012 10:03 pm
How did you hear about Mailcleaner: Google

Re: My take on MailCleaner + Exchange + IMAP Bayes Training

Postby jdixon » Fri Jul 06, 2012 4:58 pm

Have you had luck with this script on Exchange 2010 connecting with a UPN as the username? I can't seem to get it connect correctly. All the ports are open and I can telnet just fine.

Nevermind. I screwed up and put the HAM and SPAM under INbox. Moved it to the parent folder and it worked. I also read the error wrong
jdixon
Posts: 44
Joined: Wed Feb 29, 2012 10:03 pm
How did you hear about Mailcleaner: Google

Re: My take on MailCleaner + Exchange + IMAP Bayes Training

Postby jdixon » Sat Jul 07, 2012 10:54 pm

I compiled about 1000 HAMs and 1000 SPAMs and ran your script. I did have to modify the /usr/bogofilter to /opt/bogfilter with the latest mail cleaner.

I'm still getting a X-NiceBayes: disabled (no database ?) and can't find the words database. What exactly am I missing?
bexx32
Posts: 7
Joined: Tue Apr 03, 2012 2:02 pm
How did you hear about Mailcleaner: internet

Re: My take on MailCleaner + Exchange + IMAP Bayes Training

Postby bexx32 » Fri Aug 17, 2012 11:04 am

can i run the script without ssl?
gameover0
Posts: 19
Joined: Wed Jan 23, 2013 7:28 pm
How did you hear about Mailcleaner: google

Re: My take on MailCleaner + Exchange + IMAP Bayes Training

Postby gameover0 » Fri Jan 25, 2013 4:58 pm

I haven't found much documentation on setting this up. Any tips on how to setup this script?
cglmicro
Posts: 190
Joined: Thu Mar 07, 2013 2:12 am
How did you hear about Mailcleaner: google

Re: My take on MailCleaner + Exchange + IMAP Bayes Training

Postby cglmicro » Thu May 29, 2014 1:17 pm

I'm beggining to look at Nicebayes and your script. I don't know if it can take advantage of small amount of spams (i.e.: less than an 100 SPAMS per week and about 500 hams per week), or if I'm wasting my time trying to have it trained on such small amount of emails?

Eitherway, I looked at your script and I wonder if :

Code: Select all

my $deletespam = 1;
my $deleteham = 1;


...shouldn't be set both to "0" so it won't delete by default, unless we provide the -deletespam or -deleteham in the command line?
aknobbe
Posts: 15
Joined: Wed May 02, 2012 12:18 am
How did you hear about Mailcleaner: google

Re: My take on MailCleaner + Exchange + IMAP Bayes Training

Postby aknobbe » Mon Dec 14, 2015 10:20 am

Hi all, this script looks great but can someone tell me how to run this?

Return to “Tips”

Who is online

Users browsing this forum: No registered users and 3 guests