MailCleaner + Exchange + IMAP Bayes Training

Users tips and ideas

Moderators: FlorianB, Pascal, bourgeois, mentor

User avatar
jordant
Posts: 218
Joined: Sat Sep 16, 2006 2:48 pm
Location: Perth, Western Australia

MailCleaner + Exchange + IMAP Bayes Training

Postby jordant » Tue Sep 19, 2006 12:20 pm

In our setup, we have MailCleaner in a data center passing messages to remote exchange 2003 servers.
This is all pretty standard however I have modified the MailCleaner engine (MailScanner) to deliver spam messages un-modified (no subject tagging) without quarantining a copy of message.

The reason for this is we insert a header to detected spams that an Exchange 2003 event sink picks up and moves the message to the users Spam folder server side. (il write up a howto on this and provide the sink we use at a later date)

This works well for us, however it leaves no copy of the message in our MailCleaner quarantine, so bayes training has to be done another way.

This is where Public Folders and IMAP come in.

If a user find a false-positive in his/her local Spam folder, they simply drag it into the Ham Public Folder. Likewise with undetected spam, they can drag the message into the Spam Public Folder.

Using a small perl script, we are able to remotely connect to our Exchange servers over the internet using the imap connector and fetch emails from Public Folders, sending spam and ham to SpamAssassin's sa-learn to be trained.

The original imap-sa-learn.pl was created by Nick Burch, you can find more of his work at http://www.gagravarr.org/code/

I have slightly modified his script to work with MailCleaner and is available at http://waix.blue-ferret.com.au/mailcleaner/imap-sa-learn.pl

Before you can use it however, you must install perl module Mail::IMAPClient using CPAN.

Start off by entering the CPAN shell

Code: Select all

mailcleaner:~# perl -MCPAN -eshell
[snip].....[/snip]
Are you ready for manual configuration? [yes]


The only thing you have to do here is set your location so perl knows which mirror to use. You can pretty much use the defaults to all other questions asked, and dont worry that it cant find ncftpget.

Once CPAN is configured, tell it to fetch and install the module we need

Code: Select all

cpan> install Mail::IMAPClient


It will go and fetch the file, unpack and start compiling it.
Say no to extended tests and everything should be fine.

exit the CPAN shell and download the imap script making sure to edit the file and change configuration options $username, $password, $server, $defspamfolder and $defhamfolder to suit your needs.

We run the script hourly via crontab, to do this put it in /etc/cron.hourly/ and make sure its +x
Last edited by jordant on Mon Sep 10, 2007 5:29 am, edited 1 time in total.
olivier
Posts: 1348
Joined: Thu Jan 01, 1970 1:00 am
Contact:

Postby olivier » Tue Sep 19, 2006 1:52 pm

great ! thank for the tip !
User avatar
suhasingale
Posts: 298
Joined: Mon Nov 13, 2006 2:30 pm
Location: India
Contact:

Postby suhasingale » Fri Nov 17, 2006 7:00 pm

This is really a great job jordant.

Will try it tomorrow.

Thanks in advance.
User avatar
suhasingale
Posts: 298
Joined: Mon Nov 13, 2006 2:30 pm
Location: India
Contact:

Postby suhasingale » Tue Nov 21, 2006 3:53 pm

Hi jordant,

What if we apply any patches?

Do we have to install the plugin again and reschedule the cron job?
User avatar
jordant
Posts: 218
Joined: Sat Sep 16, 2006 2:48 pm
Location: Perth, Western Australia

Postby jordant » Tue Nov 21, 2006 3:57 pm

Not unless a patch somehow removes the perl library - and i dont know of any that currently do.
olivier
Posts: 1348
Joined: Thu Jan 01, 1970 1:00 am
Contact:

Postby olivier » Tue Nov 21, 2006 3:59 pm

no it doesn't, and will probably never do so.. we're friendly people :lol:
User avatar
suhasingale
Posts: 298
Joined: Mon Nov 13, 2006 2:30 pm
Location: India
Contact:

Postby suhasingale » Tue Nov 21, 2006 5:26 pm

I have configured your perl script and is working like a charm.

To run it hourly, shall i just move it to the folder /etc/cron.hourly/ or anything else apart from this has to be done? Like you said "make sure its +x" wht does that mean?
User avatar
jordant
Posts: 218
Joined: Sat Sep 16, 2006 2:48 pm
Location: Perth, Western Australia

Postby jordant » Wed Nov 22, 2006 2:36 am

Best check with Olivier as to what the best method of cron'ing something is under mailcleaner as theres a few different ways to do it.

+x is execute flag,

Code: Select all

chmod +x imap-sa-learn.pl
User avatar
suhasingale
Posts: 298
Joined: Mon Nov 13, 2006 2:30 pm
Location: India
Contact:

Postby suhasingale » Wed Nov 22, 2006 7:52 am

Ohh so that was permission you were talking abt. I already gave 777 permission for the root user and works fine now.

Olivier, can you focus some light on this, abt setting up the cron...
olivier
Posts: 1348
Joined: Thu Jan 01, 1970 1:00 am
Contact:

Postby olivier » Wed Nov 22, 2006 8:35 am

using /etc/cron.hourly is perfect.
Modifying the root cron is a little bit more dangerous because some update may well have to play with it.
User avatar
suhasingale
Posts: 298
Joined: Mon Nov 13, 2006 2:30 pm
Location: India
Contact:

Postby suhasingale » Thu Nov 23, 2006 7:01 am

I have placed the perl script in cron.hourly but does not seem to be working...

Any other option?
User avatar
jordant
Posts: 218
Joined: Sat Sep 16, 2006 2:48 pm
Location: Perth, Western Australia

Postby jordant » Thu Nov 23, 2006 7:05 am

According to /etc/crontab, cron.hourly is run at 17 minutes past every hour, you should see something in /var/log/syslog like:

Code: Select all

/var/log/syslog:Nov 23 13:17:01 localhost /USR/SBIN/CRON[1961]: (root) CMD (   run-parts --report /etc/cron.hourly)
User avatar
suhasingale
Posts: 298
Joined: Mon Nov 13, 2006 2:30 pm
Location: India
Contact:

Postby suhasingale » Thu Nov 23, 2006 7:27 am

I can see that in the log but mails in the mail box are not getting deleted. If i run the script manually then it works fine..
User avatar
suhasingale
Posts: 298
Joined: Mon Nov 13, 2006 2:30 pm
Location: India
Contact:

Postby suhasingale » Fri Nov 24, 2006 7:42 am

Hi jordant,

Any soln for my problem?
User avatar
jordant
Posts: 218
Joined: Sat Sep 16, 2006 2:48 pm
Location: Perth, Western Australia

Postby jordant » Fri Nov 24, 2006 8:11 am

can you see it connecting to the imap server ?

theres a few things you can do, the best is probably to put some debugging code in the script. I'll do something up and post a url shortly

Return to “Tips”

Who is online

Users browsing this forum: No registered users and 1 guest