MailCleaner patch 2007080901 Pre-Release

New releases and patches informations

Moderators: Pascal, oanasutoiu, mentor, stephane, jorge, JohnMertz, bourgeois

Post Reply
olivier
Posts: 1348
Joined: Thu Jan 01, 1970 1:00 am
Contact:

MailCleaner patch 2007080901 Pre-Release

Post by olivier » Thu Aug 09, 2007 5:20 pm

Hi all,

good news, the next patch is on its way. Due to the amount of modifications (more than 15'000 lines of code being modified), I'm first delivering it as a a release candidate. Although it should be quite stable as the code base is the one that is perfectly stable and running for months now within the enterprise version, I think it is a good idea to let you play with the new version before installing it on your production system.

I'm also introducing a new binary mirror. Thanks to the Swiss Education & Research Network (http://www.switch.ch) that provide us some rcync'd space. Downloads should be quite faster and more stable than with sourceforge's web space.

now for this patch, here are "a few" major new features and improvements:

- engine has been largely optimized and now uses modules (also called prefilters). These modules are processed in a configurable order and each one can be decisive as if the message is a spam or not. This allows the system to avoid any other checks, reducing the overall load. Of course these modules must be resistant to false positives. We actually provides 4 modules with the OpenSource version, and other are already being developped.
The first one is a Baysian filter, using the well known bogofilter tool (http://www.bogofilter.org), and set up to only block spam when it's bogosity (certainty) is more than 99%. We won't provide a database for it, so you will have to build one by yourself. This is very important as any statistical datas should always be adapted and carefully built. This is even more important in that case as the module *must not* do any false positive. All the tools you need to build the database can be found in /usr/bogofilter. With good data, this module can drop more than 70% of the spam without the need to be passed further. We strongly advise you not to use shared and/or third parties databases.

The second one is a RBL module. It will call some configurable public (or private) Realtime Blocking Lists. By default, if at least two of them is hit, the message will be considered as spam without any further processing. This one is also very efficient and doesn't require too much maintenance. Just check that the DNS queries are not too slow as this could just kill performances. Also make sure you configured the "Trused IP/Networks" settings in the anti-spam panel if you have any gateway in front of MailCleaner.

The third one is a call against a special ClamAV daemon. This let you use any signature database that may contains spam and phishing definitions. Just drop them in the /var/mailcleaner/spool/clamspam directory. Some good ones can be found on the sanesecurity site (http://www.sanesecurity.co.uk/). Once again, you're on your own to get them as the Enterprise version provides enhanced databases with its own update system that couldn't be backported.

The last one (for now) is a call to a SpamAssassin daemon. This one has the same settings as the old calls to SpamAssassin it replaces. It's just faster. This module should always be the last and be both negatively (ham) and positively (spam) decisive.

all these modules, once correctly set and/or fed up, will provide a great efficiency and up to a 10X better performences. For example, a typical enterprise version provides more than 99,5% spam detection rate with a capacity of more than 300'000 messages per days.
Typical scan speed is about less than 2 seconds per message. So with a 4gig system where you can have 30 processes in parrallel , you will get a theoretical limit of 1.2Mmsgs/day. Althoug we don't have real system doing that because of redundency and failover requirements, we believe this is not so theoretical. Good news, isn't it ? Of course, such performences need some tunings. By default, MailCleaner now includes profiling times in the main engine log. This will help you diagnose which module takes more time.

- user interface has some new features: graphical statistics, new scoring system and new message previsualization window (instead of the simple reasons window), user can now also get pretty html summaries (using templates for customizations just like the web interface).

- admin interface also has some new features, mainly reflecting the new anti-spam modules.

- softwares were upgraded (ClamAV, SpamAssassin, MailScanner, etc...)

- SpamAssassin got some new plugins like PDFInfo, Botnet, SPF, etc... (in the Spamc module settings panel)

- also many small bugs were fixed

I will post the full changelog once the Release Candidate patch will be frozen.


Now to get your hands dirty, here is how to apply this patch:
First note that the code is actually in another CVS branch which will be merged in the main stream once the definitive patch is released. So you will be able to install the definitive version upon this one.
You need a fully up-to-date MailCleaner. That is you must have all previous patches already applied.
You must also have full outgoing access for ports 80, 443 and 2401.

this install process is not very different from the usual one (but be sure to use the same commands as described here, the first "cd" is important for the branch):
first, do it on your master and then on all your slaves.

> cd /usr/mailcleaner/updates/
> cvs update -dP -r post2007012801
> ../bin/apply_update.sh 2007080901

and that's it.
Depending on your bandwidth and hardware, this can take some time. It needs to recompile some perl libraries that may take a few minutes too.
There is also a spool database update that may take time if you have a lot of spam in quarantine.
You can follow the process in another shell with this command:
> tail -f /tmp/2007080901.log

Hope you enjoy. As I said, MailCleaner is evolving very rapidely (just as the spams do), and it is very hard for me to keep the Open source version up-to-date. But I promise I'm doing my best. Morover, the very few donation (in fact only one, thank generous anonymous !) we had don't allow me to justify more time on it.
So sorry for the long wait, but I hope it worth it !
have fun and don't hesitate to post your feedback on the update part of the forum.
If everything goes fine, I hope to release the stable patch next week.
Thank for your help and testing !

PS: and sorry for the long post. hope you don't felt asleep..hugh...
olivier
Posts: 1348
Joined: Thu Jan 01, 1970 1:00 am
Contact:

Post by olivier » Fri Aug 10, 2007 4:29 pm

**important**
Don't forget to populate at least the ClamSpam database of your system. (put signature files in /var/mailcleaner/spool/clamspam/ and restart engine)
You can eventually turn the module off (in the antispam panel) to avoid this but you will loose some good spam catching (specially for those pdf, doc, zip and image spams).
Post Reply