Email with blacklisted URI not flagged

Discuss here all what concerns the MailCleaner anti-spam efficiency, share your rulesets and tips for SpamAssassin !

Moderators: FlorianB, Pascal, bourgeois, mentor

Livio
Posts: 4
Joined: Thu Jun 06, 2013 9:11 am
How did you hear about Mailcleaner: Google

Email with blacklisted URI not flagged

Postby Livio » Wed Jul 16, 2014 9:28 am

Hi all,

once a day my antispam server receives a lot of email like this:
Brain Fog, Slight Constipation, Heart Problems, Lack of Libido?

Over 75,000 people just like you are ecstatic over this information that can reverse these symptoms.

Change your body and quality of life in under 72 hours.

Find out how here: http://othough.me/Z04obaygMZtMNurWPuakT ... K99MQOAg==

























home assigned as his, for no record of lands bought by him can be foundContinental troops were subjected, who found themselves prisoners of thecold and suffering that was the portion of Washington's army throughouttheir right to the lands; and there these hardy pioneers were swiftlywere present became converts to the faith of Fox and Edmundson; andIsland. Cornwallis' Army was marching northward from Wilmington, andAs early as 1754, before the little settlement began to assume the airsresided within her borders, and at their homes, for lack of otherdistressed compatriots in far away Boston. Gratefully was the donationhated and dreaded Arnold, whose expedition up the James had beentown commons.friend, Mrs. Margaret Palmer; and the "Old Marsh House" is stillIn spite of the troubles and perplexities that beset Gregory in the fallwilds was Currituck noted in the early days of our State. This county,handsome building for those days. As I recall my father's description ofhospitable host of the occasion was doubtless waiting to receive theSwanns; and the long list of prominent families who afterwards livedThe lover followed his fair one across the seas, and entered in disguisestreams that flow into Albemarle Sound. Of none of these, however, canAuthor: Catherine Albertsonhonor being conferred upon him on account of his gallant conduct atbeing Dempsey Burgess, Lieutenant-Colonel, Joshua Campbell, Major, andmention of any attempt on the part of the settlers to provide a schoolatmosphere of gloom and terror which the poet Hood has so graphically


-
Research Promo Center
700 N Valley St Suite B
Anaheim CA 92801
-
Stop receiving these messages: http://othough.me/Z190MvCpJspBMOfaNPnhR ... au4nM5ch4=
--




and it let it through.

Every URI in every email is blacklisted but the UriRBLs filter is not triggered.
I tried to switch on Graylisting but all of these email are not deleyed in any way.

Here the log of the email:
Incoming MTA stage: 2014-07-15 23:25:11 1X7ADn-000606-1W DKIM: d=othough.me s=public c=relaxed/relaxed a=rsa-sha1 i=HealthExpose@othough.me [verification succeeded]
2014-07-15 23:25:11 1X7ADn-000606-1W <= HealthExpose@othough.me H=20.rejseregypten.net (othough.me) [23.229.56.84] P=esmtp S=3684 id=0.0.0.25.1CFA07207B32F14.B7BDF@othough.me
2014-07-15 23:25:11 1X7ADn-000606-1W => user@mydomain.ext R=filter_forward T=local_smtp S=3758 H=127.0.0.1 [127.0.0.1] C="250 OK id=1X7ADn-00060A-Ic"
2014-07-15 23:25:11 1X7ADn-000606-1W Completed
Filtering MTA stage: 2014-07-15 23:25:11 1X7ADn-00060A-Ic <= HealthExpose@othough.me H=(mail.mydomain.ext) [127.0.0.1] P=esmtp S=3684 id=0.0.0.25.1CFA07207B32F14.B7BDF@othough.me
Filtering Engine: Jul 15 23:25:20 localhost MailScanner[19783]: Spamc result is not spam (0.0/5.0) for 1X7ADn-00060A-Ic
Jul 15 23:25:20 localhost MailScanner[19783]: Message 1X7ADn-00060A-Ic from 23.229.56.84 (healthexpose@othough.me) to emu.it is not spam, Spamc (score=0.0, required=5.0, )
Jul 15 23:25:20 localhost MailScanner[19783]: Profiled SpamCheck for message 1X7ADn-00060A-Ic: (PreRBLs_Check:0.5483s) (Prefilters:7.5494s) (SpamCacheCheck:0.0005s) (Spamc_Check:6.6219s) (UriRBLs_Check:0.3781s)
Jul 15 23:25:20 localhost MailScanner[19783]: Count updated for 1X7ADn-00060A-Ic
Jul 15 23:25:20 localhost MailScanner[19783]: Count updated in daemon for 1X7ADn-00060A-Ic
Outgoing stage: 2014-07-15 23:25:20 1X7ADn-00060A-Ic => user@mydomain.ext R=filter_forward T=remote_smtp S=4072 H=192.168.0.26 [192.168.0.26] X=TLSv1:RC4-SHA:128 C="250 2.6.0 <0.0.0.25.1CFA07207B32F14.B7BDF@othough.me> [InternalId=47794396069920] Queued mail for de"
2014-07-15 23:25:20 1X7ADn-00060A-Ic Completed


Attached the configuration of UriRBLs.
Attachments
2014-07-16 10_23_14-MailCleaner.jpg
UriRBLs config
2014-07-16 10_23_14-MailCleaner.jpg (33.4 KiB) Viewed 9095 times
jimp
Posts: 14
Joined: Tue Aug 13, 2013 5:27 pm
How did you hear about Mailcleaner: Google searching for antispam gateway

Re: Email with blacklisted URI not flagged

Postby jimp » Thu Oct 09, 2014 7:59 pm

I have been analyzing my MailCleaner performance the past few days and I am noticing the same thing. It appears my UriRBLs are never considered. I have it requiring 2 hits, position 5, decisive, and enabled. I have seen spam after spam get past the PreRBLs and all other stages, but when I take the URL in the email (ASAP, within a couple minutes) of receiving it and directly asking the three UriRBLs, Spamhaus and Uribl both report the URL is in the block list. (I haven't seen Surbl.org report ANY URL's block yet. Hmmm...)

I have tried restarting everything. I believe it's configured correctly, and the only evidence in the logs of UriRBLs is the initialization and it gets mentioned per message. However, in the filtering log it never shows up separate from what I am showing below. PreRBLs show up frequently, but no UriRBLs which makes me think it's not really active in the MailScanner configuration.

Code: Select all

Oct  9 13:01:08 localhost MailScanner[24365]: UriRBLs module initializing...
Oct  9 13:01:08 localhost MailScanner[24365]: UriRBLs using 3 RBLs (SURBL, SPAMHAUSDBL, URIBL)
Oct  9 13:01:08 localhost MailScanner[24365]: UriRBLs loaded 0 whitelisted domains
Oct  9 13:01:08 localhost MailScanner[24365]: UriRBLs loaded 2211 TLDs
Oct  9 13:01:08 localhost MailScanner[24365]: UriRBLs loaded 41 local domains
Oct  9 13:01:15 localhost MailScanner[24365]: Profiled SpamCheck for message 1XcI1T-0006Ky-52: (ClamSpam_Check:0.0289s) (NiceBayes_Check:0.0003s) (PreRBLs_Check:3.2116s) (Prefilters:7.4028s) (SpamCacheCheck:0.0009s) (Spamc_Check:3.3746s) (TrustedSources_Check:0.0446s) (UriRBLs_Check:0.7409s)
Oct  9 13:01:23 localhost MailScanner[24000]: Profiled SpamCheck for message 1XcI1U-0006LE-Q1: (ClamSpam_Check:0.068s) (NiceBayes_Check:0.0006s) (PreRBLs_Check:5.9651s) (Prefilters:11.3146s) (SpamCacheCheck:0.001s) (Spamc_Check:4.782s) (TrustedSources_Check:0.1798s) (UriRBLs_Check:0.3172s)

(but UriRBLs never, ever reports a hit)

I think UriRBLs will catch practically all remaining spam that is getting through, even with requiring 2 hits to be conservative. How can I debug this further?
del
Posts: 497
Joined: Mon Mar 11, 2013 7:42 am
How did you hear about Mailcleaner: google
Location: Germany

Re: Email with blacklisted URI not flagged

Postby del » Thu Oct 09, 2014 8:22 pm

Hmmm...

Did you modify the max. message size on the UriRBL page?

An UriRBL hit looks like this:
UriRBLs result is spam (newsletterclickthrough.com:URIBL,SURBL - profxgroup.co:URIBL) for 1Xc0Wk-0007nI-P2
jimp
Posts: 14
Joined: Tue Aug 13, 2013 5:27 pm
How did you hear about Mailcleaner: Google searching for antispam gateway

Re: Email with blacklisted URI not flagged

Postby jimp » Thu Oct 09, 2014 8:49 pm

del wrote:Did you modify the max. message size on the UriRBL page?

Yes. I recall the initial value for all modules was 50KB, and spam with images was bypassing all modules. I set it to 50MB so our maximum message size would be fully scanned. Is this a problem for UriRBL?

del wrote:An UriRBL hit looks like this:
UriRBLs result is spam (newsletterclickthrough.com:URIBL,SURBL - profxgroup.co:URIBL) for 1Xc0Wk-0007nI-P2

I appreciate you showing that. I have searched the filter log for it and it does exist, although rarely.

My last log message matching that today was over 2 hrs ago.

Code: Select all

Oct 9 12:26:44 localhost MailScanner[18618]: UriRBLs result is spam (lormaneducation.net:URIBL - lormaneducation.com:URIBL) for 1XcHU5-0002aK-In

We have already processed 15k messages today, so I think that's odd to see it so infrequently. Additionally, I have seen SpamAssassin hit the URIBL rule on plenty of messages that UriRBLs was completely silent. The UriRBLs module is #5 decisive, SpamC is #6 decisive (but the URIBL and SPAMHAUS rules don't score high enough to spam-it). Even with SpamAsssassin flags URL content, the UriRBL module is silent in the logs.
del
Posts: 497
Joined: Mon Mar 11, 2013 7:42 am
How did you hear about Mailcleaner: google
Location: Germany

Re: Email with blacklisted URI not flagged

Postby del » Fri Oct 10, 2014 10:14 am

This may be a problem, I think I've seen some bugs here but don't know 100%.
Try to run UriRBL with 500000 bytes please
jimp
Posts: 14
Joined: Tue Aug 13, 2013 5:27 pm
How did you hear about Mailcleaner: Google searching for antispam gateway

Re: Email with blacklisted URI not flagged

Postby jimp » Wed Oct 15, 2014 12:00 am

del wrote:This may be a problem, I think I've seen some bugs here but don't know 100%.
Try to run UriRBL with 500000 bytes please

I configured that last Thursday, but it still appears to be broken. Today I have found "UriRBLs result is " in the log 17 times. Never does it find "UriRBLs result is not spam" -- It always says "UriRBLs result is spam" if matched. And only 17 times. 20k messages today already.

A sample:

Code: Select all

Oct 14 15:44:59 localhost MailScanner[1558]: New Batch: Found 2 messages waiting
Oct 14 15:44:59 localhost MailScanner[1558]: New Batch: Scanning 1 messages, 14528 bytes
Oct 14 15:44:59 localhost MailScanner[1558]: Virus and Content Scanning: Starting
Oct 14 15:44:59 localhost MailScanner[1558]: <A> tag found in message 1Xe8xm-00007r-St from languagelearning@dailycollegeschoalrsmn.com
Oct 14 15:44:59 localhost MailScanner[1558]: HTML Img tag found in message 1Xe8xm-00007r-St from languagelearning@dailycollegeschoalrsmn.com
Oct 14 15:44:59 localhost MailScanner[1558]: Spam Checks: Starting
Oct 14 15:44:59 localhost MailScanner[1558]: NiceBayes has been disabled (no database ?)
Oct 14 15:45:04 localhost MailScanner[1558]: PreRBLs result is not spam (SPAMHAUS) for 1Xe8xm-00007r-St
Oct 14 15:45:05 localhost MailScanner[1558]: UriRBLs result is spam (bit.ly:SPAMHAUSDBL - dailycollegeschoalrsmn.com:SURBL,SURBL,SPAMHAUSDBL,URIBL) for 1Xe8xm-00007r-St
Oct 14 15:45:05 localhost MailScanner[1558]: Message 1Xe8xm-00007r-St from 202.39.112.51 (languagelearning@dailycollegeschoalrsmn.com) to ***redacted***.com is spam, PreRBLs (SPAMHAUS), UriRBLs (bit.ly:SPAMHAUSDBL - dailycollegeschoalrsmn.com:SURBL,SURBL,SPAMHAUSDBL,URIBL)
Oct 14 15:45:05 localhost MailScanner[1558]: Profiled SpamCheck for message 1Xe8xm-00007r-St: (ClamSpam_Check:0.0497s) (NiceBayes_Check:0.0004s) (PreRBLs_Check:5.2448s) (Prefilters:6.1991s) (SpamCacheCheck:0.0013s) (TrustedSources_Check:0.0507s) (UriRBLs_Check:0.8513s)
Oct 14 15:45:05 localhost MailScanner[1558]: Spam Checks: Found 1 spam messages
Oct 14 15:45:05 localhost MailScanner[1558]: Spam Actions: message 1Xe8xm-00007r-St actions are deliver
Oct 14 15:45:05 localhost MailScanner[1558]: Uninfected: Delivered 1 messages
Oct 14 15:45:05 localhost MailScanner[1558]: Count updated for 1Xe8xm-00007r-St
Oct 14 15:45:05 localhost MailScanner[1558]: Count updated in daemon for 1Xe8xm-00007r-St

EDIT: "UriRBLs result is spam" is actually wrong here. I have the UriRBLs module set to require 2 lists, not just one...

I next looked for "PreRBLs result is not spam" and found an interesting case that didn't try UriRBLs at all.

Code: Select all

Oct 14 15:50:30 localhost MailScanner[3461]: New Batch: Scanning 1 messages, 10827 bytes
Oct 14 15:50:30 localhost MailScanner[3461]: Virus and Content Scanning: Starting
Oct 14 15:50:30 localhost MailScanner[3461]: <A> tag found in message 1Xe936-0000Nl-0N from sleepapnea@grucetch.co.uk
Oct 14 15:50:30 localhost MailScanner[3461]: HTML Img tag found in message 1Xe936-0000Nl-0N from sleepapnea@grucetch.co.uk
Oct 14 15:50:30 localhost MailScanner[3461]: Spam Checks: Starting
Oct 14 15:50:30 localhost MailScanner[3461]: NiceBayes has been disabled (no database ?)
Oct 14 15:50:35 localhost MailScanner[3461]: PreRBLs result is not spam (SPAMHAUS) for 1Xe936-0000Nl-0N
Oct 14 15:50:39 localhost MailScanner[3461]: Spamc result is spam (17.5/5.0) for 1Xe936-0000Nl-0N
Oct 14 15:50:39 localhost MailScanner[3461]: Message 1Xe936-0000Nl-0N from 198.89.90.235 (sleepapnea@grucetch.co.uk) to ***redacted***.org is spam, PreRBLs (SPAMHAUS), Spamc (score=17.5, required=5.0, MIME_HTML_ONLY 0.7, HTML_MESSAGE 0.0, T_URIBL_BLACK_OVERLAP 0.0, RAZOR2_CHECK 0.9, BAYES_999 0.5, T_REMOTE_IMAGE 0.0, RAZOR2_CF_RANGE_E8_51_100 1.9, URIBL_BLACK 1.7, RAZOR2_CF_RANGE_51_100 0.5, RDNS_NONE 4.0, URIBL_JP_SURBL 1.2, BAYES_99 3.5, URIBL_DBL_SPAM 2.5)
Oct 14 15:50:39 localhost MailScanner[3461]: Profiled SpamCheck for message 1Xe936-0000Nl-0N: (ClamSpam_Check:0.023s) (NiceBayes_Check:0.0004s) (PreRBLs_Check:5.3007s) (Prefilters:9.0698s) (SpamCacheCheck:0.0006s) (Spamc_Check:3.2772s) (TrustedSources_Check:0.0641s) (UriRBLs_Check:0.4028s)
Oct 14 15:50:39 localhost MailScanner[3461]: Spam Checks: Found 1 spam messages
Oct 14 15:50:39 localhost MailScanner[3461]: Spam Actions: message 1Xe936-0000Nl-0N actions are deliver
Oct 14 15:50:39 localhost MailScanner[3461]: Uninfected: Delivered 1 messages
Oct 14 15:50:39 localhost MailScanner[3461]: Count updated for 1Xe936-0000Nl-0N
Oct 14 15:50:39 localhost MailScanner[3461]: Count updated in daemon for 1Xe936-0000Nl-0N

PreRBLs said "not spam" (2 required, only hit 1 list). Then Spamc resulted in "is spam" but UriRBLs didn't report anything between there. Note that Spamc did have URIBL hits in the score, though. Also the runtime difference between a successful UriRBLs and an absent one is about 0.5 seconds longer when it actually appears to do anything.

I can look at the code if you can point me to the correct folder with the code for how the spam filter stages translate into calling the individual tools. I'm happy to help any way I can.
dimitric@haco.com
Posts: 50
Joined: Wed Sep 11, 2013 2:06 pm
How did you hear about Mailcleaner: article

Re: Email with blacklisted URI not flagged

Postby dimitric@haco.com » Fri Oct 17, 2014 7:38 am

I have exactly the same behaviour,
uribl:
module enabled,
decisive,
position 4
maximum check time: 20 seconds
max message size: 500000 bytes
url hits to be spam: 1
using these rbls:
multi.surbl.org
multi.uribl.com
dbl.spamhaus.org
resolve url shorteners enabled.

all the spam I get with links in them are in the uri bls,
but mailcleaner seems to ignore this module.
In the tracing, I never see any mention of uri check.
cglmicro
Posts: 291
Joined: Thu Mar 07, 2013 2:12 am
How did you hear about Mailcleaner: google

Re: Email with blacklisted URI not flagged

Postby cglmicro » Mon Oct 20, 2014 12:41 am

Same here: many bad URL are getting throught, any updates here ?
dimitric@haco.com
Posts: 50
Joined: Wed Sep 11, 2013 2:06 pm
How did you hear about Mailcleaner: article

Re: Email with blacklisted URI not flagged

Postby dimitric@haco.com » Thu Oct 23, 2014 3:27 pm

any updates?
jimp
Posts: 14
Joined: Tue Aug 13, 2013 5:27 pm
How did you hear about Mailcleaner: Google searching for antispam gateway

Re: Email with blacklisted URI not flagged

Postby jimp » Thu Oct 23, 2014 7:26 pm

No one seems to know what's going on. I haven't ventured into the code yet. Any tips on where to begin, I would appreciate it. I haven't tried the latest CVS code, although we are up to date within the past month.
dimitric@haco.com
Posts: 50
Joined: Wed Sep 11, 2013 2:06 pm
How did you hear about Mailcleaner: article

Re: Email with blacklisted URI not flagged

Postby dimitric@haco.com » Tue Oct 28, 2014 8:15 am

Any of the devs knows what's going on?
This is getting really bad.
dimitric@haco.com
Posts: 50
Joined: Wed Sep 11, 2013 2:06 pm
How did you hear about Mailcleaner: article

Re: Email with blacklisted URI not flagged

Postby dimitric@haco.com » Wed Nov 26, 2014 8:35 am

anyone?
Julien
Posts: 31
Joined: Mon Jul 14, 2014 8:43 am
How did you hear about Mailcleaner: job

Re: Email with blacklisted URI not flagged

Postby Julien » Fri Nov 28, 2014 4:35 pm

Hi,

The problem often come with DNS lag / timeout.
Could you check the response time of your DNS server when you request for an URIrbl ?

If you raise the "Maximum check time", the mail pass throught this filter.

Julien.
dimitric@haco.com
Posts: 50
Joined: Wed Sep 11, 2013 2:06 pm
How did you hear about Mailcleaner: article

Re: Email with blacklisted URI not flagged

Postby dimitric@haco.com » Mon Dec 01, 2014 8:47 am

Thanks a lot for the tip,
I upped the time to 60seconds,
and post back in 2 days if I see improvement or not.
dimitric@haco.com
Posts: 50
Joined: Wed Sep 11, 2013 2:06 pm
How did you hear about Mailcleaner: article

Re: Email with blacklisted URI not flagged

Postby dimitric@haco.com » Wed Dec 03, 2014 8:09 am

It works!
Thanks a lot for your help.
No more complaints from users since many weeks 8)

Return to “Filter efficiency”

Who is online

Users browsing this forum: No registered users and 2 guests