URIRBLs needing rehabilitation

Discuss here all what concerns the MailCleaner anti-spam efficiency, share your rulesets and tips for SpamAssassin !

Moderators: Pascal, mentor, FlorianB, bourgeois

Pentangle
Posts: 85
Joined: Thu Jan 31, 2013 1:20 am
How did you hear about Mailcleaner: Googling

URIRBLs needing rehabilitation

Postby Pentangle » Thu Oct 27, 2016 7:37 pm

Hi all,

Recently i've noticed a small amount of sex spam coming through the filters. This is <200 chars with a sex subject or body and a URL.

I've written a custom rule to get rid of it, but the root cause of it coming through in the first place I think is that my URIRBLs all seem to be dying and needing rehab.

e.g.

Code: Select all

Oct 27 12:12:07 localhost MailScanner[28706]: <A> tag found in message 1bzibN-0004z7-AS from constance@onlineofis.com
Oct 27 12:12:08 localhost MailScanner[19178]: UriRBLs (1bzibN-0004z7-AS) list SURBL disabled time exceeded, rehabilitating this RBL.
Oct 27 12:12:08 localhost MailScanner[19178]: UriRBLs (1bzibN-0004z7-AS) list SPAMHAUSDBL disabled time exceeded, rehabilitating this RBL.
Oct 27 12:12:08 localhost MailScanner[19178]: UriRBLs (1bzibN-0004z7-AS) list SEMURIRED disabled time exceeded, rehabilitating this RBL.
Oct 27 12:12:18 localhost MailScanner[28706]: UriRBLs (1bzibN-0004z7-AS) Check SEMURIRED timed out and was killed
Oct 27 12:12:18 localhost MailScanner[28706]: UriRBLs (1bzibN-0004z7-AS) disabling SEMURIRED, not answering ! will retry in 120 seconds.
Oct 27 12:12:21 localhost MailScanner[28706]: Spamc result is not spam (4.5/5.0) for 1bzibN-0004z7-AS


Yet all my PreRBLs work fine, it's only the URIRBLs. I'm running my own DNS servers with no forwarders, and processing a total of about 75,000 sessions daily (which should be small enough not to trigger any thresholds, especially since out of my 5 mailcleaner servers 1 of them doesn't process any mail and 2 of them use a different DNS server to the other 2).

Any idea why this might be happening?
User avatar
Martijn
Posts: 45
Joined: Wed Aug 20, 2014 5:31 pm
How did you hear about Mailcleaner: We love to work with Mailcleaner
Location: Enter - Netherlands
Contact:

Re: URIRBLs needing rehabilitation

Postby Martijn » Thu Nov 03, 2016 12:50 pm

I had the same issue.
See: viewtopic.php?f=14&t=2403

This issue persist by one of your RBL's.
MailCleaner seems to check the lists in a certain order, where one of the lists before the one that shows a time-out is not OK.
What you can do is check them manually (see my topic) and see what list is the issue and disable that on.
Image
FlorianB
Posts: 296
Joined: Wed Apr 01, 2015 2:27 pm
How did you hear about Mailcleaner: job

Re: URIRBLs needing rehabilitation

Postby FlorianB » Sat Nov 05, 2016 3:45 pm

Hello,
Martijn gave a solution for RBL problems, and about sex messages:

Code: Select all

uri   MC_BDY_LINKPHPARG1   /\/.{0,40}\.php\?[a-zA-Z]=.{1,80}&(3AJ|Dtf|12i)=/i
score   MC_BDY_LINKPHPARG1   4.5


It does the trick perfectly for our customers, i did not notice any other parameters name.
Best regards,
Florian
Pentangle
Posts: 85
Joined: Thu Jan 31, 2013 1:20 am
How did you hear about Mailcleaner: Googling

Re: URIRBLs needing rehabilitation

Postby Pentangle » Wed Nov 09, 2016 1:26 pm

Hi Martijn,

Thanks for that - but could you please give me more information? specifically....

- where do I edit and put that file with all the URIRBLs?
- what does the syntax do exactly?
- how does it identify the bad-performing RBLs? (i.e. is it something it fixes by itself, or something it flags up so you still need to disable a rogue RBL?)

Many thanks,
Mike.
User avatar
Martijn
Posts: 45
Joined: Wed Aug 20, 2014 5:31 pm
How did you hear about Mailcleaner: We love to work with Mailcleaner
Location: Enter - Netherlands
Contact:

Re: URIRBLs needing rehabilitation

Postby Martijn » Wed Nov 09, 2016 1:37 pm

Hi Mike,
Florian gave you 2 answers :)

See my topic to have a solution for you RBL issues, and the info Florian gave you is for the sexmessages.

For the bad RBL's; yes, test them manually en if you see a RBL what is desperate slow, disabled them in your config, manually.
The mechanism for temp disable is not scripted well to let MailCleaner handle this.
Image
Pentangle
Posts: 85
Joined: Thu Jan 31, 2013 1:20 am
How did you hear about Mailcleaner: Googling

Re: URIRBLs needing rehabilitation

Postby Pentangle » Wed Nov 09, 2016 6:25 pm

Hi Martijn,

I'm sorry, I understood Florian was giving me the REGEX for the sex messages, however I already have a couple of REGEX scripts that work for that.

I'm actually only asking about your URIRBL fix - where do I fix it? what file? and what does the line "time getent hosts..." actually do?

Many thanks,
Mike.
cglmicro
Posts: 291
Joined: Thu Mar 07, 2013 2:12 am
How did you hear about Mailcleaner: google

Re: URIRBLs needing rehabilitation

Postby cglmicro » Fri Nov 11, 2016 3:30 am

FlorianB wrote:Hello,
Martijn gave a solution for RBL problems, and about sex messages:

Code: Select all

uri   MC_BDY_LINKPHPARG1   /\/.{0,40}\.php\?[a-zA-Z]=.{1,80}&(3AJ|Dtf|12i)=/i
score   MC_BDY_LINKPHPARG1   4.5


It does the trick perfectly for our customers, i did not notice any other parameters name.
Best regards,
Florian


Can someone explain this regard for me too please ? I'm curious to know what it's searching for exactly, and also what it WONT match for legitimate PHP links in good emails.
Thanks.
FlorianB
Posts: 296
Joined: Wed Apr 01, 2015 2:27 pm
How did you hear about Mailcleaner: job

Re: URIRBLs needing rehabilitation

Postby FlorianB » Sat Nov 12, 2016 12:30 am

Code: Select all

/\/.{0,40}\.php\?[a-zA-Z]=.{1,80}&(3AJ|Dtf|12i)=/i

Final 'i': caps or not for all between / and final /
\/ : char '/' himself
.{0,40} : 0 to 40 characters, any characters
\. : the '.' himself, you have to blackslash it to avoid confusion with regex '.' (would means any character)
php : usual web language (why am I writing these ? ;-) )
\? : the character '?' (regex '?' alone would means 0 or 1 times for precedent character)
[a-zA-Z] : Any character from a to z or A to Z (probably stupid because i set the final 'i' for optionnal caps, i never said i spent a lots of time on it !
= : character '='
.{1,80} : between 1 and 80 of any characters, so a short or long string, simply
& : character '&' : used to separate parameters in urls
(3AJ|Dtf|12i) : means 3AJ or Dtf or 12i : one of these three string exactly: corresponds to all three repeated parameters i found in ALL messages of these wave between beginning of October to now (you could find some more maybe, so don't hesitate to write them here)
= : character '='


So here we will catch all url using php scripts and having a parameter with name 3AJ or Dtf or 12i . Simply.
If you have preciselly this you'll hit so i gave a really high score (not usual for me, i like combination a lot more).
But to be honest, you can conflict only with other url using php script and using exactly those same parameters.... it could happens, right.... but seriously, it looks so random ! (We never had a false positive return about that rules yet, wait and see, but i'm pretty confident here !)

Best regards,
Florian Billebault
MailCleaner Team
cglmicro
Posts: 291
Joined: Thu Mar 07, 2013 2:12 am
How did you hear about Mailcleaner: google

Re: URIRBLs needing rehabilitation

Postby cglmicro » Sun Nov 13, 2016 9:23 pm

Coooool !

If I want to plan ahead, when they will replace (3aj|dtf|12i) by new variations, I modified it so it would block a larger sccop, to catch at least 4 variables=answer in the link, and the 2 variables in the middle contains at least 2 character, and the fourth variable can contain 1 or more characters, so I assume something like this could be good ?

Code: Select all

uri LOCAL_BODY_LINKPHP1 /\/.{0,40}\.php\?[a-zA-Z]=.{1,80}&.{2,80}=.{2,80}&.{2,80}=.{2,80}&.{1,80}=.{1,80}/i


And what if I don't want to block HTTPS link (never seen one from a spam) ? This should work too:

Code: Select all

uri LOCAL_BODY_LINKPHP1 /http\:\/\/.{1,80}\/.{0,40}\.php\?[a-zA-Z]=.{1,80}&.{4,80}=.{4,80}&.{4,80}=.{4,80}&.{1,80}=.{1,80}/i


Here are my sample spam links I'm playing with (I've added the space before and after each & separator for better visibility) :

Code: Select all

http://galaxmovies.com/model.php?t=144 & SeAoaRxWadBo5GniCX=d63LMnAP & Dtf=4Em & 7MVx=r4
http://schoenberg.org/hand.php?b=78 & 2LKv57SNUSu2cM33mexZt=RrbAD & Dtf=CrV & 4ug=scC
http://hotelparleinternational.com/test.php?j=78 & 2LKv57SNUSu2cM33mexZtR=rbAD & Dtf=KYz & 59YHP=e
http://blog.eec-poland.com/javascript.php?g=78 & 2LKv57SNUSu2cM3=3mexZtRrbAD & Dtf=G8v & 3=WDaja
http://fightingtommyriley.com/gallery.php?g=78 & 2LKv57S=NUSu2cM33mexZtRrbAD & Dtf=3GB & 6Y=1VrY
http://jfmsportsmassage.com/user.php?h=78 & 2LKv57SNUSu2cM33mexZtR=rbAD & Dtf=Dun & 5vL=4EL
http://www.kaliberonline.se/proxy.php?g=78 & 2LKv57SNUSu2cM3=3mexZtRrbAD & Dtf=EKq & 3PD8=4G
http://highhorizonsnepal.com/dump.php?c=78 & 2LKv5=7SNUSu2cM33mexZtRrbAD & Dtf=Kvv & 5HhD=Wx
http://netishyn.org/code.php?k=78 & 2LKv57SNUSu=2cM33mexZtRrbAD & Dtf=CUg & 2w6y=Gk
http://domostrutture.it/ini.php?i=78 & 2LKv57SNUSu2cM33mexZt=RrbAD & Dtf=5K9 & 4SaZz=x
http://mycoffeebreak.net/ajax.php?t=144 & SeAoa=RxWadBo5GniCXd63LMnAP & Dtf=Ho2 & 24aWd=6
http://mycoffeebreak.net/ajax.php?t=144 & SeAoa=RxWadBo5GniCXd63LMnAP & Dtf=Ho2 & 24aWd=6
http://japanese-parts.com/proxy.php?f=78 & 2LKv57SNUSu2=cM33mexZtRrbAD & Dtf=C11 & 7VB=WmW
http://srr-pn.com/view.php?s=126 & 2LKv5=7SNUSu2cM33mexZtRrbAD & Dtf=8M1 & Gk=nYx
http://esendaltravel.com/cli/lib.php?f=78 & 2LKv57SNUSu2cM33mexZ=tRrbAD & Dtf=F9C & 4pgzd=e
http://jayvenka.com/admin.php?i=78 & 2LKv57SNUSu2=cM33mexZtRrbAD & Dtf=7z9 & 4sjL=EC
http://eagles-travelclub.com/files.php?e=78 & 2LKv57SNUSu2cM33mexZt=RrbAD & Dtf=J7D & 2=p5WNt
http://getkbeauty.com/xml.php?s=144 & MZT=ZbVHTeu4E7uZzJRiqdX & Dtf=GM3 & 4ceX=KN


Thanks for your help !!!!

EDIT: I've just edited my own post for more visibility; was missing some /CODE.
Last edited by cglmicro on Wed Nov 16, 2016 2:22 pm, edited 1 time in total.
FlorianB
Posts: 296
Joined: Wed Apr 01, 2015 2:27 pm
How did you hear about Mailcleaner: job

Re: URIRBLs needing rehabilitation

Postby FlorianB » Wed Nov 16, 2016 11:08 am

Hello,
The logic will work yes.

"And what if I don't want to block HTTPS link" : You have to add http:, sadly spammers use all possible url combination so they write without the http://, or without the href= thinking than mail client will recognize url (what they do...). So by adding http: you'll probably reduce chance to catch future spams. You could use our rule and then add a rules for checking if there is https used anywhere in URI, then do a meta rules hitting if the first hit but not the second ? I let you check for meta-rules here: https://wiki.apache.org/spamassassin/WritingRules

But there is more chance than they add or remove parameters to fail scanners than changing their code analyzing one specific parameters. I suppose :D . So i prefered use this list and don't rely on parameters numbers too much.

About your regex:

Code: Select all

/\/.{0,40}\.php\?[a-zA-Z]=.{1,80}&.{2,80}=.{2,80}&.{2,80}=.{2,80}&.{1,80}=.{1,80}/i

You should probably replace all this '.' with

Code: Select all

 [a-zA-Z0-9]
, it will be a lot faster to search.
The first parameters should be :

Code: Select all

[a-zA-Z]=[0-9]{1,3}&

All

Code: Select all

{2,80}
could probably be reduced to 35 maximum.
Again, second, third and probably four (with some adaptation) share the same code, so they could probably be replaced by

Code: Select all

(&[a-zA-Z0-9]{2,35}=[a-zA-Z0-9]{2,35}){3}

Anyway this last is more for visibility, i can't really say without querying google if this will be an optimization too.

Keep in mind than regex are always REALLY time consuming and if they are not optimised, you'll lost SECONDS during scanning.

Don't hesitate to use http://www.regex101.com to test your regex.

Best regards,
Florian Billebault
MailCleaner Team
User avatar
Martijn
Posts: 45
Joined: Wed Aug 20, 2014 5:31 pm
How did you hear about Mailcleaner: We love to work with Mailcleaner
Location: Enter - Netherlands
Contact:

Re: URIRBLs needing rehabilitation

Postby Martijn » Wed Nov 16, 2016 11:25 am

Pentangle wrote:Hi Martijn,

I'm sorry, I understood Florian was giving me the REGEX for the sex messages, however I already have a couple of REGEX scripts that work for that.

I'm actually only asking about your URIRBL fix - where do I fix it? what file? and what does the line "time getent hosts..." actually do?

Many thanks,
Mike.


Hi Mike,

The time getent hosts command gives you the total time to check the RBL.

Code: Select all

root@mailcleaner:~# time getent hosts 184.154.35.24.psbl.surriel.com
127.0.0.2       184.154.35.24.psbl.surriel.com

real    0m0.012s
user    0m0.000s
sys     0m0.000s



So in this example we check 24.35.154.184 on the psbl.surriel.com blacklist and it took 0.012 seconds.
The reply is 127.0.0.2 what means this IP is blacklisted. Otherwise, if clean, you receive no other output that the time.

When you check all your enabled lists with this command, you can see what list took excessive time to check.
Disable that list and your problem is gone :)
Image
cglmicro
Posts: 291
Joined: Thu Mar 07, 2013 2:12 am
How did you hear about Mailcleaner: google

Re: URIRBLs needing rehabilitation

Postby cglmicro » Thu Nov 17, 2016 3:52 am

Thanks Florian, will play with this.
cglmicro
Posts: 291
Joined: Thu Mar 07, 2013 2:12 am
How did you hear about Mailcleaner: google

Re: URIRBLs needing rehabilitation

Postby cglmicro » Sun Nov 20, 2016 2:48 pm

Hi Florian.

My regex catch a facebook legitimate invitation :

Code: Select all

uri      LOCAL_BODY_LINKPHP1 /\/.{0,40}\.php\?[a-zA-Z]=[a-zA-Z0-9]{1,40}(&[a-zA-Z0-9]{1,40}=[a-zA-Z0-9]{1,40}){3}/i


So if someone else read me, better use the original proposed:

Code: Select all

uri      LOCAL_BODY_LINKPHP1 /\.php\?[a-zA-Z0-9]=.{1,80}&(3AJ|Dtf|12i)=/i
Pentangle
Posts: 85
Joined: Thu Jan 31, 2013 1:20 am
How did you hear about Mailcleaner: Googling

Re: URIRBLs needing rehabilitation

Postby Pentangle » Mon Dec 05, 2016 4:45 pm

Hi guys,

Thank you ever so much for the full explanation. I've just checked a bunch of spam I needed to block and they all had one of those 3 codes in them. I'll now keep an eye out for whether I get any more of that spam and whether I can check my URIRBL performance. Cheers!

Mike.

Return to “Filter efficiency”

Who is online

Users browsing this forum: No registered users and 3 guests