Very Long Delay in Queue

Generic questions on MailCleaner usage. Please check if your question cannot find a more specific topic below before posting here !<br>
This section is now READ ONLY, please choose the section below that fit your question !

Moderators: FlorianB, Pascal, bourgeois, mentor

rebus
Posts: 141
Joined: Thu Sep 14, 2006 3:29 pm
Location: FL - USA

Postby rebus » Thu Nov 30, 2006 1:20 pm

/etc/aliases does already exist, and has 644 permissions.

Yes, patches were applied in order.
olivier
Posts: 1348
Joined: Thu Jan 01, 1970 1:00 am
Contact:

Postby olivier » Thu Nov 30, 2006 1:32 pm

just to be sure, is "newsletterbounce@domain.com" the real address you are using ?

did you modify the exim_stage2.conf_template file ? there should never be delivery attemps at this stage because it is a queue only configuration.
Eventually try to find out these messages IDs in the exim_stage1 log file.
rebus
Posts: 141
Joined: Thu Sep 14, 2006 3:29 pm
Location: FL - USA

Postby rebus » Thu Nov 30, 2006 2:50 pm

olivier wrote:just to be sure, is "newsletterbounce@domain.com" the real address you are using ?

Well, newsletterbounce@our-domain-name.com. I shortened it to domain.com for simplicity.

olivier wrote:did you modify the exim_stage2.conf_template file ?

Yes, but only after several thousand messages started piling up. The only change was to shorten the retry interval.

olivier wrote:Eventually try to find out these messages IDs in the exim_stage1 log file.

Here is what's in the exim_stage1 logfile with that message ID. It was a single entry.

Code: Select all

2006-11-29 18:31:02 1GpUD0-0000Jj-Qb => newsletterbounce@domain.com R=filter_forward T=local_smtp H=127.0.0.1 [127.0.0.1] C="250 OK id=1GpUD0-0000Jp-T3"
olivier
Posts: 1348
Joined: Thu Jan 01, 1970 1:00 am
Contact:

Postby olivier » Thu Nov 30, 2006 3:47 pm

there must be something wrong with the exim_stage2.config_template file.

remove it, fetch it back with cvs and relaunch MailCleaner.
rebus
Posts: 141
Joined: Thu Sep 14, 2006 3:29 pm
Location: FL - USA

Postby rebus » Thu Nov 30, 2006 5:21 pm

Done. We'll see how it goes. Thanks for all the help this far.
rebus
Posts: 141
Joined: Thu Sep 14, 2006 3:29 pm
Location: FL - USA

Postby rebus » Thu Nov 30, 2006 6:09 pm

olivier wrote:A full stock MailCleaner based on a decent hardware (P4, 2Ghz, 2GB RAM), will process up to 50'000 messages per day without a glitch.
With some customizations and tunings, you can go up to 150'000 or even 200'000 messages per days.

Can you comment on what types of tuning? While we're testing, the load is around 10,000 per day. But once these spool delay issues are resolved and we are comfortable doing so, we're going to shift several other high traffic domains over which will push well above 50,000 per day. If we can tune to 75,000 or 100,000, that would delay the need to invest in a second MC server (along with avoiding the double colocation expenses).
olivier
Posts: 1348
Joined: Thu Jan 01, 1970 1:00 am
Contact:

Postby olivier » Thu Nov 30, 2006 6:19 pm

these are services we provide to our customers as it depends too much on the hardware, network topology, trafic nature and so..
This cannot be generalized.
Basicaly, it's hardware optimization (drives, ram), OS optimization (partitions shemes, limits, cpus usage, etc..) and network too (dns caching, databases synchronization, etc..).

When I said 200'000 per day, it's a low estimation because we always want to keep some security, so we don't have systems that goes beyond that, although they surely could (this is per host of course, we have customers that are working with more than millions mails per dayon their clusters).

Anyway, I always suggest people to invest into at least a second host. This is for redundency and backups.
rebus
Posts: 141
Joined: Thu Sep 14, 2006 3:29 pm
Location: FL - USA

Postby rebus » Fri Dec 01, 2006 2:59 pm

olivier wrote:this really sound like network latency. Check your dns settings, and try to disable Pyzor, DCC and Razor, in order.

I don't want to jinx it, but performance appears to be much better after disabling Pyzor.

Is Pyzor known for insufficient capacity or performace?
olivier
Posts: 1348
Joined: Thu Jan 01, 1970 1:00 am
Contact:

Postby olivier » Fri Dec 01, 2006 3:03 pm

as any network check, Pyzor is prone to latency en then slowing down the whole thing.
You should first check the MailCleaner has all required ports open to the outside as this will of course be a performence killer !

http://www.mailcleaner.org/doku.php/documentation:configuration:network
rebus
Posts: 141
Joined: Thu Sep 14, 2006 3:29 pm
Location: FL - USA

Postby rebus » Fri Dec 01, 2006 3:16 pm

We colo the equipment, but have our own dedicated connectivity from the racks and we manage our own firewalls, so I can say with confidence the required ports are open.

Another server does greylisting and RBL checks (to unload that from MC), leaving MC server to do Razor, Pyzor and DCC. I'm surprised that disabling Pyzor alone had (-seemingly-) such a dramatic performance improvement.
User avatar
jordant
Posts: 218
Joined: Sat Sep 16, 2006 2:48 pm
Location: Perth, Western Australia

Postby jordant » Fri Dec 01, 2006 5:13 pm

can you give an example of the performance increase by disabling pyzor ?
X messages per second instead of Y ??
rebus
Posts: 141
Joined: Thu Sep 14, 2006 3:29 pm
Location: FL - USA

Postby rebus » Fri Dec 01, 2006 5:19 pm

I would if I could, but I don't have that nailed down specifically. For example, though, when I turned off Pyzor I had about 250 messages in the filtering queue. New arrivals from Inbound, and departures to Outbound, were fairly even so the queue was neither building nor declining. I turned off Pyzor and within 10 minutes the filtering queue was nearly empty.

I know that is very subjective, seat-of-the-pants, but I didn't take any specific metrics.
User avatar
jordant
Posts: 218
Joined: Sat Sep 16, 2006 2:48 pm
Location: Perth, Western Australia

Postby jordant » Sat Dec 02, 2006 1:41 am

how many messages do you process per day ?
rebus
Posts: 141
Joined: Thu Sep 14, 2006 3:29 pm
Location: FL - USA

Postby rebus » Sat Dec 02, 2006 3:31 am

We're only running a few domains through MC right now, as a test. Don't want to move the whole network over until we're sure this thing is going to work as expected (hoped). As of the time of writing this (10:15 pm local time) MC has processed 12,513 messages so far today.
n0lqu
Posts: 55
Joined: Tue Nov 21, 2006 6:17 pm

Have you tried a different ClamAV?

Postby n0lqu » Sat Dec 02, 2006 6:01 am

I posted earlier how a week ago we had a problem with the filtering spool growing, similar to yours, but it seemed to have resolved itself. Well, it happened again a couple days ago and I correlated it to running ClamAV. Specifically, the first time it happened was after installing the 2006111701 update (as well as the two prior updates), which installed and activated a new ClamAV (module) scanner. At that time I tried lots of things, rebooted the system several times, etc. and eventually the problem resolved itself. I'm guessing I switched from ClamAV (module) to the old ClamAV scanner, and that's really what fixed it. Well, I ran into the problem again a few days ago, right around the time of my previous post, and I think it's likely I turned on the ClamAV (module) again at that time. I opened a new topic to discuss the ClamAV angle here: http://forum.mailcleaner.org/viewtopic.php?t=269

Needless to say, I'm very curious if your problem could be related to the ClamAV (module) as well, and what would happen if you switch to the old ClamAV rather than ClamAV (module)?

Return to “General help”

Who is online

Users browsing this forum: No registered users and 6 guests