Return to BSD News archive
Path: euryale.cc.adfa.oz.au!newshost.carno.net.au!harbinger.cc.monash.edu.au!news.mel.connect.com.au!munnari.OZ.AU!news.ecn.uoknor.edu!solace!dataphone!www.nntp.primenet.com!nntp.primenet.com!enews.sgi.com!news.sgi.com!newshub.sdsu.edu!news1.best.com!nntp1.best.com!usenet
From: dillon@flea.best.net (Matt Dillon)
Newsgroups: comp.mail.sendmail,comp.mail.smail,comp.unix.bsd.freebsd.misc
Subject: Re: Sendmail vs. Smail...
Date: 10 Dec 1996 06:33:53 GMT
Organization: BEST Internet Communications, Inc.
Lines: 115
Message-ID: <58j08h$1mn@nntp1.best.com>
References: <57tf61$gq7@raven.eva.net> <58be63$eu@stdismas.bogon.com> <58i45b$apn@nntp1.best.com> <58i81f$ajf@crystal.WonderWorks.COM>
NNTP-Posting-Host: flea.best.net
Xref: euryale.cc.adfa.oz.au comp.mail.sendmail:34974 comp.mail.smail:2672 comp.unix.bsd.freebsd.misc:32328
:In article <58i81f$ajf@crystal.WonderWorks.COM>,
:Kyle Jones <kyle_jones@wonderworks.com> wrote:
:>Matt Dillon <dillon@flea.best.net> wrote:
:> > [...]
:> > * Turn ON ForkEachJob ... you don't really have a choice.
:> > If you don't, the sendmail's running the queue can build
:> > to five times their normal RSS and effectively run the
:> > machine out of memory. Unfortunately, turning
:> > ForkEachJob off also blows the connection cache... oh
:> > well. Maybe a later version of sendmail will allow one
:> > to specify how many jobs per fork one can run :-).
:>
:>You can combat this growth with smaller queues. Take fifteen
:>thousand queued message and spread them over 150 directories and
:>queue runs aren't so bad. The sendmail process runs out of jobs
:>before it gets really fat. Limiting queue size is a good idea
:>anyway because of the linear directory searches combined with
:>directory update locking that can keep open() and unlink()
:>blocked for a long time.
I've found that limiting the queue size causes some pieces
of mail to sit in there for hours to perfectly valid destinations
while others breeze through in minutes. I've pretty much
given up on it as a means to limit queue runs.
I've used the multiple-queue approach before too, but threw
it away when MinQueueAge came out... you can get nearly
the same effect simply by increasing MinQueueAge, or running
X sendmail's with one MinQueueAge value and Y sendmail's with
another. The only reason I was using the multiple-queue approach
at all was due to some catastrophic cascade failures due to
linear searches of the spool direct by the kernel for
file create/remove. MinQueueAge and the .hoststat stuff pretty
much fixed it, and the problem went away entirely when we switched
to FreeBSD.
The huge queues also created problems for us when we were
still using -q5m no, -q15m, no... -q30m, no ... :-) It never
worked. Basically this sort of cascade failure occurs
when the directory gets large enough such that -qXXXm
winds up starting more sendmail's then the system can deal
with, all due to the large queue, and file create/remove starts
to create clogged directories (processes sitting on filesystem
locks trying to update the directory). God, what a mess
that was.
The current system gets some pretty spectacular tests...
whenever the network to a particular huge.. actually very
huge but unnamed provider barfs (heh), our mail queues shoot
up at a rate of 3000 messages an hour. The pagers start going
off when it hits 10,000 messages, and I start praying when
it passes 30,000. It taught me a very important lesson:
NEVER, NEVER mount /var/spool/mqueue as it's own partition...
you not only can't rename it, you can't clear the blocks
allocated to the directory either!
Oh, another good reason to run with ForkEachJob turned on...
it allows you to kill sendmail 'nicely'... you simply kill
the daemon and the children of the queue-running maintenance
program... the children of the children are the ones doing
the actual queue processing, and you let them finish up
their current queue file and exit normally. Poof, you've
brought down your mail system without a single repeated
message! Nice!
:> > There's a story here: We once received a mail bomb where
:> > the bomber sent the entire message body as a header.
:> > There were only about 50 of these messages in the mail
:> > queue, but they caused the sendmail's running the queue
:> > to grow to about 8 MBytes RSS. The machine, with 128 MB
:> > of ram, started to swap! Holy cow!
:>
:>Groovy. sendmail used to crash when faced with such headers,
:>freeing memory in the process. Sometimes sendmail bugs are you
:>friends. :)
Tell me about it! There was one spam that had a badly munged
address that caused sendmail to:
* make connection to destination
* send the email message to the destination
* then crash before it could remove the queue file
* repeat ...
OUCH! The real clincher: It didn't update the queue file
either, so MinQueueAge had no effect on the retries.
I call it the auto-remote-spamming tool. I've got the
blown up address save away in case I ever need to use
it on someone ;-)
:>
:> > I've also tried to turn off ForkEachJob with the latest
:> > 8.8.4 release... it doesn't work. The sendmail's still
:> > build up to around a 3 MB RSS and kill the machine.
:>
:>You might be able to get by with fewer queue run processes if you
:>run some of them with drastically smaller connection timeout
:>values. If a host doesn't respond in 15 seconds. it probably
:>isn't going to respond at all. Instead of (typically) waiting 60
:>seconds, give up and move on. A complete pass of the queue takes
:>much less time, and responsive hosts are rewarded by getting
:>their mail delivered sooner. The sluggish hosts will be serviced
:>eventually by the queue runs that use the RFC 1123 minimum
:>timeouts. You should not let the fast queue runnners write into
:>the persistent host status cache, or slow hosts will never get
:>their mail.
Ah, an interesting idea. I may experiment with this some.
.hoststat seems to have made the DNS delays an order of magnitude
less invasive, and sendmail 8.8.x has some nifty options for
connect and initial-connect timeouts.
-Matt