Return to BSD News archive
#! rnews 4729 bsd
Newsgroups: comp.unix.bsd.freebsd.misc
Path: euryale.cc.adfa.oz.au!newshost.anu.edu.au!harbinger.cc.monash.edu.au!news.rmit.EDU.AU!news.unimelb.EDU.AU!munnari.OZ.AU!spool.mu.edu!howland.reston.ans.net!newsfeed.internetmci.com!in2.uu.net!news.new-york.net!spcuna!spcvxb!terry
From: terry@spcvxb.spc.edu (Terry Kennedy, Operations Mgr.)
Subject: Re: News &FreeBSD
X-Nntp-Posting-Host: spcvxa.spc.edu
References: <819230783.26685@loddon.demon.co.uk> <4b2pkd$o9b@news.quanta.com> <4b6u0o$hpi@tzlink.j51.com> <4b9b66$342@sidhe.hsc-sec.fr> <1995Dec20.163332.1@spcvxb.spc.edu> <4baiid$6o2@vanbc.wimsey.com>
Sender: news@spcuna.spc.edu (Network News)
X-Nntp-Posting-User: TERRY
Organization: St. Peter's College, US
Date: Thu, 21 Dec 1995 07:26:49 GMT
Message-ID: <1995Dec21.022649.1@spcvxb.spc.edu>
Lines: 69
Distribution: inet
In article <4baiid$6o2@vanbc.wimsey.com>, jhenders@wimsey.com (John Henders) writes:
[Let me address your questions out of sequence for a more coherent answer 8-]
>>[...] response is nearly instant, while many of the INN customers exhibit a
>>second or so of "think time" before responding. This repeats for subsequent
>>articles, so it isn't just a one-time startup thing.
>
> This is because INN is checking to see if it has the article before
> accepting it, which c-news and nntp don't do. This, as you note, makes
> the nntp performance considerably faster, but can bog down the machine
> later when unbatching.
Actually, C News + NNTP does check the history file to see if it should
accept an article. The problem is that C News can take a good deal of time
between NNTP presenting it an article and it actually making it into the
history file. INN does this as a single operation, so it doesn't have this
window.
However, any server (INN or C News + NNTP) is going to have problems keep-
ing up if it takes a second to verify that it wants the article you're offer-
ing it - assuming 0 time for article transfer, that's going to limit you to
86400 articles/day, which is far short of a full feed these days.
>> In fact, we even run C News on our news machines, though that's mostly
>>for historical reasons. We exchange news with about 10 backbone servers and
>>have no problem keeping up (which is more than I can say for some of those
>>backbone servers 8-).
>
> How do you handle the duplicates with this setup? It would seem that
> with multiple feeds you'd spend a lot of cycles unbatching articles you
> already had.
There's a number of things we do. First, we do our best to keep up with the
incoming feeds - at the moment, we're 60 articles behind our incoming feeds
(meaning that there's only 60 articles received but not yet in history/spool).
Second, our feeds are temporally dispersed enough that we don't get offered
the same article from multiple hosts closer than about 30 seconds apart. Third,
we run a heavily modified msgidd to deal with the received-but-not-in-history
articles.
You can get into a situation which is similar to falling into a black hole -
the harder you try to get out, the more impossible it is. If you fall far
enough behind that you start accepting articles you've accepted from other
sites, your unbatching gets further and further behind, leading to accepting
ever more duplicates. The trick is to keep a close enough watch on your back-
log and throw more resources at the problem when needed. For C News, the most
important thing seems to be the filesystem buffer cache. On my backbone server
I'm currently running with a 32MB buffer cache.
> I agree with you, however, that a site with a single feed and limited
> resources would probably be better off with c-news, however, if a site
> is attempting to get whatever you consider a "full feed" nowadays, I
> also think it's unwise to rely on a single incoming feed, unless it's
> very well run. As well, any site doing multiple large outgoing nntp
> feeds would be well advised to look at nntplink in stdin mode, despite
> what the INN faq says, as the performance improvements well outweight
> the few drawbacks.
I hope that our customers consider us very well run 8-). Certainly the num-
ber of people at backbone servers that asked to exchange news with us indicates
that we're doing something right.
There doesn't seem to be a good place to learn about managing large news
servers - there's a lot of expertise out there, but much of it was learned
the hard way by individual sites. That generates a goodly amount of "my way
works, so it must be the One True Way", which certainly isn't the case...
Terry Kennedy Operations Manager, Academic Computing
terry@spcvxa.spc.edu St. Peter's College, Jersey City, NJ USA
+1 201 915 9381 (voice) +1 201 435-3662 (FAX)