*BSD News Article 63801


Return to BSD News archive

Path: euryale.cc.adfa.oz.au!newshost.anu.edu.au!harbinger.cc.monash.edu.au!news.mira.net.au!Germany.EU.net!news.dfn.de!hookup!news.mathworks.com!uunet!not-for-mail
From: kls@MAILBOX.SLAC.Stanford.EDU (Karl L. Swartz)
Newsgroups: comp.unix.bsd.freebsd.misc,news.admin.technical,news.software.nntp
Subject: Re: Poor performance with INN and FreeBSD.
Date: 20 Mar 1996 00:52:34 -0500
Organization: Stanford Linear Accelerator Center
Lines: 75
Sender: zorch@ftp.UU.NET
Approved: zorch@uunet.UU.NET
Distribution: world
Message-ID: <DoJ1Js.H2t@unixhub.SLAC.Stanford.EDU>
References: <311F8C62.4BC4@pluto.njcc.com> <4go23v$8hp@fozzie.sun3.iaf.nl>
NNTP-Posting-Host: ftp.uu.net
Xref: euryale.cc.adfa.oz.au comp.unix.bsd.freebsd.misc:15628 news.admin.technical:1352 news.software.nntp:20746

In article <199602292319.SAA01686@po_box.cig.mot.com>,
Kevin Kadow <kadow@komondor.cig.mot.com> wrote:
>I don't know- I've been contemplating rewriting the entire operating system
>just for news servers- perhaps this is a market for the makers of the FASwerver
>dedicated NFS fileserver to get in to...

An entire news server is pretty heavy-weight, but Network Appliance, the
makers of the FAServer aka toaster, is in the news market in a sense -- their
machines make an excellent /var/spool/news.  We're in the process of
converting to such a system at SLAC, and I know of at least two other major
news sites that use them in a similar role.

Frankly, I was pretty skeptical of the idea of having an NFS-based
/var/spool/news for my news server, figuring the access over the network
would be sure death.  The appeal of having an enormous filesystem was
sufficiently tempting for me to at least give it a try, and I set up some
tests to compare a local disk with a model 1400 "toaster."  The tests
consisted of a mixture of unbatching a bunch of news and deleting selected
files in random order, with other activity running "off the clock."
Unbatching was two to three times faster using than toaster than a local
disk, while the deletes were nearly an order of magnitude faster.

One place where you probably lose (I haven't tested it yet) is with INN and
outgoing feeds which can keep up in real time.  In this case, as long as
articles stay in cache, the local disk is better, though not because of
anything to do with disk performance.

Hopefully, I'll have a paper with all the details at LISA this year.

>Of course, I was thinking more along the lines of a log filesystem with the
>overviews stored in a separate SQL-oriented database, indexed by message-id,
>starting sector, and group:article#.

I talked with Margo Seltzer a bit about news after one of her talks on the
BSD 4.4 log-structured filesystem and she thought news would be an excellent
application for LFS.  I'd like to give it a try, but so far the code is still
supposed to be pretty buggy and I haven't had the time to fiddle with it.
I've never given the NOV part of it much thought since that's never seemed to
be the bottleneck, though maybe integrating it to handle the directory
portion of the FS would be interesting.

>Once the filesystem has been filled, the oldest articles are
>automatically overwritten as new messages are added to the spool.

Nice idea, but it doesn't address varying expiration times, not to mention
seasonally varying loads.

>Want to add more capacity? Stripe the data in one 'newsfs' across multiple
>drives.

This is one of the very nice features of using a "toaster" for the job.

>Need to have different expiration times for different group hierarchies?
>break the spool into separate 'newsfs's by group.

That's awfully coarse.  What about certain subsets of hierarchies which you
want to have different expiration times, perhaps because your readers care
more about those particular groups?  There are also varying expiration
periods built into news in the form of cancelation and superseded articles
(shorter expire times) and expiration dates on individual articles
(potentially much longer expire times, such as for FAQs).

Separate filesystems also mean you have to invest more time in system
management, looking not just at how much space you expect to need overall,
but at which pockets you think you'll need it in.  One of the attractions to
the "toaster" is that you just throw more disk at it when necessary, without
worrying about where exactly it should go.  With disks being cheap and
reduction in staffing being plentiful, that's very attractive.

-- 
Karl Swartz		|INet	kls@unixhub.slac.stanford.edu
SLAC Computing Services	|	   or kls@chicago.com
1-415/926-3630		|UUCP	uunet!lll-winken!unixhub!kls  -or-  ditka!kls
(SLAC and the US Dept. of Energy don't necessarily agree with my opinions.)