Return to BSD News archive
Path: euryale.cc.adfa.oz.au!newshost.anu.edu.au!newshost.telstra.net!asstdc.scgt.oz.au!metro!metro!munnari.OZ.AU!news.ecn.uoknor.edu!news.uoknor.edu!news.nodak.edu!netnews1.nwnet.net!news.u.washington.edu!uw-beaver!nntp.cs.ubc.ca!newsxfer.itd.umich.edu!news.mathworks.com!gatech!swrinde!howland.reston.ans.net!nntp.crl.com!reason.cdrom.com!usenet From: "Jordan K. Hubbard" <jkh@FreeBSD.org> Newsgroups: comp.unix.bsd.freebsd.misc Subject: My recent comments on the Baker/ Lai paper at USENIX Date: Mon, 08 Apr 1996 15:57:27 -0700 Organization: Walnut Creek CDROM Lines: 72 Message-ID: <316999D7.167EB0E7@FreeBSD.org> NNTP-Posting-Host: time.cdrom.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Mailer: Mozilla 3.0b2 (X11; I; FreeBSD 2.2-CURRENT i386) CC: mgbaker@cs.stanford.edu Some of you may recall a recent message of mine where I called the benchmarking paper presented at the '96 USENIX "a travesty", "utter garbage" and used other highly emotionally-charged labels of that nature in denouncing what I felt to be an unfair presentation. Well, Mary Baker called me on the phone today and was, quite understandably, most upset with my characterization of her paper and the fact that I did not even see fit to present my issues with it to her personally, resulting in her having to hear about this 3rd-hand. While it must be said that I still have some issues with the testing matrix they used (most specifically for NFS client/server testing) and do not feel that the paper met *my* highly perfectionist standards for those who would dare to tread the lava flow that is public systems benchmarking, I do fully understand and acknowledge that both the words I used and the forum I used to express them (this newsgroup) were highly inappropriate and merit a full apology to Mary and her group. I can scarcely mutter about scientific integrity from one side of my mouth while issuing angry and highly emotional statements from the other, especially in a forum that gives the respondant little fair opportunity to respond. Mary, my apologies to you and your group. I was totally out of line. Those who know me also know that I've spent the last 3 years of my life working very hard on FreeBSD and, as the Public Relations guy, struggling to close the significant PR gap we're faced with. When I see something which I feel something to be unfair to FreeBSD, I react rather strongly and sometimes excessively. I've no excuse for this, but it nonetheless sometimes occurs. This last USENIX was one of many highs and lows for us, the highs coming from contacts established or renewed and the lows coming from presentations like Larry McVoy's lmbench talk which started as a reasonable presentation of lmbench itself and turned into a full pulpit-pounding propaganda piece for Linux, using comparisons that weren't even *fair* much less accurate. I did confront Larry over this one personally, and only mention it at all because it immediately followed the Baker/Lai talk and is no doubt significantly responsible for my coming away from that two hour session with a temporary hatred for anyone even mentioning the word "benchmark" - a degree of vitriol which no doubt unfairly biased me more heavily against the B/L talk than I would have been otherwise. Overall, USENIX was quite a bit more about highs than lows, and it's really only this session that stood out on the low side for me. As I told Mary, I do hope that future benchmarking attempts will at least pay us (in the free software world overall) the courtesy of a contact before the measurement runs, not to give us a chance to unfairly bias the results in any way (which would only reflect badly on us) but to simply review the testing methodologies used and make comments where we feel certain changes might be made to improve the objectivity and fairness of the results. I do think that benchmarking is important and that many types of useful "real world" data can be derived from them - our very own John Dyson puts significant amounts of time and effort into running benchmarks with Linux and other operating systems so as to "keep FreeBSD honest" in our own performance, looking for areas where they've made improvements which we have not. People like Mary Baker perform a very useful service in attempting to do the same thing for the academic world, and I definitely do not want my initially harsh words to discourage her and her students from doing further objective analysis of free and commercial operating systems - quite the contrary. It's a dirty, thankless job, but somebody has to do it. If she and her students would care to run another comparative benchmark suite using the more advanced technologies that have since been released by both the Linux and *BSD groups, endevouring also to expand the base of hardware that's being tested (since PCs are a notoriously mixed bag when it comes to issues like this), I would be pleased to extend every cooperation from the FreeBSD Project. --- - Jordan Hubbard President, FreeBSD Project