Return to BSD News archive
Xref: sserve comp.os.linux:36180 comp.bugs.2bsd:398 comp.bugs.4bsd:1948 comp.lang.tcl:3533 comp.os.386bsd.bugs:571 comp.os.linux.announce:563
Newsgroups: comp.os.linux,comp.bugs.2bsd,comp.bugs.4bsd,comp.lang.tcl,comp.os.386bsd.bugs,comp.os.linux.announce
Path: sserve!newshost.anu.edu.au!munnari.oz.au!news.Hawaii.Edu!ames!agate!doc.ic.ac.uk!warwick!uknet!mcsun!news.funet.fi!hydra!klaava!wirzeniu
From: andyo@ora.com (Andy Oram)
Subject: Testing GNU utilities and Linux -- please join us
Message-ID: <1993Apr26.214433.18601@klaava.Helsinki.FI>
Followup-To: comp.os.linux
Keywords: testing
Sender: wirzeniu@klaava.Helsinki.FI (Lars Wirzenius)
Reply-To: andyo@ora.com
Organization: O'Reilly & Associates, Inc.
Date: Mon, 26 Apr 1993 21:44:33 GMT
Approved: linux-announce@tc.cornell.edu (Lars Wirzenius)
Lines: 177
If you have built up your own tests for system calls, library
routines, networking protocols, or common utilities, this posting
offers you an opportunity to put them to good use. You can do a good
deed for the Linux community, the Berkeley UNIX community, and the
wider world of free software users.
Also, if you would like to learn some good test tools and strategies,
and devote some time on a volunteer basis to writing tests -- perhaps
a couple of weeks full-time, or a couple hours per week over a period
of several months -- then here is a project you should get involved
in.
I am helping to coordinate a test effort for Linux. In the following
four sections of this message I will describe the goals, the process,
the people trying to do it, and what you can do to help.
Goals
Linux and BSD share a number of libraries and utilities, both
because BSD software was ported to Linux, and because several free
software (GNU) utilities have been ported to both operating
systems.
Linux can show unexpected differences and failures across different
releases of the OS and different ports of utilities; some PC
hardware combinations may cause additional problems. People who
hope to base their own commercial products on Linux would like to
see proof that it is robust, portable, and standard-conforming. The
problem is that cross-platform testing is very hard to do, and few
decent test suites exist either for free software or for UNIX.
(Look at all the differences you find among systems that are
SVID-conforming, particularly at higher levels such as the
utilities.)
Therefore, a number of us who are interested n Linux have decided to
enter the modern age of formal, automated software testing -- but in
a manner that is proper for the free software community.
Using DejaGnu, a test platform developed by Cygnus Support and
placed under the GPL, we want to collect the best tests we can from
people everywhere, and integrate them into suites that any user can
download, run, and interpret with a few commands. DejaGnu already
runs under BSD, and Cygnus is porting it to Linux.
Our goal is to test as many parts of the system as possible,
including system calls (particularly for POSIX 1003.1 compliance),
libraries in the official distribution, networking protocols,
and utilities. We need help with
1) integrating good tests donated by the community (and probably
written in a variety of programming languages and command
shells) into the DejaGnu framework, and
2) writing new tests for major functional areas for which no
adequate tests exist.
The tests we use will be placed under the GPL and distributed by
Cygnus Support along with DejaGnu. H. J. Lu, who maintains the C
library for Linux, has offered to run the POSIX 1003.1 tests and as
many others as he can.
Process
First let me summarize the advantages of using DejaGnu, a free
software product that will be the umbrella for all the tests, and
then explain how we plan to conduct this project.
Cygnus Support released DejaGnu to the public on January 3, along
with several test suites for GNU language tools. The product is
designed for portability and easy cross-platform development and
execution. It works more uniformly than typical tests using the
UNIX shell, and also supports interactive testing better than most
test platforms -- for instance, Cygnus has written over 1300 unit
tests for the gdb debugger using it.
The implementation of DejaGnu is based on tcl and expect, two simple
existing languages for writing commands. You can develop new tests
in two major ways: by placing keyword-based comments in C source
code, or by writing tests in tcl.
While Cygnus is still increasing their body of tests for language
tools, they are also turning their resources toward the libraries
and GNU utilities. While most of the tests are written at Cygnus,
they have reached out to communities of testers and are now porting
substantial sets of donated tests.
We are hoping to broaden this successful use of collaboration across
user communities. We hope that suites of useful tests are sitting
in desk drawers out there. We also hope to gather and mobilize
people who appreciate the value of formal tests in legitimizing free
software, and would like to help write them.
I am not an employee of Cygnus, but have volunteered to talk to
interested people and do an initial classification of tests, just to
offload some of this routine work from them. I will discuss all
offers with Cygnus staff to find out what is most needed and decide
which tests to incorporate into DejaGnu. There are several criteria
for choosing tests, including the degree to which an area is
covered, and its importance in the overall stability of the
operating system. Some tests may have to be rejected just because
they are hard to fit into the DejaGnu model.
People
Now you can find out why each of us got involved with this project.
Cygnus writes and distributes free software, selling support as a
means to make money. The company is not directly in the business of
supporting operating systems or common UNIX utilities, so this kind
of testing is tangential to their main goals. But they may want to
support those things in the future. In any case, they would like to
see Linux do well, and this big test project will be a good
promotion for DejaGnu.
H. K. Lu, as a volunteer for Linux, has been using his own system to
implement, port, maintain, and ensure ANSI and POSIX compliance for
the C library (mostly the GNU C library along with the iostream
class from the g++ library). He is looking for ways to validate the
work that he and many other contributors have put in.
As an editor at a publisher of computer books, my relationship to
Linux and BSD is even more distant than that of Cygnus. But we are
thinking about putting out books about Linux, from either the Linux
Documentation Project or independent authors, and would like to make
sure Linux is stable enough to be documented. The testing of
utilities is particularly important to me, because it can provide
quality assurance for our books, including the BSD version of UNIX
in a Nutshell.
What you can do
We ask people who have written tests in the areas I have described
to donate them to this project and see them benefit the public.
Don't just send me stuff -- write or call to discuss what you've got
in general. You also have to assign the tests to the Free Software
Foundation (a simple matter, so long as you are the owner of the
tests) so that they can be distributed under the GPL.
Also, please don't starting flaming about the GPL. Either join our
project and donate your tests, or don't. Personally, I do not take
a side in this war.
People who are interested in testing as a discipline, and have some
time to donate, are invited to contact me to help write tests. Tell
me:
Your knowledge of computer languages and tools
Previous software test efforts you have engaged in, if any
How much time you can devote
What functional areas you think are important and have a
particular interest in
Cygnus staff will tell volunteers which areas need testing, and
offer guidance while you learn and apply DejaGnu. The most
important trait for the project is rigorous thinking along the lines
of software quality (so the best preparation is to have done some
formal testing before). You do not need to know any particular
language or tool; you can learn tcl or other aspects of DejaGnu
fairly quickly.
Contact information for me is in my signature. I am at the phone
number during traditional U.S. business hours. So send mail any
time, or try calling from Monday through Friday, 9:00 AM to 5:00 PM,
Eastern Daylight Saving Time (four hours earlier than Greenwich Mean
Time).
Feel free to distribute and repost this message elsewhere, in its
entirety.
----------------------------------------------------------------------
Andy Oram O'Reilly & Associates, Inc. andyo@ora.com
90 Sherman Street, Cambridge, MA 02140 (617) 354-5800
fax (617) 661-1116
----------------------------------------------------------------------