Return to BSD News archive
Xref: sserve comp.os.linux.misc:18096 comp.os.386bsd.misc:2623
Path: sserve!newshost.anu.edu.au!harbinger.cc.monash.edu.au!bunyip.cc.uq.oz.au!munnari.oz.au!spool.mu.edu!agate!library.ucla.edu!europa.eng.gtefsd.com!MathWorks.Com!news.kei.com!travelers.mail.cornell.edu!newsstand.cit.cornell.edu!news.graphics.cornell.edu!ghost.dsi.unimi.it!univ-lyon1.fr!swidir.switch.ch!scsing.switch.ch!news.dfn.de!news.dfn.de!Germany.EU.net!EU.net!uunet!sparky!dwb
From: dwb@ITD.Sterling.COM (David Boyd)
Newsgroups: comp.os.linux.misc,comp.os.386bsd.misc
Subject: CFD: Reactivation of comp.sources.reviewed and its policies
Followup-To: comp.sources.testers
Date: 30 Jun 1994 03:50:54 GMT
Organization: Sterling Software
Lines: 442
Sender: dave_boyd#sterling.com
Distribution: world
Message-ID: <2utfeu$g1h@sparky.sterling.com>
NNTP-Posting-Host: carto.itd.sterling.com
(Moderators Note: I am posting this discussion to linux and bsd related
newsgroups in hopes of attracting both reviewers and packages from these
communities. Please note the followup to comp.sources.testers)
Subject: Introduction & Proposed Guidelines for CSR
This posting provides background on the current status and possible future
directions for the newsgroup "comp.sources.reviewed" (CSR). It contains a
background on the history of the group, a description of the group as it
currently is chartered, a description on how I see the group evolving,
information about the CSR archives, and some guidelines for getting started
as a submitter/reviewer.
I have issued the RFD in order to foster discussion within the community
as to how to again make CSR a viable and valuable newgroup. As a way of
introduction my name is David Boyd and I volunteered to take over the role
of CSR moderator in February from Kevin. Due to some extended business
related travel I was unable to actually pick up and get things going with
the group until now. As part of getting things going with CSR I have undertaken
a review of both current and past policies for the group and attempted to
come up a new policy which will hopefully bring together the best aspects
of previous policies and revitalize comp.sources.reviewed.
Past Policies
-------------
Each of the past two moderators has had a slightly different policy for
conducting the reviews. The key difference between these policies is
that in one case the Call For Reviews (CFR) where sent only to a standing
list of reviewers versus sent as an announcement to the net as a whole.
My hope is to combine these two policies so that people who review packages
regularly will get notified of the availability of packages for review via
E-mail (they won't need to monitor a news group). In addition, a general
CFR will go out to the community via comp.sources.testers so that we pick
up new reviewers and those who may only be interested in specific packages.
The other major past policy which I am considering changing is that of the
reviewers deciding whether a package should be posted following the review.
Although reviewer input on the suitability of a package for release is
important, the moderator was in the position of making the final determination
what may be considered a major problem by one reviewer may not even be
noticeable or important to another reviewer (or the vast number of potential
users of the package). In addition, the author of the package who owns
the code and whose reputation will be directly affected by the release of
the software is left completely out of the loop. It is my opinion that
any author who is professional enough to make their software available
for public review and criticism is:
a. Genuinely interested in making quality software available
to the community.
b. Would not want to post software with major flaws
Given the above I believe that the decision to post a package should
generally be left to the author. The key exception to this would that
the moderator could reject a package if it fails to meet the basic
submission criteria (i.e. unpacks cleanly, makefile, man pages/docs, etc)
or if the package failed to fundamentally perform as advertised or had
other fatal problems (would not build, created security holes, etc).
Basically, once a package has been reviewed the author would have the
following choices:
a. Post the package as-is with the list of know problems
identified by the reviewers.
b. Opt to re-submit the package at a later date.
c. Submit patches withing 5 days or so of the completion of
the review to correct identified problem and request an
abbreviated review (retest) of the fixed package by the
existing review team.
d. Withdraw the package from the review process (and possibly
submit to another group for posting.
This approach offers several benefits. First, it reduces the time from
submission to posting in a number of instances (which has been a problem
in the past). Second, it allows the author to select an approach which
best fits within the constraints of his schedule (most of us do have jobs
other than writing free code or other demands on our time).
As pointed out to me by Kent Landfield this is a very radical change from
previous policies. An alternative to this that Kent and I discussed
was what I call the security council approach. All reviewers and the
moderator and the author would each get a vote on posting the package.
However, the moderator and the author would have veto power and be able
to stop a package from being posted (in reality the author always has and
will have this power). The votes could be a simple yes/no or more
complex such a voting for one of the alternatives (a-d) above.
That basically covers the changes I am considering. I look forward to
hearing any questions, comments, or criticisms the community has about
my proposals, along with any suggestions on how to improve CSR. The
following sections will detail the charter for CSR, procedures
for submitting packages for review, how the review will be moderated,
how to become a reviewer and review a package, and how the archives will
be managed. These procedures currently reflect my favored thinking but
are net set in anything firmer than warm jello. Please provide any
feedback or suggestions you may have on this or any other aspect of
CSR. (Evan flames are welcome, I just bought some new flame retardent
underware).
What is Comp.Sources.Reviewed?
------------------------------
"Comp.sources.reviewed" is a moderated newsgroup (moderator: David
Boyd [csr-request@sparky.sterling.com]) with the following guidelines:
Comp.sources.reviewed is a moderated newsgroup for the distribution of
program sources that have been subjected to a Peer Review process.
Similar to the process used for academic journals, submissions are sent
to a moderator who then sends the sources to Peer Review volunteers for
evaluation. The Reviewers are asked to provided a timely evaluation of
the software by compiling and running it on their machine. If time does
not permit them to complete a review, they are responsible for asking
the moderator to select another reviewer. The completed reviews will
be collected by the moderator and provided to the author of the software
along with a recommendation as to whether the software should be posted
in its current form. The author can then decide whether to post the
software, make the recommended corrections and re-submit the software
for another round of reviews, or withdraw the software from the review
process. In cases of dysfunctional software the moderator reserves
the right to override the Author's decision and reject the package for
posting. Upon agreement between the Author and moderator to post
the software, the sources will be posted along with the written comments
provided by the Reviewers. In all cases the Reviewers' comments will
be posted as well.
How to submit software
----------------------
- Software is mailed to the csr submission address (csr-request@sterling.com)
in small (> 80K) shar files.
All software packages must include:
a. An introduction to the package for the moderator containing:
1. Information on any hardware or software dependencies.
2. Any suggestions for or limitations on the review period.
3. A brief description of the package suitable for inclusion in
the CFR.
b. A makefile, or makefile equivalent.
c. Directions for building and installing the software
d. Manual Page(s) or other documentation detailing how to use
the software.
- Upon acceptance of the package for review, your name will be added to
a restricted mailing list for the author and reviewers (the moderator
will also monitor the list). Respond as necessary to any questions or
comments from the reviewer. Providing patches during the review cycle
is discouraged but allowed to fix specific problems encountered by
reviewers. (NOTE - The moderator will not be tracking patches provided via
this mailing list. The Author will be required to provide a complete set
of patch updates to the moderator following the review period and the
package will require a regression review prior to being posted).
- Any author who becomes abusive or derogatory towards the reviewers, or
the moderator will have their review terminated. Future reviews of
software by that author will be at the discretion of the moderator.
- At the completion of the review period the moderator will provide the
author with a summary of the review comments and a recommendation for the
posting of the package along with the complete text of the reviews.
- The author will review the comments and with the help of the moderator,
make a decision on how to proceed along one of the following paths:
a. Withdraw the package from the review process.
b. Request that the package be posted as-is.
c. Provide the moderator with a set of patches to the original
baseline and request an abbreviated regression review by the
current set of reviewers (No CFR will be issued). (Note -
The moderator reserves the right to determine if the patches
are too extensive and the package requires a complete review
cycle.
d. Opt not to post the package at this time and to resubmit for
a complete review at a later date.
What the moderator will do
--------------------------
- On receipt of the package the moderator will check the package to
determine if it meets the submission criteria above, unpacks cleanly, and
if feasible at least builds on the moderator's work machine. If the package
does not meet the submission criteria the author will be notified and asked
to re-submit the package.
- The moderator will generate a CFR which includes the following:
a. A brief description of the package.
b. A list of known limitations and hardware/software requirements
for the package.
c. The dates for the review period.
d. Directions on how to obtain the package for review (Both mail-server
and anonymous ftp support for retrievals are planned).
e. The address of the mailing list for discussions during the review
period.
- The moderator will mail the CFR to the standing reviewers list (see
below for how to be added to that list) and will post the CFR to
comp.sources.testers along with any other appropriate groups (e.g.
comp.windows.x for X based packages, comp.os.linux.announce for linux
packages, etc).
- The moderator will monitor who retrieved the package for review and the
discussion list during the review period. If during the initial part
review period the package appears to be dysfunctional the moderator (after
consultation with reviewers) may terminate the review cycle and provide
the author with a description of the problems encountered by the reviewers.
- If no one volunteers to review the package within a reasonable time
following the CFR the moderator will either:
a. Reissue the CFR with new review dates.
b. Suggest to the author that the package be posted to another
appropriate source group.
- Prior to the expiration of the review period the moderator will remind
any reviewers who have not submitted reviews that they are due. This will
be done via either private mail or the package discussion list.
- At the expiration of the review period the moderator will summarize the
reviews and provide the summary, complete reviews, and a recommendation for
the package to the author.
- If the author opts to provide a set patches to correct identified
problems the moderator will:
a. First verify that the patches apply to the baseline source cleanly.
b. The patch set will then be provided to the existing reviewers via
the package discussion list along with the expiration date of the
regression review period. Only one set of patches will be accepted
for a package during the review cycle. If the patches render the
software dysfunctional the review will be terminated and the author
author asked to select another option from above.
c. The moderator will collect the regression review comments and provide
them to the author who will select one of the other options for
the package.
- If the author opts to post the package, the moderator will post the
baseline software along with the reviews to the c.s.r group and will
update the archives at ftp.uu.net and ftp.sterling.com.
- The moderator will post the review summary to all groups to which the
CFR was posted along with a "thank you" to the reviewers, and the location
of the software in the archives.
What do the reviewers do
------------------------
- Reviewers receive a copy of the CFR either via their favorite new group
or via the standing reviewers mailing list (which all reviewers will be
given opportunity to join. No one will be automatically placed on the
list).
- Reviewers request the package from the mail-server announced in the
CFR or retrieve it via ftp from the designated directory on ftp.sterling.com.
If the reviewer retrieves the package via ftp they will need to notify the
moderator at comp-sources-reviewed@sterling.com that they will be reviewing
the package.
- Reviewers should also retrieve a copy of the Guide for Reviewers either
from the mail server or from ftp.sterling.com
- Reviewers should unpack the distribution, and build the software on
their platform(s). Any problems with the build should be coordinated
with Author of the package via the discussion mailing list. This is
expected to be the most step where the reviewer and author will agree
to patch the software in order for it to build on a new platform.
- The reviewer should note ALL problems either building or executing the
software during the review period.
- Specifically the reviewer should note:
a. Whether the documentation (both for building and using the
software was clearly written and adequate.
b. Whether the makefile (or makefile equivalent) was well
structured and easily modified if necessary for the build
(i.e. select compiler, etc).
c. Does the software perform as defined in the documentation.
- The reviewer is also encouraged to comment on more subjective areas:
a. Reviewers are encouraged to review the source code and comment
on the structure and design. (Note: Comments on coding style
and indenting are discouraged. If you don't like the indenting,
run the code through cb).
b. Does the package meet one or more functional requirements
for your organization.
c. Are there areas in which the package could be improved (MMI,
additional functionality, etc).
d. Are there extraneous functions performed by the package that
really don't belong.
e. If you ported the package to a new platform (i.e. One not
originally noted by the author) how easy was it, and were there
areas which could be made more portable.
- Reviewers are free to run the software through any automated test
tools they feel are appropriate (CodeCenter, ObjectCenter, Purify,
Insight, Sentinal, etc).
- Reviewers are to submit their reviews to the CSR moderator by the
review deadline. At a minimum the review must state the name of the
package, the platform (hardware, os) the review was run on, and whether
the package successfully built and executed.
- Reviewers will be reminded prior to the expiration of the review period
that the reviews are due. If the reviewer is not able to submit a review
by the deadline they should notify the moderator. (Note - Work requirements
and deaths in the family are acceptable excuses ;-).
- Any questions about the software should be routed through the discussion
list provided. Any reviewer who is abusive or derogatory towards either
the author, the moderator, or other reviewers will be removed from
the discussion list and will only be permitted future reviews at the
discretion of the moderator. We are all professionals and I expect
us to behave that way.
- If the author provides a set of patches the reviewer will be asked
to apply the patches and regression test the package
- Bask in the knowledge that you helped to contribute to the quality of
free software on the internet.
Where are the Archives?
-----------------------
The official archive sites for comp.sources.reviewed are ftp.uu.net
(uunet.uu.net) and ftp.sterling.com, where the sources are available
by anonymous FTP (and other means) in /usenet/comp.sources.reviewed.
Like most sources groups, the archives for CSR are organized in terms
of "volumes". In the future CSR may go to package based archives such
as are provided for comp.sources.misc and comp.sources.x in addition
to the volume/issue based archives.
We have also been told that:
The USENET archive section of the Anonymous FTP area of kirk [in
Australia] now holds an archive of comp.sources.reviewed. This
archive is updated daily (i.e. any posting will can be found in the
archive and associated indexes within 24 hours of the news article
being received by kirk).
The archive is available via anonymous FTP only on kirk.bu.oz.au:
(131.244.1.1). Fetchfile access will be available in the near
future.
For more information, please contact ftp@kirk.bu.oz.au.
An Index of sources already posted to the group is posted at the beginning
of each volume of CSR. It can be obtained via anonymous FTP from
ftp.sterling.com:/usenet/comp.sources.reviewed/index. If you do not have
FTP capabilities you can use the following command to have it sent to
you via email.
echo "send index from comp.sources.reviewed" | /bin/mail netlib@uunet.uu.net
How do I use that silly mail-server?
------------------------------------
(Moderators Note - This is solely in draft form. I am still working out
the kinks and the mail server is not currently supporting CSR)
Send a mail message to it like:
To: majordomo@sterling.com
Subject: help
Thanks!
<your signature here>
And it will send you a help file. Here is a summary of the Subject lines
the mail-server might do something useful with:
Info commands:
Subject: help
Subject: index
Subject: send index
Subject: send guide
To get source:
Subject: send product
To ACK or NAK and product:
Subject: got product
Subject: nix product
To (back out of a) review:
Subject: review product
Subject: drop product
To submit a product:
Subject: submit product-version part/total
What can I review right now?
----------------------------
There are currently no reviews in progress. If you submitted a
package for review but it has become lost in the transition please
notify the moderator.
What else should I do?
----------------------
Subscribe to comp.sources.testers -- we need you.
Thanks for you time and efforts!
--
David Boyd, comp-sources-reviewed-request@sterling.com
--
--
David W. Boyd UUCP: uunet!sparky!dwb
Sterling Software ITD INTERNET: Dave_Boyd@Sterling.COM
1404 Ft. Crook Rd. South Phone: (402) 291-8300
Bellevue, NE. 68005-2969 FAX: (402) 291-4362
I survived - Seoul Sea of Fire Tour 94