13. Usenet software: a historical perspective

13.1.1. History

Usenet came into being in late 1979, shortly after the release of V7 Unix with UUCP. Two Duke University grad students in North Carolina, Tom Truscott and Jim Ellis, thought of hooking computers together to exchange information with the Unix community. Steve Bellovin, a grad student at the University of North Carolina, put together the first version of the news software using shell scripts and installed it on the first two sites: unc and duke. At the beginning of 1980 the network consisted of those two sites and phs (another machine at Duke), and was described at the January Usenix conference. Steve Bellovin later rewrote the scripts into C programs, but they were never released beyond unc and duke. Shortly thereafter, Steve Daniel did another implementation in C for public distribution. Tom Truscott made further modifications, and this became the ``A'' news release.

In 1981 at U. C. Berkeley, grad student Mark Horton and high school student Matt Glickman rewrote the news software to add functionality and to cope with the ever increasing volume of news -- ``A'' News was intended for only a few articles per group per day. This rewrite was the ``B'' News version. The first public release was version 2.1 in 1982; the 1.* versions were all beta test. As the net grew, the news software was expanded and modified. The last version maintained and released primarily by Mark was 2.10.1.

Rick Adams, at the Center for Seismic Studies, took over coordination of the maintenance and enhancement of the B News software with the 2.10.2 release in 1984. By this time, the increasing volume of news was becoming a concern, and the mechanism for moderated groups was added to the software at 2.10.2. Moderated groups were inspired by ARPA mailing lists and experience with other bulletin board systems. In late 1986, version 2.11 of B News was released, including a number of changes to support a new naming structure for newsgroups, enhanced batching and compression, enhanced ihave/sendme control messages, and other features.

The final release of B News was 2.11, patchlevel 19. B News has been declared ``dead'' by a number of people, including Rick Adams, and is unlikely to be upgraded further; most Usenet sites are using C News or INN (see next paragraphs).

In March 1986 a package was released implementing news transmission, posting, and reading using the Network News Transfer Protocol (NNTP) (as specified in RFC 977). This protocol allows hosts to exchange articles via TCP/IP connections rather than using the traditional UUCP. It also permits users to read and post news (using a modified news user agent) from machines which cannot or choose not to install the Usenet news software. Reading and posting are done using TCP/IP messages to a server host which does run the Usenet software. Sites which have many workstations like the Sun and SGI, and HP products find this a convenient way to allow workstation users to read news without having to store articles on each system. Many of the Usenet hosts that are also on the Internet exchange news articles using NNTP because the load impact of NNTP is much lower than UUCP (and NNTP ensures much faster propagation).

Our comments: This remark about relative loadings of UUCP and NNTP is no longer applicable with faster machines and networks, and with hugely increased traffic volumes. Today's desktop computers, let alone servers, can all handle both NNTP and UUCP loads effortlessly, if traffic volumes can be restricted. This is partly due to performance enhancements to UUCP as embodied in Taylor UUCP, and partly due to vastly faster processors.

NNTP grew out of independent work in 1984-1985 by Brian Kantor at U. C. San Diego and Phil Lapsley at U. C. Berkeley. Primary development was done at U. C. Berkeley by Phil Lapsley with help from Erik Fair, Steven Grady, and Mike Meyer, among others. The NNTP package (now called the reference implementation) was distributed on the 4.3BSD release tape (although that was version 1.2a and out-of-date) and is also available on many major hosts by anonymous FTP. The current version is 1.5.12.2. It includes NOV (News Overview -- see below) support and runs on a wide variety of systems. It is available from ftp.academ.com:/pub/nntp1.5/nntp.1.5.12.2.tar.gz. For those with access to the World-Wide Web on the Internet, the WWW page http://www.academ.com/academ/nntp.html contains a description and news about NNTP. A different variant, called nntp-t5, implements many of the extensions provided by INN (including NOV support). It is available from ftp.uu.net:/networking/news/nntp/nntp-t5.tar.gz.

One widely-used version of news, known as C News, was developed at the University of Toronto by Geoff Collyer and Henry Spencer. This version is a rewrite of the lowest levels of news to increase article processing speed, decrease article expiration processing and improve the reliability of the news system through better locking, etc. The package was released to the net in the autumn of 1987. For more information, see the paper ``News Need Not Be Slow,'' published in The Winter 1987 Usenix Technical Conference proceedings. This paper is also available from ftp://ftp.cs.toronto.edu/doc/programming/c-news.*, and is recommended reading for all news software programmers. The most recent version of C News is the Sept 1994 ``Cleanup Release.'' C News can be obtained by anonymous ftp from its official archive site, ftp.cs.toronto.edu:pub/c-news/c-news.tar.Z.

Our comments: C News is no longer maintained by anyone that we know, other than ourselves. However, after fixing the remaining bugs in the source, we have not found the need for further maintenance. NNTPd from Brian Kantor and Phil Lapsley is in the same state, but we are working on enhancements to the source for access control and other functionality.

Another Usenet system, known as InterNetNews, or INN, was written by Rich Salz (rsalz@uunet.uu.net). INN is designed to run on Unix hosts that have a socket interface. It is optimized for larger hosts where most traffic uses NNTP, but it does provide full UUCP support. INN is very fast, and since it integrates NNTP many people find it easier to administer only one package. The package was publicly released on August 20, 1992. For more information, see the paper ``InterNetNews: Usenet Transport for Internet Sites'' published in the June 1992 Usenix Technical Conference Proceedings. INN can be obtained from many places, including the 4.4BSD tape; its official archive site is ftp.uu.net in the directory /networking/news/nntp/inn. Rich's last official release was 1.4sec in Dec 1993.

Our comments: The original paper by Rich Salz about INN, where he proposed the design of an alternate Usenet server software, is a must-read for readers interested in Usenet server software. So is the paper by C News authors, cited before it. Most of the issues that Rich Salz had with C News, as stated in his paper, were very relevant at that time. Today, with the current version of NNTPd and the incorporation of the message ID daemon and NOV, these issues are no longer relevant, and the choice of C News+NNTPd versus INN is now based more on the level of maintenance of source code, familiarity and personal preferences than on core design factors.

In June 1995, David Barr began a series of unoffical releases of INN based on 1.4sec, integrating various bug-fixes, enhancements and security patches. His last release was 1.4unoff4, found in ftp://ftp.math.psu.edu:/pub/INN. This site is also the home of contributed software for INN and other news administration tools.

INN is now maintained by the Internet Software Consortium (inn@isc.org). The official INN home is now http://www.isc.org/isc/ and the latest version (1.7.2) can be obtained from ftp://ftp.isc.org/isc/inn/.

Our comments: The URL for the INN home page above is probably incorrect. Try http://www.isc.org/products/INN/.

Towards the end of 1992, Geoff Collyer implemented NOV (News Overview): a database that stores the important headers of all news articles as they arrive. This is intended for use by the implementors of news readers to provide fast article presentation by sorting and ``threading'' the article headers. (Before NOV, newsreaders like trn, tin and nn came with their own daemons and databases that used a nontrivial amount of system resources). NOV is fully supported by C News, INN and NNTP-t5. Most modern news readers use NOV to get information for their threading and article menu presentation; use of NOV by a newsreader is fairly easy, since NOV comes with sample client-side threading code.

...

Details on many other mail and news readers for MSDOS, Windows and OS/2 systems can be found in the FAQ posted to comp.os.msdos.mail-news.

  <ftp://rtfm.mit.edu/pub/usenet/comp.os.msdos.mail-news/intro>
  <ftp://rtfm.mit.edu/pub/usenet/comp.os.msdos.mail-news/software>

13.1.2. Newsfeed management software

Gup, the Group Update Program is a Unix mail-server program that lets a remote site change their newsgroups subscription on their news feed without requiring the intervention of the news administrator at the feed site. Gup operates with the INN (and likely the C News) batching mechanisms. The news administrators at the remote sites simply mail commands to gup to make changes to their own site's subscription list. The mail/interface is password protected. Gup checks the requests for valid newsgroup names, patterns that have no effect and so on. Gup's authors are Mark Delany (markd@mira.net.au) and Andrew Herbert (andrew@mira.net.au). Its official FTP location is ftp.mira.net.au:/unix/news/gup-0.4.tar.gz, but since that's not as well connected as UUNET, people are strongly advised to obtain it from a mirror site, e.g. ftp.uu.net:/networking/news/misc/gup-0.4.tar.gz.

dynafeed is a package from Looking Glass Software Limited that maintains a .newsrc for every remote site and generates the batches for them. Remote sites can use UUCP or run a program to change their .newsrc dynamically. It comes with a program that the remote site can run to monitor readership in newsgroups and dynamically update the feed list to match reader interest. The goal of this is to get a feed that sends only exactly the groups currently being read. dynafeed can be obtained from ftp://ftp.clarinet.com/sources/dynafeed.tar.Z.

13.1.3. News processing software

Software also exists to automatically archive Usenet newsgroups. The package rkive, written by Kent Landfield (kent@sterling.com) can be configured to archive news automatically based on different headers -- Archive-Name, Volume-Issue, Chronological, Subject and External-Command to name a few. It can be run in batch mode from the command line or from cron. It can also be installed in the sys or newsfeeds file to process articles as they are received. rkive supports local spool directories as well as NNTP based access. rkive is available via FTP from ftp://ftp.sterling.com/rkive.

Newsclip is a programming language for writing news filtering programs, from Looking Glass Software Limited, marketed by ClariNet Communications Corp. It is C-like, and translates to C, so a C compiler is required. It has data-types to represent the kinds of things found in article headers and bodies. It can maintain databases of users, message-ids, patterns, subjects, etc. These can be used to decide whether to ignore or select an article. Newsclip can either operate as a standalone program or as part of rn. It is free for non-commercial use and is available from ftp://ftp.clarinet.com/sources/nc.tar.Z. Contact clari-info@clarinet.com with a subject line of ``newsclip'' for more info.

13.1.4. Commercial software

DNEWS is a commercial product from NetWin. DNEWS licenses are provided free to educational institutions for non profit use. With DNEWS, the news is stored in a database so as not to overload the raw file system. DNEWS supports 'sucking' where only groups which users read are pulled over from the feeder site. DNEWS is currently known to run on VMS, Windows NT, Solaris, SunOS, Unixware, HP/UX. DNEWS binaries are available by anonymous FTP from ftp://ftp.std.com/ftp/vendors/netwin/dnews or from http://world.std.com/~netwin/ DNEWS sources can be obtained on request, see the file source.txt in the FTP area for more information.

Our comments: The information on DNEWS may be dated. We have been seeing DNEWS on their own Website for quite a few years now. Check www.netwinsite.com. Moreover, there are other commercial Usenet server software systems available, including the one bundled with the Internet Information Server of Microsoft Windows NT and the ones from iPlanet. And for carrier class systems, there are many commercial Usenet routers available.

13.1.5. Special note on ``notes'' and old versions of news

...

``B'' news software is currently considered obsolete. Unix sites joining the Usenet should install C news or INN to ensure proper behavior and good performance. Most old B news software had compiled-in limits on the number of newsgroups and the number of articles per newsgroup; the increasing volume of news means that B news software cannot reliably cope with a moderately-full newsfeed.