Linuxdoc Linux Questions
Click here to ask our community of linux experts!
Custom Search

4. Setting up CNews + NNTPd

4.1. Getting the sources and stuff

4.1.1. The sources

C-News software can be obtained from ftp://ftp.uu.net/networking/news/transport/cnews/cnews.tar.Z and will need to be uncompressed using the BSD uncompress utility or a compatible program. The tarball is about 650 KBytes in size. It has its own highly intelligent configuration and installation processes, which are very well documented. The version that is available is Cleanup Release revision G, on which our own version is based.

NNTPd (the NNTP Reference Implementation) is available from ftp://ftp.uu.net/networking/news/nntp/nntp.1.5.12.1.tar.Z. It has no automatic scripts and processes to configure itself. After fetching the sources, you will have to follow a set of directions given in the documentation and configure some C header files. These configuration settings must be done keeping in mind what you have specified when you build the C-News sources, because NNTPd and C-News must work together. Therefore, some key file formats, directory paths, etc., will have to be specified identically in both software systems.

The third software system we use is Nestor. This too is to be found in the same place where the NNTPd software is kept, at ftp://ftp.uu.net/networking/news/nntp/nestor.tar.Z. This software compiles to one binary program, which must be run periodically to process the logs of nntpd, the NNTP server which is part of NNTPd, and report usage statistics to the administrator. We have integrated Nestor into our source base.

The fourth piece of the system, without which no Usenet server administrator dares venture out into the wild world of public Internet newsfeeds, is pgpverify.

We have been working with C-News and NNTPd for many years now, and have fixed a few bugs in both packages. We have also integrated the four software systems listed above, and added a few features here and there to make things work more smoothly. We offer our entire source base to anyone for free download from http://www.starcomsoftware.com/proj/usenet/src/news.tar.gz. There are no licensing restrictions on our sources; they are as freely redistributable as the original components we started with.

When you download our software distribution, you will extract it to find a directory tree with the following subdirectories and files:

  • c-news: the source tree of the CR.G software release, with our additions like pgpverify integration, our scripts like mail2news, and pre-created configuration files.

  • nntp-1.5.12.1: the source tree of the original NNTPd release, with header files pre-configured to fit in with our configuration of C-News, and our addition of bits and pieces like Nestor, the log analysis program.

  • howto: this document, and its SGML sources and Makefile.

  • build.sh: a shellscript you can run to compile the entire combined source tree and install binaries in the right places, if you are lucky and all goes well.

Needless to say, we believe that our source tree is a better place to start with than the original components, specially if you are installing a Usenet server on a Linux box and for the first time. We will be available on email to provide technical assistance should you run into trouble.

4.2. Compiling and installing

For installing, first make sure you have an entry for a user called news in your /etc/password file. This is setting the news-database owner to news. Now download the source from us and untar it in the home directory of news. This creates two main directories viz. c-news and nntp. To install and compile, run the script build.sh as root in the directory that contains the script. It is important that the script run as root as it sets ownerships, installs and compiles the source as user news. This is a one-step process that puts in place both the C-News and the NNTP software, setting correct permissions and paths. Following is a brief description of what build.sh does:

  • Checks for the OS platform and exits if it is not Linux.

  • Again, exits if you are not running as root.

  • Looks for and exits if cannot find the above two directories.

  • Compiles C-News and performs regression tests if the compilation was successfull. Sends out a warning to read the error file make.out.r and to fix 'em. Compilation erros are written to a file called make.out.

  • Performs the above operation in the nntp directory, too.

  • Checks for the presence of the three key directories: $NEWSARTS - (/var/spool/news) that houses the artciles, $NEWSCTL -(/var/lib/news) that contain configuration, log and status files and $NEWSBIN - (/usr/lib/newsbin) that contain binaries and executables for the working of the Usenet News system. Tries to create them if non-existent and exits if it results in failure.

  • Changes the ownership of these directories to news.news. This is important since the entire Usenet News System runs as user news. It will not function properly as any other user.

  • Then starts the installation process of C News. It runs make install to install binaries at the right locations; make setup to set the correct paths and umask, create directories for newsgroups, determine who will receive reports; make ui to set up inews and injnews and make readpostcheck to use readnews, postnews and checknews scripts provided by C News. The errors, if any are to be found in the respective make.out files. e.g. make.setup will write errors to make.out.setup

  • Newsspool, which queues incoming batches in $NEWSARTS/in.coming directory should run as set-userid and set-groupid. This is done.

  • A softlink is made to /var/lib/news from /usr/lib/news.

  • The NNTP software is installed.

  • Sets up the manpages for C News and makes it world readable. The NNTP manpages get installed when the software is installed. Compiles the C News documentation guide.ps and makes it readable and available in /usr/doc/packages/news or /usr/doc/news.

  • Checks for the PGP binary and asks the administrator to get it, if not found.

4.3. Configuring the system: What and how to configure files?

Once installed, you have to now configure the system to accept feeds and batch them for your neighbours. You will have to do the following:

  • nntpd: Copy the compiled nntpd into a directory where executables are kept and activate it. It runs on port 119 as a daemon through inetd unless you have compiled it as stand-alone. An entry in the /etc/services file for nntp would look like this:

    nntp   119/tcp    \# Network News Transfer Protocol
    
    An entry in the inetd.conf file will be:
     nntp    stream    tcp    nowait   news    path-to-tcpd  path-to-nntpd 
    
    The last two fields in the inetd.conf file are paths to binaries of the tcp and the nntp daemon respectively.
  • Configuring control files: There are plenty of control files in $NEWSCTL that will need to be configured before you can start using the news system. The files mentioned here are also discussed in the first section of the section titled "Components of a running system>". These control files are dealt in detail in the following below.

    • sys: One line per system/NDN listing all the newsgroup hierarchies each system subscribes to. Each line is prefixed with the system name and the one beginning with

      ME:
      
      indicates what your server is willing to receive. Following are typical entries that go into this file:
      ME:comp,news,misc,netscape
      
      This line indicates what newsgroups your server has subscribed to.
      server/server.starcomsoftware.com:all,!general/all:f
      
      This is a list of newsgroups your server will pass on to your NDN. The newsgroups specified should be a comma separated list and the entire line should contain no spaces. The f flag indicates that the newsgroup name and the article number alongwith its size will make up one entry in the togo file in the $NEWSARTS/out.going directory.
    • explist: This file has entries indicating which articles expire and when and whether they have to be archived. The order in which the newsgroups are listed is important. An example follows:

      comp.lang.java.3d    x    60    /var/spool/news/Archive
      
      This means that the articles of comp.lang.java expire after 60 days and shall be archived in the directory mentioned in the fourth field. Archiving is an option. The second field indicates that this line applies to both moderated and unmoderted newsgroups. m would specify moderated and u would specify unmoderated groups. If you want to specify an extremely large no. as the expiry period you can use the keyword "never".
    • batchparms: sendbatches is a program that administers batched transmission of news articles to other sites. To do this it consults the batchparms file. Each line in the file specifies the behaviour for each of your NDN mentioned in the sys file. There are five fields for each site to be specified.

       server   u     100000    100    batcher | gzip -9 | viauux -d gunzip 
      

      The first field is the site name which matches the entry in the sys file and has a corresponding directory in $NEWSARTS/out.going by that name.

      The second field is the class of the site,u for UUCP and n for NNTP feeds. A "!" in this field means that batching for this site has been disabled.

      The third field is the size of batches to be prepared in bytes.

      The fourth field is the maximum length of the output queue for transmission to that site.

      The fifth field is the command line to be used to build, compress and transmit batches to that site. The contents of the togo file are made available on standard input.

    • controlperm: This file controls how the news system responds to control messages. Each line consists of 4-5 fields separated by white space. Control messages has been discussed in "Section 2.4>".

      comp,sci    tale@uunet.uu.net   nrc    pv   news.announce.newsgroups
      

      The first field is a newsgroup pattern to which the line applies.

      The second field is either the keyword "any" or an e-mail address. The latter specifies that the line applies to control messages from only that author.

      The third field is a set of opcode letters indicating what control operations need to be performed on messages emanating from the e-mail address mentioned in the second field. n stands for creating a newgroup, r stands for deleting a newsgroup and c stands for checkgroup.

      The fourth field is a set of flag letters indicating how to respond to a control message that meets all the applicability tests:

                  y  Do it.
                   n  Don't do it.
                   v  Report it and include the entire control
                      message in the report.
                   q  Don't report it.
                   p  Do it iff the control message carries a valid PGP signature. 
              
      
      Exactly one of y, n or p must be present.

      The fifth field, which is optional, will be used if the fourth field contains a p. It must contain the PGP key ID of the public key to be used for signature verification.

    • mailpaths: This file describes how to reach the moderators of various hierarchies of newsgroups by mail. Each line consists of two fields: a news group pattern and an e-mail address. The first line whose group pattern matches the newsgroup is used. As an example:

                comp.lang.java.3d            somebody@mydomain.com
                 all                          %s@moderators.uu.net
            
      
      In the second example, the %s gets replaced with the groupname and all dots appearing in the newsgroup name are substituted with dashes.
    • Miscellaneous files: The other files to be modified are:

      • mailname: Contains the Internet domain name of the news system. Consider getting one if you don't have it.

      • organization: Contains the default value for the Organization: header for postings originating locally.

      • whoami: Contains the name of the news system. This is the site name used in the Path: headers and hence should concur with the names your neighbours use in their sys files.

    • active file: This file specifies one line for each newsgroup (not just the hierarchy) to be found on your news system. You will have to get the most recent copy of the active file from ftp://ftp.isc.org/usenet/CONFIG/active and prune it to delete newsgroups that you have not subscribed to. Run the script addgroup for each newsgroup in this file which will create relevant directories in the $NEWSARTS area. The addgroup script takes two paramters: the newsgroup name being created and a flag. The flag can be any one of the following:

                 y           local postings are allowed
                  n           no local postings, only remote ones
                  m           postings to this group must be approved
                              by the moderator
                  j           articles in this group are only passed and not kept
                  x           posting to this newsgroup is disallowed
                  =foo.bar    articles are locally filed in
                              "foo.bar" group
              
      
      An entry in this file looks like this:
      comp.lang.java.3d      0000003716      01346   m 
      
      The first field is the name of the newsgroup. The second field is the highest article number that has been used in that newsgroup. The third field is the lowest article number in the group. The fourth field is a flag as explained above.
    • newsgroups file: This contains a one-line description of each newsgroup to be found in the active file. You will have to get the most recent file from ftp://ftp.isc.org/usenet/CONFIG/newsgroups and prune it to remove unwanted information. As an example:

      comp.lang.java.3d      3D Graphics APIs for the Java language
      
    • Aliases: These aliases are required for trouble reporting. Once the system is in place and scripts are run, anomalies/problems are reported to addresses in the /etc/aliases file. These entries include email addresses for newsmaster, newscrisis, news, usenet, newsmap. They should ideally point to an email address that will be accessed at regularly. Arrange the emails for newsmap to be discarded to minimize the effect of sendsys bombing by practical jokers.

    • Cron jobs: Certain scripts like newsrun that picks up incoming batches and maintenance scripts, should run through news-database owner's cron which is news. The cron entries ideally will be for the following: A more detailed report can be found in "Section 9.4>"

      1. newsrun: This script processes incoming batches of article. Run this as frequently as you want them to get digested.

      2. sendbatches: This script transmit batches to the NDNs. Set the frequency according to your requirements.

      3. newsdaily: This should be run ideally once a day since it reports errors and anomalies in the news system.

      4. newswatch: This looks for errors/anomalies at a more detailed level and hence should be run atleast once every hour

      5. doexpire: This script expires old articles as determined by the explist file. Run this once a day.

    • newslog: Make an entry in the system's syslog.conf file for logging messages spewed out by nntpd in newslog . It should be located in $NEWSCTL. The entry will look like this:

      news.debug             -/var/lib/news/newslog
      
    • Newsboot: Have this run (as news the news-database owner) when the system boots to clear out debris left around by crashes.

    • Add a Usenet mailer in sendmail: The mail2news program provided as part of the source code is a handy tool to send an e-mail to a newsgroup which gets digested as an article. You will have to add the following ruleset and mailer definition in your sendmail.cf file:

      • Under SParse1, add the following:

                   R$+ . USENET < @ $=w . >      $#usenet     $: $1
                    
        
      • Under mailer definitions, define the mailer Usenet as:

                   MUsenet      P=/usr/lib/newsbin/mail2news/m2nmailer, F=lsDFMmn, 
                        S=10, R=0, M=2000000, T=X-Usenet/X-Usenet/X-Unix, A=m2nmailer $u
                
        

      In order to send a mail to a newsgroup you will now have to suffix the newsgroup name with usenet i.e. your To: header will look like this:

      To: misc.test.usenet@yourdomain.
      
      The mailer definition of usenet will intercept this mail and post it to the respective newsgroup, in this case, misc.test

    This, more or less, completes the configuration part.

4.4. Testing the system

To locally test the system, follow the steps given below:

  • post an article: Create a local newsgroup

        cnewsdo addgroup mysite.test y
        
    
    and using postnews post an article to it.
  • Has it arrived in $NEWSARTS/in.coming?: The article should show up in the directory mentioned. Note the nomenclature of the article.

  • When newsrun runs: When newsrun runs from cron , the article disappears from in.coming directory and appears in $NEWSARTS/mysite/test. Look how the newsgroup, active, log and history (not the errorlog) files and .overview file in $NEWSARTS/mysite/test reflect the digestion of the file into the news system.

  • reading the article: Try to read the article through readnews or any news client. If you are able to, then you have set most everything right.

4.5. pgpverify and controlperms

As mentioned in "Section 2.4>", it becomes necessary to authenticate control messages to protect yourself from being attacked by pranksters. For this, you will have to configure the $NEWSCTL/controlperm file to declare whose control messages you are willing to honour and for what newsgroups alongwith their public key ID. The controlperm manpage shall give you details on the format.

This will work only in association with pgpverify which verifies the Usenet control messages that have been signed using the signcontrol process. The script can be found at ftp://ftp.isc.org/pub/pgpcontrol/pgpverify. pgpverify internally uses the PGP binary which will have to be made available in the default executables directory. If you wish to send control messages for your local news system, you will have to digitally sign them using the above mentioned signcontrol program which is available at ftp://ftp.isc.org/pub/pgpcontrol/signcontrol. You will also have to configure the signcontrol program accordingly.

4.7. Configuring outgoing feeds

If you are a leaf node, you will only have to send feeds back to your news provider for your postings in public newsgroups to propagate to the outside world. To enable this, you need one line in the sys and batchparms files and one directory in $NEWSARTS/out.going. If you are willing to transmit articles to your neighbouring sites, you will have to configure sys and batchparms with more entries. The number of directories in $NEWSARTS/out.going shall increase, too. Refer to first two sections of the chapter titled "Components of a running system>"for a better understanding of outgoing feeds. Again, you will have to determine how you wish to transmit the feed: UUCP or NNTP.

4.7.2. By NNTP

For NNTP feeds, you will have to decide whether your server will be the connection initiator or connection recipient. If you are the connection initiator, you can send outgoing NNTP feeds more easily. If you are the connection recipient, then outgoing feeds will have to be pulled out of your server using the NNTP NEWNEWS command, which will place heavy loads on your server. This is not recommended.

Connecting to your NDN server for pushing out outgoing feeds will require the use of the nntpsend.sh script, which is part of the NNTPd source tree. This script will perform some housekeeping, and internally call the nntpxmit binary to actually send the queued set of articles out. You may have to provide authentication information like usernames and passwords to nntpxmit to allow it to connect to your NDN server, in case that server insists on checking the identity of incoming connections. (You can't be too careful in today's world.) nntpsend.sh will clean up after an nntpxmit connection finishes, and will requeue any unsent articles for the next session. Thus, even if there is a network problem, typically nothing is lost and all pending articles are transmitted next time.

Thus, pushing feeds out via may mean setting up nntpsend.sh properly, and then invoking it periodically from cron. If your Usenet server connects to the Internet only intermittently, then the process which sets up the Internet connection should be extended or modified to fire nntpsend.sh whenever the Internet link is established. For instance, if you are using the Linux pppd, you can add statements to the /etc/ppp/ip-up script to change user to news and run nntpsend.sh