There are many bad things users can do to a system if they have an interactive shell account. There are also many ways to prevent these things from happening. User quotas on disk usage, CPU usage, etc. are a good start, more advanced techniques such as monitoring users for large environments also help. One of the simplest things a user can do is use up all the memory by launching many copies of a memory hungry program, or use up all the file descriptors with a fork bomb.
Most modern Linuxs ship with PAM support, one of the things PAM provides is environmental settings. Settings such as limiting the amount of memory a user is allowed to use. In Red Hat and Caldera this is configurable from the /etc/security/ directory which contains a number of files. The most interesting file is: /etc/security/limits.conf, which allows you to define rules for users or groups, whether the rule is soft or hard (more on this later), and what the rule applies to, which can be CPU, memory, maximum filesize, and so on. For example:
* hard core 0 bob soft nproc 100 bob hard nproc 150
This first rule disabled core dumps for everyone, the second rule sets a soft limit on bob to 100 processes, and the third rule sets a hard limit for bob to 150 processes. A soft limit can be exceeded, and is usually a warning mark, the hard limit cannot be exceeded. As you can imagine this is quite useful since it applies to all login shells, and other services such as ftp.
Bash has a built in limiter, accessed via ulimit. Any hard limits cannot be set higher, so if you have limits defined in /etc/profile, or in the users .bash_profile (assuming they cannot edit/delete those files) you can enforce limits on users with Bash shells. This is useful for older Linux distributions that lack PAM support. You must also ensure that the user cannot change their login shell. Settings the limits is similar to PAMs method, you define various such as:
ulimit Sc 0 ulimit Su 100 ulimit Hu 150
These three rules would achieve the same result as the ones in the PAM example. The first rule disables core dumps, the second rule sets a soft limit of 100 processes, and the third rule sets a hard limit of 150 processes. More help on ulimit is available by typing help ulimit at the bash prompt.
Quota is a system for restricting disk usage by users. It is built into most distributions and help is available from the man page man quota.
One issue common to shell servers is making sure users do not abuse the server. This is rather easy to monitor for standard resources (such as disk usage, CPU usage, and so forth) but one of the most frequently abused items is bandwidth, luckily there are a variety of ways to monitor this abuse.
Of course this is all well and good if nothing goes wrong. But what if you actually want to monitor what a user is doing (be warned, there are legal implications that can get you in trouble, ask your lawyers first). This is where a tool such as ttysnoop comes in. ttysnoop allows you to monitor what a user is doing, and record it. You can get ttysnoop from: http://uscan.cjb.net/.
UserIPAcct allows you to monitor the bandwidth usage by user, it involves patching the kernel, and setting up rules (similar in concept to firewalling) to monitor the amount of data a users programs send or receive. You cannot account for data on PPP connections however since the PPP daemon does not run as the user logging in (although you could hack it to do this). I would recommend this highly for shell servers in order to monitor users (generally speaking a minor percentage of users will make up the bulk of usage). You can download the complete package from: http://zaheer.grid9.net/useripacct/.