Managing System Logs

Managing System Logs

Often times it is useful to log messages from scripts. This can done using the logger command (usually found in /usr/bin). Without any options it takes the user name as the facility and “notice” as the priority. However, you can specify both a facility and priority from the command line by using -p option for example:

logger -p kern.warning The kernel has been recompiled.

This would send the specified message to the same place other kernel messages are sent. For details on the other options, see the logger(1) man-page.

One common problem is what to do with all of the log messages. If you do a lot of logging (particularly if everything is sent to a central server), you can fill up your filesystem faster than you think. The most obvious and direct solution is to remove them as after a specific length of time or when they reach a particular size.

It is a fairly simple matter to write a shell script that is started from cron that looks at the log files and takes specific actions. The nice thing is that you do not have to. Linux provides this functionality for you in the form of the logrotate command.

As its name implies, the goal of the logrotate program is to “rotate” log files. This could be as simple as moving a log file to a different name and replacing the original with an empty file. However, there is much more to it.

Two files define how logrotate behaves. The state file (specified with the -s or –state option) basically tells logrotate when the last actions were taken. The default is /var/state/logrotate.

The configuration file tells logrotate when to rotate each of the respective files. If necessary, you can have multiple configuration files which can all be specified on the same command line or you include configuration files within another one.

The logrotate configuration file is broken into two parts. At the beginning are the global configuration options, which apply to all log files. Next, there are the configuration sections of each of the individual files (the logfile definitions). Note that some options can be global or for a specific log file, which then overwrites the global options. However, there are some that can only be used within a logfile definition.

A very simple logrotate configuration file to rotate the /var/log/messages might look like this:

errors root@logserver compress /var/log/messages { rotate 4 weekly postrotate /sbin/killall -HUP syslogd endscript }

At the top are two global options, followed by a logfile definition for /var/log/messages. In this case, we could have included the global definitions within the log file definition. However, there is normally more than one logfile definition.

The first line says that all error messages are sent (mailed) to root at the logserver. The second line says that log files are to be compressed after they are rotated.

The logfile definition consists of the logfile name and the directives to apply, which are enclosed within curly brackets. The first line in the logfile definition says to rotate the 4 times before being removed. The next line says to rotate the files once a week. Together these two lines mean that any given copy of the /var/log/messages file will be saved for 4 weeks before it is removed.

The next three lines are actually a set. The postrotate directive says that what follows should be done immediately after the log file has been rotated. In this case, syslogd is sent a HUP signal to restart itself. There is also a prerotate directive, which has the same basic functionality, but does everything before the log is rotated.

It is also possible to specify an entire directory. For example, you could rotate all of the samba logs by specifying the directory /var/log/samba.d/*.

As I mentioned, you can also rotate logs based on their size. This is done by using the size= option. Setting size=100K would rotate logs larger than 100 KiB and 100MiB would rotate logs larger than 100 MiB.

Although you can ease the management of your log files with just the options we discussed, there are an incredible number of additional options which you can use. Table 3 contains a list of options you can use with a brief explanation. For more details see the logrotate(1) man-page.

Table 3 – logrotate options

compress/nocompress – compresses or does not compress old versions of logs.

delaycompress – Wait until the next cycle to compress the previous log.

create mode owner group – Log file is recreated with this mode, owner and group. (nocreate overrides this.)

daily, weekly, monthly – Rotate logs in the indicated interval.

errors address – Send errors to the address indicated.

ifempty – Rotate the logs even if they are empty. (notifempty overrides this.)

include file_or_directory – Include the indicate file at this point. If a directory is given, all real files in that directory are read.

mail address – Logs rotate out of existence are mailed to this address. (nomail overrides this)

olddir directory – old logs are moved to this directory, which must be on the same physical device. (noolddir overrides this.)

postrotate/endscript – delimits commands run after the log is rotated. Both must appear on a line by themselves.

prerotate/endscript – delimits commands run before the log is rotated. Both must appear on a line by themselves.

rotate count – Rotates the log ‘count’ times before being removed.

size size – Log files greater than ‘size’ are removed.

tabooext [+] list – list of files not to include. A plus-sign means the files are added to the list rather than replacing it.