We discussed some of PCI’s local logging requirements for maintaining a reliable system audit trail in PCI DSS Requirement 10: Part 1. Section 10.5 goes on to make it clear that more is required to secure this audit trail:

10.5 Secure audit trails so they cannot be altered.

10.5.1 Limit viewing of audit trails to those with a job-related need.

10.5.2 Protect audit trail files from unauthorized modifications.

10.5.3 Promptly back up audit trail files to a centralized log server or media that is difficult to alter.

?10.5.4 Write logs for external-facing technologies onto a log server on the internal LAN.

What this section tells us is we need to have our audit trail (which consists of all our log entries) sent to a centralised log server with restricted access. This can all be achieved using standard logging software, rsyslog.

(The same result could be achieved using syslog-ng also, and probably to an extent the stock syslog. However, rsyslog is now the standard for Debian so this is the one we chose.)

Rsyslog is an enhanced syslog and is available for most Linux distributions. As each machine in our environment logs entries to its local /var/log/syslog or /var/log/rootsh.log etc., we want each machines to also send that entry to a centralised log server. This is exactly what rsyslog was designed to do so it doesn’t pose much of a problem to implement.

rsyslog Server Configuration

Rsyslog can receive syslog messages via UDP or TCP. However, using these protocols there is no real guarantee that messages get to their destination. If our aim is to maintain a reliable and accurate audit trail we would prefer to have a reliable transport methodology (although the real chance of TCP or UDP messages going AWOL is really quite small). rsyslog provides this in the RELP protocol.

So to configure our log server (lets call it LOG_SRV1) so it receives RELP messages we simply install the Debian package and activate it in the /etc/rsyslog.conf:

apt-get install rsyslog-relp

And in the /etc/rsyslog.conf we enable the receiver for RELP messages on port 20514:

# provides RELP syslog reception
$ModLoad imrelp
$InputRELPServerRun 20514

Although it is no problem generating RELP syslog messages from Debian machines, some other systems may only be able to generate TCP or UDP messages. So for that reason you may want to have the log server receive these also:

# provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 10514

# provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

Client Configuration

On the clients, to send RELP messages you need to install the rsyslog-relp package there also. Then to activate in the /etc/rsyslog.conf add:

$ModLoad omrelp

Then, at the top of the RULES section, to send everything to LOG_SRV1 before logging it to disk locally we simply add the following:

###############
#### RULES ####
###############
#
# Log everything to server

*.*                             :omrelp:LOG_SRV1:20514

And at the simplest level, that’s really all there is to it. However, as always, there are some complications.

Central Logging For Non-syslog Applications

A common issue with implementing centralised logging with rsyslog is that some applications do not support logging to syslog. One such application which we were using is Nginx. As standard Nginx just logs directly to disk and requires a patch to log using syslog.

Fortunately, if you want to avoid recompiling your application (or simply haven’t got access to the source code for the application in question), rsyslog has provided us with a way to pull a log file from disk and pass it to rsyslog’s rule engine.

Using Nginx as an example, we have it generating /var/log/nginx/error.log which we would like to have sent to the log server. To do this we must add a few lines to the /etc/rsyslog.conf as follows:

$ModLoad imfile

$InputFileName /var/log/nginx/error.log
$InputFileTag nginx_error_log:
$InputFileStateFile nginx_error_log_state
$InputFileFacility local4
$InputFileSeverity notice
$InputRunFileMonitor

To go through this line by line:

Load the imfile module:

$ModLoad imfile

Specify the file to send to the rsyslog rule engine:

$InputFileName /var/log/nginx/error.log

The tag to use in the logs for messages which are generated from this file (we refer to this again later the OSSEC configuration if we are to add any OSSEC exceptions regarding this log file):

$InputFileTag nginx_error_log:

rsyslog keeps a record of its position in the file it is processing processing. Specify the file it uses to store that position:

$InputFileStateFile nginx_error_log_state

Specify the facility and severity to use for logging. This is important when we come to organising the log files on the server:

$InputFileFacility local4
$InputFileSeverity notice

Activate the file monitor:

$InputRunFileMonitor

Note that the use of imfile is a compromise and it does have a few limitations, namely the current log file must always have the same name for imfile to work. Also, this method does not provide real-time logging to syslog. The file is regularly polled a default of every 10 seconds. This period can be altered using the $InputFilePollInterval directive.

Log Server File Organisation

Depending on the size of your environment, you may now have 10-20 servers all logging to the same log server. By default, log messages will be written out to the same file on the server as they would have been on the client. For example, anything which is logged to /var/log/syslog, will get logged to /var/log/syslog on the server and you will end up with one very very busy file.

To have the rsyslog server log each host’s syslog in a separate directory we can define a template in /etc/rsyslog.conf as follows:

$template syslog_perhost,"/var/log/hosts/%HOSTNAME%/syslog.log"

Then the selector rule for syslog on the server would, instead of pointing at /var/log/syslog, point at our template:

*.*;auth,authpriv.none          -?syslog_perhost

%HOSTNAME% will be substituted with the hostname of the sender. This will have the effect of sending any log messages from a server called, say HOST1, to the file /var/log/hosts/HOST1/syslog.log. (Note, there is a reason we have added the .log extension to syslog. In our post about OSSEC we discuss log file integrity checking and it makes life a lot easier then if all logs have the same file extension.) So if such a template is set up for all possible log files, each host will have their own log directory (and this includes the local log server logs which will end up in /var/log/hosts/LOGSRV1/).

Also, remember the imfile log for nginx we set up previously, this is where the facility and priority used in the imfile configuration (local4.notice) comes into use. We would set up a nginx error log template as follows:

$template nginx_perhost,"/var/log/hosts/%HOSTNAME%/nginx.log"

Then send all local4.notice messages to ?nginx_perhost and discard:

local4.=notice                  ?nginx_perhost
& ~

You will need to be mindful of clashes where other applications may be using the same logging facility. So choose carefully depending on what software you are running in your environment.

Firewall Configuration

Your firewall should allow communication to your log server (which should be in your most secure network segment) on port 20514 (and ports 10514 and 514 if you use TCP & UDP also).

Data Integrity

10.5.5 Use file-integrity monitoring or change-detection software on logs to ensure that existing log data cannot be changed without generating alerts (although new data being added should not cause an alert).

10.6 Review logs for all system components at least daily. Log reviews must include those servers that perform security functions like intrusion-detection system (IDS) and authentication, authorization, and accounting protocol (AAA) servers (for example, RADIUS).

Note: Log harvesting, parsing, and alerting tools may be used to meet compliance with Requirement 10.6.

For sections, 10.5.5 and 10.6, we can configure OSSEC to monitor all the central log files. This will be covered in a later post about OSSEC.

Log Retention

10.7 Retain audit trail history for at least one year, with a minimum of three months immediately available for analysis (for example, online, archived, or restorable from back-up).

We want log file archives to be stored with the minimum permissions (remember 10.5.1 & 10.5.2), and for a period of 365 days (you will need to do some growth projections to work out how much disk space this will require).

This requires some configuration of logrotate on the log server. In case you haven’t come across is before, logrotate is another standard tool which deals with rotating and compressing the system log files. It is the reason your old syslog files end up called syslog.1, syslog.2.gz, syslog.3.gz etc.

For rsyslog, the logrotate settings are held in /etc/logrotate.d/rsyslog. We can keep the default settings for local log files just for safety (although they should never be used, if something slips through the net and logs to /var/log you want it to be compressed), and add the following at the start to handle /var/log/hosts/

/var/log/hosts/*/*.log
{
        rotate 365
        create 600 root adm
        daily
        dateext
        missingok
        ifempty
        compress
        sharedscripts
        postrotate
                invoke-rc.d rsyslog reload > /dev/null
        endscript
}

These directives mean:

  • rotate 365: allow 365 rotations before deleting any files (must be used with daily directive to achieve 1 year retention).
  • create 600 root adm: create log rotations with 600 root permissions.
  • daily: perform rotations daily.
  • dateext: use date as the file extension on rotation.
  • missingok: don’t raise an error if there is no log file.
  • ifempty: rotate even if empty.
  • compress: gzip upon rotation.
  • sharedscripts: only run postrotate script once for all matches (stops rsyslog being reloaded for every match of /var/log/hosts//.log).
  • postrotate: script to run after rotation.