Requirement 10: Track and monitor all access to network resources and cardholder data

This PCI DSS requirement is all about maintaining a secure and demonstrably accurate audit trail of users accessing anything which could be used to reach the cardholder data environment. It was an awkward requirement to get right. On Linux there are some well known tools which get us part of the way, but leave some fairly major holes.

Lets look at the subsections with some discussion of the implications:

10.1 Establish a process for linking all access to system components (especially access done with administrative privileges such as root) to each individual user.

It’s at this point many people are thinking, “Easy… sudo!” And to a degree they’d be right since sudo does help fulfil much of this requirement in the cases where the environment is running entirely on Unix or Linux as was the case for us.

The implications of this requirement are the following:

  1. No-one should be able to log in directly as root (or any other shared user id – see what Requirement 8 has to say about shared user IDs).
  2. All users must have their own unprivileged user id.

To achieve 1. we can edit /etc/ssh/sshd_config and ensure we have:

PermitRootLogin no

To allow system admin’s to do their job sudo can be installed so that, to execute a command which requires root privileges, the sys admin runs, for instance:

$ sudo tail /var/log/syslog

This will allow access, but create a log entry recording who did what in /var/log/syslog:

Sep 14 16:59:35 debian sudo:    david : TTY=pts/11 ; PWD=/var ; USER=root ; TSID=0002RM ; COMMAND=/usr/bin/tail /var/log/syslog

If you are feeling brave you can happily lock the root account completely once you have sudo up and running:

passwd -l root

However, this is only part of the story. Partly because a user can just issue sudo su - and then the audit trail is lost. But also, if we look at some of the subsequent requirements we can see that it means more than just sudo is required.

10.2 Implement automated audit trails for all system components to reconstruct the following events:

10.2.1 All individual accesses to cardholder data

10.2.2 All actions taken by any individual with root or administrative privileges

10.2.3 Access to all audit trails

10.2.4 Invalid logical access attempts

10.2.5 Use of identification and authentication mechanisms

10.2.6 Initialization of the audit logs

10.2.7 Creation and deletion of system-level objects

10.3 Record at least the following audit trail entries for all system components for each event:

10.3.1 User identification

10.3.2 Type of event

10.3.3 Date and time

10.3.4 Success or failure indication

10.3.5 Origination of event

10.3.6 Identity or name of affected data, system component, or resource.

Most of these requirements are covered by the fact you need to be a privileged user, and hence have your actions recorded in syslog via sudo (ignoring sudo su - for the moment), to be able to change anything within the environment.

But the main problem comes from “10.3.4 Success or failure indication”. With sudo you know what the user tried to do but not whether it was successful or not.

$ sudo rm /var/log/i_dont_exist
rm: cannot remove `/var/log/i_dont_exist': No such file or directory
$ sudo tail /var/log/syslog
Sep 14 17:25:28 debian sudo: david : TTY=pts/11 ; PWD=/var ; USER=root ; TSID=0002RQ ; COMMAND=/bin/rm /var/log/i_dont_exist

Also, these requirements mean interactive sessions (eg. psql) initiated by users would have to be logged and sudo does nothing for us there. We would see that the user started a SQL session, but not what they did.

We decided to get round this by using rootsh.

As the SourceForge page explains:

Rootsh is a wrapper for shells which logs all echoed keystrokes and terminal output to a file and/or to syslog. It’s main purpose is the auditing of users who need a shell with root privileges. They start rootsh through the sudo mechanism.

It is intended to be used as a way of accessing root privileges via sudo. The user would call sudo rootsh and from then on, everything which is displayed on the terminal is logged. But given all our Linux users are system administrators, we decide to use rootsh as the default means of accessing the shell on all the servers in the card holder data environment.

It may seem a tad heavy handed but it ensures that all actions, including any interactive sessions like psql, and the results of those actions, are logged which is good news for PCI Requirement 10.

rootsh Installation

We have packaged a .deb of rootsh at 1.5.3-1 to make installation more convenient. If you would like to try this version, add the following to your /etc/apt/sources.list

deb http://debian.qcode.co.uk squeeze main

Then do:

sudo apt-get update
sudo apt-get install rootsh

rootsh Configuration

To make this the default way of accessing a shell, we added the following to the end of /etc/profile:

exec /usr/local/bin/rootsh --no-logfile

The exec is important. We could just call the program directly, but in that case a user could accidentally exit out of rootsh and back out into a standard non-logging shell and we’ve lost our audit trail. exec will replace the current shell with the rootsh process so there is nowhere to back out to.

The --no-logfile option is also important since otherwise, rootsh by default attempts to directly to a file in /var/log as the user in question… who won’t have the permissions to do so. It’s a sure fire way of locking yourself out of the system.

rootsh will log to /var/log/syslog using the local5.notice facility (why this is important to know will be discussed later). For example:

$ sudo rm /var/log/i_dont_exist
rm: cannot remove `/var/log/i_dont_exist': No such file or directory
$ sudo tail /var/log/syslog
Sep 18 16:05:47 debian rootsh[06b92]: david: 033: $ #033[Ksudo rm /var/log/i_dont_exist
Sep 18 16:05:47 debian rootsh[06b92]: david: 034: rm: cannot remove `/var/log/i_dont_exist': No such file or directory
Sep 18 16:06:08 debian rootsh[06b92]: david: 036: $ #033[Ksudo tail /var/log/syslog

Or

$ psql customer1
psql (9.0.3)
Type "help" for help.

customer1 # select order_number from orders order by order_date desc limit 1;
order_number
--------------
550509090
(1 row)
customer1 # \q
$

and in the syslog you'll find

Sep 18 17:26:30 debian rootsh[028c5]: david: 088: customer1 # select order_number from orders order by order_date desc limit 1;
Sep 18 17:26:30 debian rootsh[028c5]: david: 089:
Sep 18 17:26:32 debian rootsh[028c5]: david: 090: order_number
Sep 18 17:26:32 debian rootsh[028c5]: david: 091:
Sep 18 17:26:32 debian rootsh[028c5]: david: 092: --------------
Sep 18 17:26:32 debian rootsh[028c5]: david: 093:
Sep 18 17:26:32 debian rootsh[028c5]: david: 094: 550509090
Sep 18 17:26:32 debian rootsh[028c5]: david: 095:
Sep 18 17:26:32 debian rootsh[028c5]: david: 096: (1 row)

You can see it’s not perfect and the output is sometimes a little obtuse, but you can always piece together who has done what which is what we need. It even means we don’t need to worry about users using sudo su - and the audit chain being broken:

[david@host:~]$ sudo su -
[root@host:~]# cat /etc/secret.txt
This file might contain sensitive information.
[root@host:~]# exit
exit
[david@host:~]$

And in the rootsh log file we can see the following (because another shell has been started, both the user and root sessions are logging giving double entries):

Sep 19 12:06:54 debian rootsh[07653]: david: 058: [david@debian:~#033[0m]$ sudo su -
Sep 19 12:06:55 debian rootsh[037dd]: root: root=root,/dev/pts/2: logging new session (rootsh[037dd])
Sep 19 12:06:55 debian rootsh[07653]: david: 061:
Sep 19 12:07:02 debian rootsh[037dd]: root: 002: [root@debian:~#033[0m]# cat /etc/secret.txt
Sep 19 12:07:02 debian rootsh[07653]: david: 062: [root@debian:~#033[0m]# cat /etc/secret.txt
Sep 19 12:07:02 debian rootsh[07653]: david: 063:
Sep 19 12:07:02 debian rootsh[037dd]: root: 003: This file might contain sensitive information.
Sep 19 12:07:03 debian rootsh[07653]: david: 064: This file might contain sensitive information.
Sep 19 12:07:03 debian rootsh[07653]: david: 067:
Sep 19 12:07:03 debian rootsh[07653]: david: 068:
Sep 19 12:07:05 debian rootsh[037dd]: root: 006: [root@debian:~#033[0m]# exit
Sep 19 12:07:05 debian rootsh[037dd]: root: 007: exit
Sep 19 12:07:05 debian rootsh[037dd]: root: 008: *** rootsh session ended by user
Sep 19 12:07:05 debian rootsh[037dd]: root: 009:
Sep 19 12:07:05 debian rootsh[037dd]: root: root,/dev/pts/2: closing rootsh session (rootsh[037dd])

rsyslog Configuration

Now, if you’re trying for the first time this you’ll probably already have discovered a quirk with this configuration… especially if like me you’re partial to doing tail -f /var/log/syslog (don’t try this just yet please!). Of course, if rootsh is logging everything that occurs on the terminal to /var/log/syslog, and you’re displaying everything that gets logged to /var/log/syslog on the terminal, it doesn’t take long to realise you’ll end up in a rather unpleasant infinite loop that will attempt to fill your /var filesystem in record time.

So you probably want to split off rootsh logging into its own log file. We use rsyslog for logging, and once you know rootsh logs via local5 then it’s quite a simple task to split it off.

In /etc/rsyslog.conf at the start of the logging rules, add:

local5.=notice /var/log/rootsh.log
& ~

The & tells rsyslog to also apply the next action to anything matching the previous line, and the ~ is the discard action. Without this the entries will be logged to syslog as well as your new log file.

logrotate

At this point we must remember that the new log file /var/log/rootsh.log will not be processed by logrotate and so will grown indefinitely. We need to add it to /etc/logrotate.d/rsyslog:

/var/log/rootsh.log
{
        rotate 7
        daily
        missingok
        notifempty
        compress
        postrotate
                invoke-rc.d rsyslog reload > /dev/null
        endscript
}

The meaning of some of these directives are covered in our blog post on centralised logging.

motd

And a final point on rootsh logging. Given everything the system administrators do is being logged, we need to be transparent about this since it’s not about spying, it’s about maintaining a valid audit trail. The message of the day which is displayed upon log in should be updated to tell the users their session is being logged.

If you simply edit the /etc/motd this will work until a restart. A simple way make the change persistent is to add your message to /etc/motd.tail.

Next…

The above creates a robust audit trail which links system actions to an individual user. However, in an environment where the environment is made up of multiple servers (lets face it, with PCI, ALL cardholder data environments will be made up of multiple servers), if your server times aren’t 100% accurate and consistent, your audit trail counts for nothing.

Was result B on server2 at 12:00:00 caused by action A on server1? If the clocks on server1 and server2 are reporting different times, how can you tell at what time action A would have had to have happened? If the clocks are 15mins apart now, that doesn’t tell you they were 15 mins apart when action A occurred.

And this is the subject of our Requirement 10: Part 2 – NTP Time Synchronisation blog post.