Next we are going to look at log file monitoring. Here’s what section 10.6 has to say about it:

10.6 Review logs for all system components at least daily. Log reviews must include those servers that perform security functions like intrusion-detection system (IDS) and authentication, authorization, and accounting protocol (AAA) servers (for example, RADIUS).

Note: Log harvesting, parsing, and alerting tools may be used to meet compliance with Requirement 10.6.

We also have a requirement from 10.5 we still need to address which is:

10.5.5 Use file-integrity monitoring or change-detection software on logs to ensure that existing log data cannot be changed without generating alerts (although new data being added should not cause an alert).

And, to jump ahead, we have this:

11.5 Deploy file-integrity monitoring tools to alert personnel to unauthorized modification of critical system files, configuration files, or content files; and configure the software to perform critical file comparisons at least weekly.

…snip… Examples of files that should be monitored:

  • System executables
  • Application executables
  • Configuration and parameter files
  • Centrally stored, historical or archived, log and audit files

These requirements are all things that OSSEC can help us with. OSSEC is an open source host-based intrusion detection system.

It has many features, but what we are mainly interested in here is:

  • Log Monitoring
  • File Integrity Checking

Logfile Monitoring

OSSEC works by having a Manager and Agents. The Agents forward information to the Manager which applies a set of rules to govern whether alerts need to be raised or not. By default OSSEC monitors:

  • /var/log/messages
  • /var/log/auth.log
  • /var/log/syslog
  • /var/log/mail.info
  • /var/log/dpkg.log
  • /var/log/apache2/error.log
  • /var/log/apache2/access.log

And it has a very good set of rules to cover many events we would like to hear about eg. packages being installed, authentication errors, SQL errors etc.
See later for how to add extra log files to be monitored.

Manager Installation

OSSEC has no Debian package available as standard and has a slightly non-standard installation procedure (from a Debian point of view) which compiles from source via an interactive installation script. It’s easy enough to perform but makes you jump through a few extra hoops if you want to automate the task.
If you’d like to avoid compiling from source on each agent, there is an option do a binary installation of pre-compiled binaries. We found we had too many different architectures for this to be a significant gain however.

Download and unpack the tarball:

cd /usr/local/src
wget http://www.ossec.net/files/ossec-hids-2.6.tar.gz
tar -zxvf ossec-hids-2.6.tar.gz
cd ossec-hids-*

Install some pre-reqs:

apt-get install openssl make gcc libssl-dev dpkg-dev libc-dev libc6-dev linux-libc-dev zlib1g-dev

Rather that use the interactive installation routine (which is ./install.sh if you are happily to install manually), we preferred to distribute a pre-populated /usr/local/src/ossec-hids-2.6/etc/preloaded-vars.conf which would automate the task.

For the server install our /usr/local/src/ossec-hids-2.6/etc/preloaded-vars.conf looked like this:

USER_LANGUAGE="en"     # For english
USER_NO_STOP="y"
USER_INSTALL_TYPE="server"
USER_DIR="/var/ossec"
USER_DELETE_DIR="y"
USER_ENABLE_ACTIVE_RESPONSE="n"
USER_ENABLE_SYSCHECK="y"
USER_ENABLE_ROOTCHECK="y"
USER_UPDATE_RULES="y"

### Server/Local Installation variables. ###

# USER_ENABLE_EMAIL enables or disables email alerting.
USER_ENABLE_EMAIL="y"

# USER_EMAIL_ADDRESS defines the destination e-mail of the alerts.
USER_EMAIL_ADDRESS="security@yourdomain.com"

# USER_EMAIL_SMTP defines the SMTP server to send the e-mails.
USER_EMAIL_SMTP="localhost"

# USER_ENABLE_SYSLOG enables or disables remote syslog.
USER_ENABLE_SYSLOG="n"

# USER_ENABLE_FIREWALL_RESPONSE enables or disables
# the firewall response.
USER_ENABLE_FIREWALL_RESPONSE="n"

#### exit ? ###

You won’t be affected by this now, but during Debian upgrades while having OSSEC installed, you might come across a missing LSB tags and overrides error for OSSEC which breaks the upgrade (it happened to us during the Lenny to Squeeze upgrade). To avoid this we can simply add LSB headers to the existing init script in the source directory.

Replace /usr/local/src/ossec-hids-2.6/src/init/ossec-hids.init with:

#!/bin/sh
# OSSEC         Controls OSSEC HIDS
# Author:       Daniel B. Cid 
# Modified for slackware by Jack S. Lai
### BEGIN INIT INFO
# Provides:          ossec
# Required-Start:    $network $named $remote_fs $syslog
# Required-Stop:     $network $named $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
### END INIT INFO                                                                                   

. /etc/ossec-init.conf
if [ "X${DIRECTORY}" = "X" ]; then
    DIRECTORY="/var/ossec"
fi


start() {
        ${DIRECTORY}/bin/ossec-control start
}

stop() {
        ${DIRECTORY}/bin/ossec-control stop
}

status() {
        ${DIRECTORY}/bin/ossec-control status
}


case "$1" in
  start)
        start
        ;;
  stop)
        stop
        ;;
  restart)
        stop
        start
        ;;
  status)
    status
        ;;
  *)
        echo "*** Usage: $0 {start|stop|restart|status}"
        exit 1
esac

exit 0

Now if we run the installation script, the installation should complete without prompting:

./install.sh

The commands to stop and start the server are:

/var/ossec/bin/ossec-control stop
/var/ossec/bin/ossec-control start

Once installed, we like to tidy up the packages required for the installation (see PCI Requirement 2.2.4 – “Remove all unnecessary functionality, such as scripts, drivers, features, subsystems, file systems, and unnecessary web servers”)

apt-get remove zlib1g-dev patch make  g++ gcc gcc-4.4 g++-4.4 fakeroot dpkg-dev cpp cpp-4.4 bzip2 build-essential binutils

Authd

An issue when performing unattended installs of OSSEC agents is that the manage_agents command must be run to import a key generated by the server for the client. This is a manual process so when installing OSSEC on many servers we need another way.

That other way is authd. Is it a service which runs on the manager and automatically provides keys to agents which connect on the port authd is listening on. It does not do any authentication so never leave the authd daemon running when you are not actively installing agents.

So at this point we need to generate the SSL keys authd requires before we start it for the first time.

openssl genrsa -out /var/ossec/etc/sslmanager.key 2048
openssl req -new -x509 -key /var/ossec/etc/sslmanager.key -out /var/ossec/etc/sslmanager.cert -days 365

To start authd the following command is run:

/var/ossec/bin/ossec-authd -p 1515

Remember to stop it again when the installations are finished.

Firewall Configuration

Since the agents need to be able to connect to the OSSEC manager during install, you need to allow communication from the agents to the manager on 1514 (the standard OSSEC port) and 1515 (the port used for authd).
Port 1515 should only be open when you are actively installing agents.

Agent Installation

This is exactly the same as the manager install except that the /usr/local/src/ossec-hids-2.6/etc/preloaded-vars.conf file used to automate the install is different, and we need to connect to the authd server daemon at the end to to be allocated our key.

Note you must replace 192.168.1.2 with the IP address of your OSSEC manager.

USER_LANGUAGE="en"     # For english
USER_NO_STOP="y"
USER_INSTALL_TYPE="agent"
USER_DIR="/var/ossec"
USER_DELETE_DIR="y"
USER_ENABLE_ACTIVE_RESPONSE="n"
USER_ENABLE_SYSCHECK="y"
USER_ENABLE_ROOTCHECK="y"
USER_UPDATE_RULES="y"

### Agent Installation variables. ###

USER_AGENT_SERVER_IP="192.168.1.2"

#### exit ? ###

Once the install is complete, the final step is to be allocated a key. Using authd this is done by:

/var/ossec/bin/agent-auth -m 192.168.1.2 -p 1515

Adding Alerts

Out of the box, OSSEC will alert and a great many very useful events. But you will almost certainly want to create additional alerts.

As an example, say we wanted OSSEC to generate an alert when the following log entry is found:

Sep 18 10:26:01 client1 stunnel: LOG3[15636:140069884897024]: SSL_accept: Peer suddenly disconnected

We can test this using a useful binary supplied called ossec-logtest. On your manager machine, run ossec-logtest and paste in the log entry in question to see what will currently happen:

[david@ossecmgr:/var/ossec/bin]# sudo ./ossec-logtest
2012/09/26 13:06:39 ossec-testrule: INFO: Reading local decoder file.
2012/09/26 13:06:40 ossec-testrule: INFO: Started (pid: 29628).
ossec-testrule: Type one log per line.

Sep 18 10:26:01 client1 stunnel: LOG3[15636:140069884897024]: SSL_accept: Peer suddenly disconnected

**Phase 1: Completed pre-decoding.
       full event: 'Sep 18 10:26:01 client1 stunnel: LOG3[15636:140069884897024]: SSL_accept: Peer suddenly disconnected'
       hostname: 'client1'
       program_name: 'stunnel'
       log: 'LOG3[15636:140069884897024]: SSL_accept: Peer suddenly disconnected'

**Phase 2: Completed decoding.
       No decoder matched.

So as we already knew, no alert is generated.
The file /var/ossec/rules/local_rules.xml on the OSSEC manager machine is where we can add rules to trigger new alerts. (During upgrades, this rules file is left alone, so if you don’t want to lose your customer rules, keep them here). We won’t go into all the ins and outs of rule creation since there plenty of documentation on the OSSEC website.

We can see it has parsed the program_name, hostname, and log entries. So at the most basic level we can use these to raise our alert. Local rules should use a rule id from 100,000 to 119,999 to ensure they won’t interfere with existing rules.

The following rule is the fairly self-explanatory addition to /var/ossec/rules/local_rules.xml:

 <rule id="100100" level="10">
   <program_name>stunnel</program_name>
   <match>SSL_accept: Peer suddenly disconnected</match>
   <description>Stunnel peer disconnect</description>
 </rule>

So we can try ossec-logtest again with the new rule:

[david@ossecmgr:/var/ossec/bin]# ./ossec-logtest 
2012/09/26 13:24:58 ossec-testrule: INFO: Reading local decoder file.
2012/09/26 13:24:59 ossec-testrule: INFO: Started (pid: 29670).
ossec-testrule: Type one log per line.

Sep 18 10:26:01 client1 stunnel: LOG3[15636:140069884897024]: SSL_accept: Peer suddenly disconnected


**Phase 1: Completed pre-decoding.
       full event: 'Sep 18 10:26:01 client1 stunnel: LOG3[15636:140069884897024]: SSL_accept: Peer suddenly disconnected'
       hostname: 'client1'
       program_name: 'stunnel'
       log: 'LOG3[15636:140069884897024]: SSL_accept: Peer suddenly disconnected'

**Phase 2: Completed decoding.
       No decoder matched.

**Phase 3: Completed filtering (rules).
       Rule id: '100100'
       Level: '10'
       Description: 'Stunnel peer disconnect'
**Alert to be generated.

Success.

Ignoring Alerts

Another problem you are likely to have is receiving alerts which you feel are unnecessary.

One such alert for us was the mail server rejecting emails because of greylisting. This is normal operation as far as we are concerned so we didn’t want to hear about it.

The log file entry in question is:

Apr  6 12:42:51 mailsrv1 postfix/smtpd[20508]: NOQUEUE: reject: RCPT from unknown[183.0.0.1]: 450 4.2.0 : Recipient address rejected: Greylisted, see http://postgrey.schweikert.ch/help/domain.co.uk.html; from= to= proto=ESMTP helo=

Testing this in ossec-logtest we can see where the alert is coming from:

[david@ossecmgr1:/var/ossec/bin]# ./ossec-logtest 
2012/09/26 13:55:46 ossec-testrule: INFO: Reading local decoder file.
2012/09/26 13:55:46 ossec-testrule: INFO: Started (pid: 29698).
ossec-testrule: Type one log per line.

Apr  6 12:42:51 mailsrv1 postfix/smtpd[20508]: NOQUEUE: reject: RCPT from unknown[183.0.0.1]: 450 4.2.0 : Recipient address rejected: Greylisted, see http://postgrey.schweikert.ch/help/domain.co.uk.html; from= to= proto=ESMTP helo=


**Phase 1: Completed pre-decoding.
       full event: 'Apr  6 12:42:51 mailsrv1 postfix/smtpd[20508]: NOQUEUE: reject: RCPT from unknown[183.0.0.1]: 450 4.2.0 : Recipient address rejected: Greylisted, see http://postgrey.schweikert.ch/help/domain.co.uk.html; from= to= proto=ESMTP helo='
       hostname: 'mailsrv1'
       program_name: 'postfix/smtpd'
       log: 'NOQUEUE: reject: RCPT from unknown[183.0.0.1]: 450 4.2.0 : Recipient address rejected: Greylisted, see http://postgrey.schweikert.ch/help/domain.co.uk.html; from= to= proto=ESMTP helo='

**Phase 2: Completed decoding.
       decoder: 'postfix'
       srcip: '183.0.0.1'
       id: '450'

**Phase 3: Completed filtering (rules).
       Rule id: '3303'
       Level: '5'
       Description: 'Sender domain is not found (450: Requested mail action not taken).'
**Alert to be generated.

Now we know the alert is coming from rule id 3303, we can override this in /var/ossec/rules/local_rules.xml:

 <rule id="100102" level="0">
   <if_sid>3303</if_sid>
   <match>Greylisted</match>
   <description>Greylisting ignored</description>
 </rule>

This says the following:

  • level="0" – don’t alert upon matching
  • if_sid – matches if the specified rule id has matched
  • match – matches if the specified string is found in the log entry

So testing once more with this extra rule:

[david@ossecmgr1:/var/ossec/bin]# ./ossec-logtest 
2012/09/26 14:05:58 ossec-testrule: INFO: Reading local decoder file.
2012/09/26 14:05:59 ossec-testrule: INFO: Started (pid: 29709).
ossec-testrule: Type one log per line.

Apr  6 12:42:51 mailsrv1 postfix/smtpd[20508]: NOQUEUE: reject: RCPT from unknown[183.0.0.1]: 450 4.2.0 : Recipient address rejected: Greylisted, see http://postgrey.schweikert.ch/help/domain.co.uk.html; from= to= proto=ESMTP helo=


**Phase 1: Completed pre-decoding.
       full event: 'Apr  6 12:42:51 mailsrv1 postfix/smtpd[20508]: NOQUEUE: reject: RCPT from unknown[183.0.0.1]: 450 4.2.0 : Recipient address rejected: Greylisted, see http://postgrey.schweikert.ch/help/domain.co.uk.html; from= to= proto=ESMTP helo='
       hostname: 'mailsrv1'
       program_name: 'postfix/smtpd'
       log: 'NOQUEUE: reject: RCPT from unknown[183.0.0.1]: 450 4.2.0 : Recipient address rejected: Greylisted, see http://postgrey.schweikert.ch/help/domain.co.uk.html; from= to= proto=ESMTP helo='

**Phase 2: Completed decoding.
       decoder: 'postfix'
       srcip: '183.0.0.1'
       id: '450'

**Phase 3: Completed filtering (rules).
       Rule id: '100101'
       Level: '0'
       Description: 'Greylisting ignored'

So we shouldn’t hear about these any more.

File Integrity Checking

Now you will be receiving regular alerts on log file entries, we will focus on the other use of OSSEC which helps us in PCI. Namely:

11.5 Deploy file-integrity monitoring tools to alert personnel to unauthorized modification of critical system files, configuration files, or content files; and configure the software to perform critical file comparisons at least weekly.

…snip…
Examples of files that should be monitored:\

  • System executables\
  • Application executables\
  • Configuration and parameter files\
  • Centrally stored, historical or archived, log and audit files

OSSEC provides us with syscheck which allows us to preform file integrity checking.

By default, OSSEC will scan everything in the following directories every 22 hours and raise an alert if anything about the files changes (eg. checksum, owner, permissions etc.):

  • /etc
  • /usr/bin
  • /usr/sbin
  • /bin
  • /sbin

These are configured in each agent’s /var/ossec/etc/ossec.conf
So as a basic level we will hear about configuration files and executables being changed. But we need to carefully configure each agent so “critical system files, configuration files, or content files” are all included in the syscheck.

For example, on a webserver with a public Javascript directory, you will want to alert on changes to this also.

Do add this, on the webserver in question, edit /var/ossec/etc/ossec.conf and add to the syscheck section:

    <directories check_all="yes">/var/www/Javascript</directories>

Again, using unattended installation methods, we preferred to avoid having to have each agent be configured with a different ossec.conf, and OSSEC have provided a means to standardise the configuration on all agents.

Shared agent.conf

On the OSSEC manager you can create a file called /var/ossec/etc/shared/agent.conf in which you can specify syscheck entries for specific agents. This file will then be automatically distributed to all agents, and if an agent finds a configuration entry with it’s name on it, it will load it.

We had some issues with the OSSEC’s automatic file distribution, so if you do also, feel free to distribute the file yourselves to each agent. It will have the same effect.

Adding Log Files for Monitoring

It so happens, /var/ossec/etc/shared/agent.conf is where we can add extra log files to be parsed for log file monitoring.

To add /var/log/app1/app1.log on APP_SRV1 we simply add the following to /var/ossec/etc/shared/agent.conf:

<agent_config name="APP_SRV1">
    <localfile>
        <location>/var/log/app1/app1.log</location>
        <log_format>syslog</log_format>
    </localfile>
</agent_config>

Log File Integrity

So to the final part:

10.5.5 Use file-integrity monitoring or change-detection software on logs to ensure that existing log data cannot be changed without generating alerts (although new data being added should not cause an alert).

Referring back to out post about PCI DSS Centralised Logging, we can use /var/ossec/etc/shared/agent.conf to tell our log server to monitor the audit logs in /var/log/hosts for changes. Bear in mind, we cannot monitor everything in there because logs, being logs, are changing constantly. We have to accept that and only monitor the rotated files.

So putting this together, we can create a /var/ossec/etc/shared/agent.conf which will monitor log file integrity on the log server, javascript integrity on the webserver, and parse app1.log on the app server:

<agent_config name="WEB_SRV1">
  <syscheck>
      <directories check_all="yes">/var/www/Javascript</directories>
  </syscheck>
</agent_config>

<agent_config name="LOG_SRV1">
  <syscheck>
      <directories check_all="yes">/var/log/hosts</directories>
      <ignore type="sregex" >.log$</ignore>
  </syscheck>
</agent_config>

<agent_config name="APP_SRV1">
    <localfile>
        <location>/var/log/app1/app1.log</location>
        <log_format>syslog</log_format>
    </localfile>
</agent_config>

You can see how we’ve excluded all the files ending .log with a sregex match since they will constantly be written to.

The finished file will of course be a lot larger to cover your full file integrity needs but this is the basic structure you can use to achieve your aims.