Log Files

Table of Contents

Overview

The AppFirst collector will access log data on Linux from files and/or the syslog protocol. Our Windows log information is extracted from files and the Event Log. Events can be created based on log content using both severity levels and explicit strings.

How to add your Logs

Navigate to the Administration – Setup page and click the Logs tab. In the Logs tab, click “Add a new Log” on the right side.

1

On the next screen, select the server you want to ingest log files from.

2

Once you have selected the server from which you want to ingest a log file, select the log type (File or Syslog), the File Path or Port, the limit (number of messages to upload per minute from this source), and an optional filter. See more about filters below.

3

Viewing Logs for a certain Server

In the Servers Page

Once a log file is added for a respective server, you can view a live tailing of those logs in the Servers page. Select the server you want to view and then click the Logs tab to view all your logs for that server. You can hover over a message to see the complete entry.

4

In the Dashboard

You can add a Log Count Metric widget from the Widget Market. Just choose the server, log, an type of metric and click Save.

5

In Correlate

Graph any of your logs in Correlate to see the number of messages over time.

6

Clicking on a point in the graph will give you all the log messages at that specific time:

7

In Log Search and Log Watch

Click here to read more about Log Search and Log Watch.

Alerting on Logs

You have the ability to alert on any Log file added in AppFirst. Navigate to Administration – Setup and click the Alerts tab. Choose the log, a trigger, and the recipients.

8

Linux Logs

Filtering

You can create filters for your log data by navigating to the Administration – Logs and clicking the green pencil icon to the right of the log file. Filters limit the data that is sent based on a regular expression matching of your log data. For example, if you only wanted to see messages that contain “ERROR:,” insert “ERROR:” into your filter and only messages that contain the string “ERROR:” will be sent.

We support the use of POSIX Extended Regular Expression matching for filters. Any POSIX Extended Regular Expression will work as a filter. For a definition of eregex, please see: http://en.wikipedia.org/wiki/Regular_expression

Many useful examples of regular expressions can be found at http://en.wikipedia.org/wiki/Regular_expression#POSIX.

Severity Levels

The collector will assign a severity level of Info, Warning, or Critical to log messages. Severity levels are summarized as log data is aggregated and made available through public REST APIs. These severity levels are graphed in Correlate. The collector calculates severity levels based on the “Syslog Protocol” definition as defined by the IETF standard, RFC 5424. A complete description of the standard can be found at http://www.ietf.org/rfc/rfc3164.txt.

The internet standard for the syslog protocol uses a value described as Priority, which consists of a Facility value and Severity value combined into a single integer. The Priority value is calculated by first multiplying the Facility number by 8 and then adding the numerical value of the Severity.

Examples:

The syslog standard defines a facility value for user-level messages of 1. A network news subsystem is assigned a facility value of 7. The standard defines severity levels of 4 for Warning and 2 for Critical.

For example, a user-level message (1 multiplied by 8, Facility=8) added to a severity of Warning (Severity=4) would have a Priority value of 12 ((1 * 8) + 4). In the PRI part of a syslog message, these values would be placed between the angle brackets: <12>

In another example, a network news subsystem (7 multiplied by 8, Facility=56) added to a severity of Critical (Severity=2) would have a Priority value of 58 ((7 * 8) + 2). In the PRI part of a syslog message, these values would be placed between the angle brackets: <58>

Syslog Protocol

You can specify any port number. By default, syslog loggers will use port 514.

Anything on your network that sends log data to the collector on port 514 will add that log data to the collector data stream. This is important because:

  • Any device on your network can send log data to that IP:port number.
  • We will put that log info into your data stream. You can view that data with Correlate (can see both file and syslog protocol log data with Correlate).
    • This can include network devices (it is common for routers to log using the Syslog protocol).

When collecting log data generated by syslog and written to files, for example, /var/log/messages or /var/log/syslog, there are a few points to be aware of:

  1. Default behavior for syslog, syslog-ng and rsyslog results in no facility or severity values logged to files.
  2. In order to cause facility and severity values to be included in file output, configuration changes are required followed by restart of the syslog service.
  3. AppFirst log data collection supports facility and severity values from files in either RFC3164 format or BSD syslogd format.
  4. The syslog protocol as defined by RFC 5424 and 1364 includes facility and severity values without the need for additional configuration. Defining a log source of type syslog using the AppFirst web application will cause the collector to use a network connection listening on the port provided in the log source definition. In this case the log data will always include facility and severity values.

syslogd file output

  • Facility and severity logging can be enabled by using the command line option “-S” when syslogd is started.
  • With RedHat or CentOS the file /etc/sysconfig/syslog is used to configure options for syslogd.
    • The default option set:
      SYSLOGD_OPTIONS="-m 0"
    • This enables syslogd to perform verbose logging with facility & severity:
      SYSLOGD_OPTIONS="-m 0 -S"
  • Facility and severity values will be output to log files in the form “{facility:severity}”

syslog-ng file output

  • Facility and severity values are not enabled with command line options
  • The configuration file for syslog-ng in Red Hat and CentOS is /etc/syslog-ng/syslog-ng.conf
  • Facility and severity values are logged to files by using a template
  • An example use of a template to include facility & severity values:
    destination af_file {
            file("/var/log/afmessages" 
            template("$ISODATE <$PRI> $MESSAGE\n"));
    };
    log { source(s_sys); destination(af_file); };
    

There are 2 important points to note in the template definition:

  1. A new line character “\n” is required
  2. The macro $PRI must be surrounded by braces; <$PRI>

rsyslog file output

  • Facility and severity values are not enabled with command line options
  • The configuration file for rsyslog in Red Hat and CentOS is /etc/rsyslog.conf
  • Facility and severity values are logged to files by using a template
  • An example use of a template to include facility & severity values:
    $template RFC3164fmt,"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag%%msg%\n"
    *.info;mail.none;authpriv.none;cron.none                /var/log/messages;RFC3164fmt
    

There are 2 important points to note in the template definition:

  1. A new line character “\n” is required
  2. The macro %PRI% must be surrounded by braces; <%PRI%>

Windows Logs

File Log

  • We read one line at a time and add the log data it into the data stream

Event Log

  • You can choose the source of the event log
    • System logs – logged by the OS
    • App logs – logged by the App
    • Security logs – logged by the OS

When choosing the source of the Event log, there’s an additional service that gets started named AppAccessLog.

.NET

The .NET framework is incremental in nature when it comes to updates. A 2008 server may come installed with 2.0 and logic would indicate that one gets the latest software and install it. But what it really requires is that the intermediate releases be installed as well. Unfortunately, the installation of .NET 4.0 doesn’t automatically install the other releases.

If .NET 2.0 is not installed, the installation of 3.5 should install it automatically.

With the 2008 server, one needs to install .NET via Windows Server Manager app (Start – Administrative Tools – Server Manager). Under “Features,” select “add Features,” then open “.NET Framework Features” and select “.NET Framework.” Click “Next,” and then click “Install.” If you select the entire “.NET Framework Features,” you install a couple of other items not needed by AppAccess.

Once that completes, .NET 4 needs to be downloaded from MSDN (http://msdn.microsoft.com/en-us/netframework/) and installed per their instructions.

Once this completes, the user MAY need to be restarted via the Windows Services app.