Skip to main content
Version: User Guide 6.2

Extracting Data from Syslog Messages

You can use a field extractor to extract additional data fields from a syslog message. Field extractors can be added at different points of the message processing pipeline. For example, you can add a field extractor to a syslog parser.

info

To configure parsers, you should have write permissions to the syslog storage. In some cases, you need to run Syslog Watcher Manager as Administrator.

To add a field extractor to a parser:

  1. Click Parsers
  2. Create a new parser or select the one you want to modify
  3. Select Field extractor and click the ... button to open Regular Expression Editor
  4. Enter a regular expression with at least one capturing subexpression.
  5. Click OK and Apply configuration.

Field Extractor Syntax​

A field extractor is a regular expression with a least one capturing group. A capturing group has the format: (?<FIELD>DATA), where:

  • FIELD is one of the standard syslog fields or a custom field. Custom fields start with $.
  • DATA is a subexpression that captures data.

Field Extractor Example​

The easier way to understand how a field extractor works is to look at the example. Here are a few messages from a FortiGate device. The goal is to extract the source IP address + port number and the destination IP address + port number.

Messages from FortiGate device

The Regular Express Editor window allows a sample syslog message to test the edited expression in real-time. To use it for testing, select the entire message text in the Message Details area and copy it to the Clipboard using Ctrl+C.

Adding New Parser​

The first step is to add a new parser. It's more flexible than adding a filed extractor to the default parser, especially if you have different originator types.

Adding new syslog parser

Click the ... button for Field extractor to open Regular Expression Editor.

Editing Field Extractor​

A regular expression that captures the source and the destination can look like this:
srcip=(?<$srcip>.*?) srcport=(?<$srcport>\d*).*dstip=(?<$dstip>.*) dstport=(?<$dstport>\d*)

It extracts data from the message and stores into 4 custom fields: $srcip, $srcport, $dstip, $dstport.

You can paste a sample message copied to the Clipboard to check if the field extractor works in real time.

Editing field extractor expression

Assigning Parser to Originator​

The next step is to assign the created parser to one or more syslog originators. You can do that through the Originators tab. Select one or more originators and click Edit.... Here is an example:

Assigning syslog parser to syslog originator

Using Extracted Fields​

You can use the extracted custom fields where standard fields are used, e.g., forwarders, exporters, viewers, etc.

To add a grid layout: Properties -> Grid Layouts -> New Grid Layout

For example, we created a grid layout with two new columns: Source and Destination. We also deleted columns AppName, ProcID, MsgID as they are blank for this syslog originator.

Creating new grid layout

Opening a new syslog viewer, select the grid layout you've just created:

Assigning syslog parser to syslog originator