2.1.3. Sigflow engine

2.1.3.1. Presentation

The Sigflow engine analyses all network traffic and can, according to the rules, generate alerts, metadata, and content.
Coming from different sources, these rules must describe the characteristics of the attacks to be detected as well as being optimised to reduce false positives.
The GCenter enables compiling several sources, thus several rule sets to create a specific file called Ruleset that will then be transferred to the Sigflow engine in the GCap.

2.1.3.2. Organizing the rules

The rules sent to the Sigflow engine are organized as follows:

A list of sources providing signature set files grouped in categories
A list of signatures capable of adapting to the needs of the environment to be monitored
A list of Ruleset enabling signatures to be linked to a GCap

Note

Managing the sources, categories, rules, and rulesets is done in the rules manager built into the GCenter.

The following paragraphs describe the steps required to provide these rules in Ruleset form to the Sigflow module of the GCap through the GCenter:

Management of available rule sources: see paragraph Sigflow engine signature sources

Sources are used to report repositories where signatures are made available.

Once downloaded and unzipped, the rules must be added to the GCenter interface.

These sources update automatically in the case of public source/HTTP if the GCenter is connected to the internet, otherwise, a manual update can be made at this interface in order to have the latest signatures of available.
The creation of rulesets from the sources: see the paragraph Rulesets

Once the rules have been added, it is possible to directly assign this source to different rulesets
The generation of rulesets: refer to Generating rulesets
Applying rulesets to the GCap: refer to Detection Rulesets
Advanced configuration of GCap parameters: refer to GCAP Profiles

2.1.3.3. Sigflow engine signature sources

The sources include Suricata type detection signatures.

There are native sources: CTI, ETPRO.

It is also possible to add public sources to the rules manager.
Each source issues a file of rule sets, grouped into categories.
It is possible to manage categories and rules.
The rules manager enables to:

Define and manage the sources of signatures for the detection engine
Manage the rule set files made available by the sources
Manage the categories and rules of these files
View the rules contained in the available sources

The signatures are updated using the Gatewatcher Update Manager (GUM).

The user (operator) can directly assign one or more sources to a Ruleset.

The source management manager is described in the paragraph `Config - sigflow/sources` screen of the legacy web UI.

For implementation, see SIGFLOW engine rule sources.

2.1.3.4. Rulesets

Ruleset is the security and detection policy to be applied to the GCap.
As noted above, a ruleset is therefore composed of sources that include the various detection signatures.
The creation of the ruleset is mandatory for the GCap probe to be able to analyze the network traffic and raise alerts.
Modifications can be made to signatures or categories in order to adapt a rule to the specificities of the information system or to a particular need.

The source management manager is described in the paragraph `Config - sigflow/sources` screen of the legacy web UI.

For implementation, see Creating a SIGFLOW engine ruleset.

2.1.3.4.1. Optimization of rulesets

Optimization of a ruleset consists of adapting it to the type of traffic analyzed and the different collection points.
The ruleset can be edited at any time by adding, modifying, or deleting rules, categories, or sources.
A modified ruleset must be applied to the GCap in order for the changes to take effect.

For implementation, see Creating a SIGFLOW engine ruleset.

2.1.3.4.2. Changing signatures

Signatures and categories can be enabled, disabled, or modified for a given Ruleset.

It is possible to:

Activate signatures and categories
Deactivate signatures and categories
It is possible to modify the functioning of a signature in order to adapt it to the supervised information system by setting up a threshold or suppression of alert for a specific network.

It is possible to modify a signature in the following way:

Create a Suppress Rule on a rule: removes the raising of an alert according to a source or destination IP
Create a Threshold Rule: limit the number of alerts to be displayed based on a source or destination IP

Note

It is not possible to directly interact with the rule content from the rule manager.

For its implementation, see the procedure Modifying SIGFLOW engine rules.

2.1.3.4.2.1. Definition of signatures

Signatures in the CTI and ETPRO sources can include references to blogs, CVEs, websites, and the like that can be accessed from the rules manager.

To better understand how a signature works, here is an example of a signature:

A signature is always made up of:

An action
A protocol
Network parameters such as IP and source and destination port
A message

Example of a signature:

The following protocols may be subject to a rule:

This signature is composed of:

The first part "action": corresponds to the action to be performed in case of detection. For example: "alert, pass, drop..."
The 2nd part "the header": it is this part that allows to define the meaning of the alert as well as the networks and protocols. For example: "tcp any -> any "

This part is composed of:

"Protocol" part: indicates the monitored protocol. For example: "tcp, udp, icmp"

"Source and destination" section: specifies the source of traffic and the destination of traffic respectively. It is possible to assign IP addresses (IPv4 and IPv6 are supported) and IP ranges with operations if needed.

It is also possible to use variables such as $HOME_NET and $EXTERNAL_NET that are managed in the `Net variables` section of the `Config Gcaps profiles` menu and Net variables

These variables are used to increase the accuracy of the alerts provided by the signatures.

The following syntax can be used to specify the addresses:

The part "Ports (source and destination)" : indicates the ports : the first is the source port, the second is the destination port (see the direction of the directional arrow).

Traffic between and out of ports. Different protocols have different port numbers.

For example, the default port for HTTP is 80 while 443 is usually the port for HTTPS.

Note however that the port does not dictate which protocol is used in the communication. Rather, it determines which application receives the data.

The list below gives examples of port specifications.

the "Direction" part: indicates the direction of the flow.

The last part corresponds to the options applied to the rule.

These are surrounded by parentheses and separated by semicolons.

Some options have parameters (such as msg), which are specified by the option keyword, followed by a colon, followed by the parameters.

Others have no parameters, they are simply the keyword (like nocase).

For its implementation, see the procedure Modifying SIGFLOW engine rules.

2.1.3.4.3. Generating rulesets

Important

As long as the rulesets have not been generated after modifications, no configuration will be deployed on the GCap.

Once the ruleset is created, it needs to be generated to validate the creation. It will then be possible to transfer to the GCap.

For implementation, see the Generating a SIGFLOW engine ruleset.

2.1.3.4.3.1. Secret Local Rule

It is also possible to define certain rules locally on a GCap probe that will intentionally not appear in the GCENTER rule manager.

This may occur in the following instances:

Making signatures confidential without the GCenter operators being able to see them ('need-to-know' concept)
In complex cases, perform a local modification on the probe signatures
If the GCenter is entrusted to a third party and the latter cannot handle markers or signatures of a certain level

This procedure is detailed in the GCAP-documentation in the section Add secret rules locally.

2.1.3.5. GCAP Profiles

From this configuration interface, users will be able to apply specific policy rules. They can customize the settings from the following categories:

Detection Rulesets
Base variables
Net variables
Flow timeouts
Files rules management
Packet filtering

In order for the GCap to raise alerts, the user must first apply a ruleset to the GCap.
The GCap profile manager and its configurations are described in the Web UI `Config - Gcaps profiles` screen.
For implementation, see Configuring GCaps.

2.1.3.5.1. Detection Rulesets

The `Detection Rulesets` section enables applying Rulesets to the GCap paired with the GCenter.

It is also possible to configure the Codebreaker module by enabling or disabling shellcode and powershell detection.

Note

It is necessary to generate rules for a ruleset before applying it to GCAPs. Failure to do so will result in no rules being applied to the probe.

Note

Codebreaker is configurable via the `Detection Rulesets` menu, but unfairly so for licenses with this module

The GCap `Detection Rulesets` menu enables three configuration options:

The Single-tenant
The Multi-tenant by interface
The Multi-tenant by vlan

Note

These configuration options are exclusive. This means that it will not be possible to apply a single tenant and multi-tenant configuration at the same time.

2.1.3.5.1.1. Single-tenant

Single-tenant mode enables:

Assigning a ruleset for all GCap monitoring interfaces
Enabling/disabling the Codebreaker module for all GCap monitoring interfaces

The detection ruleset manager is described in the `Detection Rulesets` section of the `Config Gcaps profiles` menu.

For implementation, see Configure Codebreaker then apply the Sigflow rulesets to the GCaps.

2.1.3.5.1.2. Multi-tenant by interface

The Multi-tenant by interface mode enables:

Assigning a ruleset per GCap monitoring interface
Enabling/disabling the Codebreaker module per GCap monitoring interface

The multi-tenant by interface enables applying a ruleset for each of the GCap's interfaces and therefore having a different monitoring per interface.

Indeed, it is possible to apply a different ruleset, as well as to configure Codebreaker for each of the GCap interfaces.

The detection ruleset manager is described in the `Detection Rulesets` section of the `Config Gcaps profiles` menu.

For implementation, see Configure Codebreaker then apply the Sigflow rulesets to the GCaps.

2.1.3.5.1.3. Multi-tenant by vlan

The Multi-tenant by vlan mode enables:

Assigning one ruleset per vlan
Assigning a ruleset for the default vlan for those vlans not created via the interface
Enabling/disabling the Codebreaker module per vlan
Enabling/disabling the Codebreaker module for the default vlan i.e. vlans not created via the interface

The multi-tenant by vlan enables a configuration to be applied for each vlan previously created in the interface and to have distinct monitoring on different networks.
Thus, it is possible to apply a ruleset as well as to configure Codebreaker independently for each vlan.
A vlan named "default" is created as standard in the interface. It enables a ruleset to be applied and Codebreaker to be configured for all vlans not explicitly specified in the interface.

The detection ruleset manager is described in the `Detection Rulesets` section of the `Config Gcaps profiles` menu.

For implementation, see Configure Codebreaker then apply the Sigflow rulesets to the GCaps.

2.1.3.5.2. Base variables

The Base variables section enables the probe's capture parameters to be adjusted using the advanced Sigflow functions that can be configured from GCenter.
Changes to this configuration have an impact on the alerts sent from the GCap probe to the GCENTER.
Enabling certain options will enable the sending of alerts, anomalies, metadata, file information, and protocol-specific records.
Alerts are records of events triggered by the matching of a rule with network traffic.
An alert will be created with associated metadata, such as the application layer record (HTTP, DNS, etc).
The graphical interface of this base variable manager is described in paragraph `Base variables` section of the `Config Gcaps profiles` menu.

The basic variables section is composed of:

Stream analysis and file extraction
HTTP Proxy
Payload
Community ID
Alerting and logging

2.1.3.5.2.1. Stream analysis and file extraction

The stream analysis and file extraction area enables you to control how the Sigflow engine handles maximum stream and file extraction sizes.

The graphical interface and the details of the parameters are described in the `Stream analysis and file extraction` zone.

2.1.3.5.2.2. HTTP Proxy

The HTTP Proxy area enables enhanced metadata and alerts for streams mandated with the X-Forwarded-For (XFF) http header.

The graphical interface and the details of the parameters are described in the `HTTP Proxy` zone.

Note

XFF is a standard header enabling to identify the original IP address of a client connecting to a web server through an HTTP proxy or load balancer.

2.1.3.5.2.3. Payload

The Payload section enables enabling or disabling various fields present (Http body, Payload printable, Http body printable...) in the events.

The graphical interface and the details of the parameters are described in the `Payload` zone.

2.1.3.5.2.4. Community ID

The Community ID section allows this field to be enabled or disabled in events.

This enables identifying the network streams being analyzed.

Note

This field is present in solutions other than Gatewatcher. It can therefore enable correlating the different tools of the same information system.

The graphical interface and the details of the parameters are described in the `Community ID` zone.

2.1.3.5.2.5. Alerting and logging

The Alerting and logging section enables configuring the alerting and logging of the protocols used by the Gcap.

Note

If GCap is one version ahead of the GCenter, it is possible that some protocols are not yet implemented in the latter.

This is discussed in more detail in the GCAP-documentation in the section Sigflow detection engine > Rebuilding files.

Terminology for parsing and logging:

alerting consists in activating the Sigflow signature detection for a given protocol. Indeed, if Sigflow is activated for a protocol, then the stream that is identified by a signature will raise an alert on the GCenter side.
logging consists in enabling the generation of metadata for a given protocol. Indeed, if the latter is enabled for a protocol, then each observed session will generate metadata for that protocol on the GCenter side.

Managing the configuration of the protocols is done:

by using default profiles such as Minimal, Balanced, MPL, Paranoid, and Intuitio: in order from most to least permissive
by modifying the content of the profile uploaded to the GCap.

The choice of the default profile and the loading on the GCap is done in the graphical interface detailed in the `Admin-GCaps pairing and status` screen of the legacy Web UI.
Changing the configuration of the profile loaded into the GCap is done in the graphical interface detailed in `Alerting and logging` zone.
The default configuration of protocols varies depending on the GCap profile used: the list is shown in Default settings for existing profiles available.
The list of protocols that can be configured with the alerting option and with the logging option is provided in the paragraph Default settings for existing profiles available.
For implementation, see Configure GCap Sigflow module specific parameters (Base variables).

2.1.3.5.3. Net variables

The Net variables area of the GUI enables defining the network variables used in the Sigflow rules.
The list of variables and their default values are given in `Net variables` section of the `Config Gcaps profiles` menu.
This graphical interface is described in `Net variables` section of the `Config Gcaps profiles` menu.
For implementation, see Configure network variables used by rules (Net variables).

2.1.3.5.4. Flow timeouts

The Flow timeouts section enables configuring the time in seconds that Sigflow keeps a flow in memory depending on its status.
The list of variables and their default values are given in `Flow timeouts` section of the `Config Gcaps profiles` menu.
This graphical interface is described in `Flow timeouts` section of the `Config Gcaps profiles` menu.

2.1.3.5.5. Files rules management

The Files rules management section enables choosing the file types that the probe will retrieve for a given protocol.
File extraction works in parallel with the Sigflow signatures defined for these same protocols.
Files are reconstructed and then saved to disk with metadata that includes information such as:

Time stamp
Source/destination IP address
Protocol
Source/destination port
Size
Md5sum, etc.

Managing the configuration of the file extraction rules is done:

By using default profiles such as Minimal, Balanced, MPL, Paranoid, and Intuitio: in order from most to least complete
By modifying the content of the profile uploaded to the GCap.

The choice of the default profile and the loading on the GCap is done in the graphical interface detailed in the `Admin-GCaps pairing and status` screen of the legacy Web UI.
Changing the configuration of the profile loaded into the GCap is done in the graphical interface detailed in `File rule management` section of the `Config Gcaps profiles` menu.
The list of protocols as well as the list of file types are given in the Web UI `Config - Gcaps profiles` screen.

For implementation, see Configure File Reconstruction Rules (File rules management).

2.1.3.5.6. Packet filtering

The `Packet filters` section enables specific traffic to be ignored directly at the GCap network card level.
This feature enables the GCap to avoid overloading the GCap with "unnecessary" traffic such as encrypted streams, backup streams, etc., or traffic that may cause the cpus to overload such as Elephant Flow, Miles Flow, etc.
The selection of traffic to be ignored is based on vlan, network prefix, protocol, and network ports.

This graphical interface is described in `Packet filters` section of the `Config Gcaps profiles` menu.

For implementation, see Configure filters on targeted parts of the analyzed traffic (Packet filters).

2.1.3.6. Events generated

The events generated by Sigflow are:

Events of type "alert" when the engine has found a match with the signature defined in the alert rules.

Events are displayed in the main GCenter interface as well as in Kibana.

In the main interface, it is possible to view the found signature and its definition.

In the main interface named WEB UI of the GCenter in the `Alerts` screen (the main interface named WEB UI is described in the WEB UI Overview).

To view the alerts, you must select the IDS filter and thus view the list of alerts: see the presentation of the `Alerts` screen of the web UI.

By clicking on an alert, the detailed information of this alert is displayed: see Counters of the source part of Sigflow logs of alert type.

In the Kibana UI interface

To view the alerts, select the Sigflow filter and view the list of alerts: see the `Alerts` screen of the web UI.

By clicking on an alert, select on the command `Flow details` then select the arrow to the left of the alert.

The interface displayed is the interface named Kibana UI (described in the Kibana GUI Overview).

The detailed information of this alert can be viewed in table or jason format.

The detailed information of this alert is displayed: see Counters of the source part of Sigflow logs of alert type.
Events of type "fileinfo" when the Sigflow engine has reconstructed the network flow files`.

These fileinfo events are visible in the Kibana interface.

To do this, select the Hunting command, then the tab `Sigflow` and the category `Messages`, then select the arrow to the left of the event.

The detailed information of this event can be viewed in table or jason format.

The detailed information of this event is displayed (see Events of type "fileinfo").
Events of type "meta-data" when the Sigflow engine has recognized the network flow.

These metadata events are visible in the Kibana interface.

To do this, select on the Hunting command then the tab `Metadata` and the category `Messages`, then select the arrow to the left of the event.

The detailed information of this event can be viewed in table or jason format.

The detailed information of this event is displayed (see Metadata counters)

These events (or logs) are structured in the same way and this structure is presented in the paragraph Example and log structure (Events) Sigflow.

2.1.3.6.1. Example and log structure (Events) Sigflow

2.1.3.6.1.1. Exemple d'un log Sigflow

Below is an example of log sigflow (alert log) is displayed:

{
"_index": "suricata-2023.10.09-000162",
"_type": "_doc",
"_id": "lb-iE4sBeBoubSygsnlA",
"_version": 1,
"_score": 1,
"_source": {
  "proto": "TCP",
  "@timestamp": "2023-10-09T08:51:50.124Z",
  "uuid": "1b6e03fa-d325-4f3d-ac54-e2dd6152c8fd",
  "timestamp_analyzed": "2023-10-09T08:51:50.124Z",
  "gcap": "gcap-xxxxxxxxx.domain.local",
  "host": "gcap-xxxxxxxxx.domain.local",
  "flow": {
    "pkts_toclient": 4,
    "pkts_toserver": 4,
    "bytes_toserver": 421,
    "bytes_toclient": 647,
    "start": "2023-10-09T08:51:17.974120+0000"
  },
  "dest_port": 48740,
  "alert": {
    "signature_id": 2034636,
    "rev": 2,
    "action": "allowed",
    "severity": 3,
    "metadata": {
      "updated_at": [
        "2021_12_08"
      ],
      "attack_target": [
        "Client_Endpoint"
      ],
      "former_category": [
        "INFO"
      ],
      "signature_severity": [
        "Minor"
      ],
      "deployment": [
        "Perimeter"
      ],
      "affected_product": [
        "Windows_XP_Vista_7_8_10_Server_32_64_Bit"
      ],
      "created_at": [
        "2021_12_08"
      ]
    },
    "signature": "ET INFO Python SimpleHTTP ServerBanner",
    "gid": 1,
    "category": "Misc activity"
  },
  "type": "suricata",
  "packet": "ivOZFqBXfu+N6xq...AqAABwKgAAwBQvmQfXmk2N+fYu4AYAfzRowAAAQEIClfrLQ32MlIpUEsDBAoAAAAAAOCYuCg8z1FoRAAAAEQAAAAJAAAAZWljYXIuY29tWDVPIVAlQEFQWzRcUFpYNTQoUF4pN0NDKTd9JEVJQ0FSLVNUQU5EQVJELUFOVElWSVJVUy1URVNULUZJTEUhJEgrSCpQSwECFAAKAAAAAADgmLgoPM9RaEQAAABEAAAACQAAAAAAAAABACAA/4EAAAAAZWljYXIuY29tUEsFBgAAAAABAAEANwAAAGsAAAAAAA==",
  "timestamp_detected": "2023-10-09T08:51:17.976Z",
  "dest_ip": "X.X.X.X",
  "src_port": 80,
  "ether": {
    "dest_mac": "7e:ef:8d:eb:1a:81",
    "src_mac": "8a:f3:99:16:a0:57"
  },
  "community_id": "1:r6LvcE7ltny4a6Y9xt1VroBgcKs=",
  "event_type": "alert",
  "@version": "1",
  "severity": 3,
  "stream": 1,
  "flow_id": 1666858537573672,
  "gcenter": "gcenter-xxx.domain.local",
  "payload_printable": "HTTP/1.0 200 OK\r\nServer: SimpleHTTP/0.6 Python/3.7.3\r\nDate: Thu, 20 Aug 2020 08:27:52 GMT\r\nContent-type: application/zip\r\nContent-Length: 184\r\nLast-Modified: Thu, 20 Aug 2020 08:26:32 GMT\r\n\r\n",
  "src_ip": "X.X.X.X",
  "in_iface": "monvirt",
  "http": {
    "hostname": "eicar.com",
    "status": 200,
    "http_user_agent": "Wget/1.20.1 (linux-gnu)",
    "url": "/eicar_com.zip",
    "http_content_type": "application/zip",
    "length": 0,
    "http_method": "GET",
    "protocol": "HTTP/1.1"
  },
  "payload": "SFRUUC8xLjAgMjAwIE9LDQpTZXJ2ZXI6IFNpbXBsZUhUVFAvMC42IFB5dGhvbi8zLjcuMw0KRGF0ZTogVGh1LCAyMCBBdWcgMjAyMCAwODoyNzo1MiBHTVQNCkNvbnRlbnQtdHlwZTogYXBwbGljYXRpb24vemlwDQpDb250ZW50LUxlbmd0aDogMTg0DQpMYXN0LU1vZGlmaWVkOiBUaHUsIDIwIEF1ZyAyMDIwIDA4OjI2OjMyIEdNVA0KDQo=",
  "tx_id": 0,
  "app_proto": "http",
  "packet_info": {
    "linktype": 1
  }
},
"fields": {
  "alert.category": [
    "Misc activity"
  ],
  "alert.metadata.signature_severity": [
    "Minor"
  ],
  "http.url": [
    "/eicar_com.zip"
  ],
  "type": [
    "suricata"
  ],
  "uuid": [
    "1b6e03fa-d325-4f3d-ac54-e2dd6152c8fd"
  ],
  "alert.metadata.created_at": [
    "2021_12_08"
  ],
  "event_type": [
    "alert"
  ],
  "payload": [
    "SFRUUC8xLjAgMjAwIE9LDQpTZXJ2ZXI6IFNpbXBsZUhUVFAvMC42IFB5dGhvbi8zLjcuMw0KRGF0ZTogVGh1LCAyMCBBdWcgMjAyMCAwODoyNzo1MiBHTVQNCkNvbnRlbnQtdHlwZTogYXBwbGljYXRpb24vemlwDQpDb250ZW50LUxlbmd0aDogMTg0DQpMYXN0LU1vZGlmaWVkOiBUaHUsIDIwIEF1ZyAyMDIwIDA4OjI2OjMyIEdNVA0KDQo="
  ],
  "flow_id": [
    1666858537573672
  ],
  "host": [
    "gcap-xxxxxxxxx.domain.local"
  ],
  "ether.src_mac": [
    "8a:f3:99:16:a0:57"
  ],
  "alert.metadata.former_category": [
    "INFO"
  ],
  "dest_port": [
    48740
  ],
  "alert.severity": [
    3
  ],
  "gcenter": [
    "gcenter-xxx.domain.local"
  ],
  "flow.bytes_toclient": [
    647
  ],
  "packet": [
    "ivOZFqBXfu+N6xq...AqAABwKgAAwBQvmQfXmk2N+fYu4AYAfzRowAAAQEIClfrLQ32MlIpUEsDBAoAAAAAAOCYuCg8z1FoRAAAAEQAAAAJAAAAZWljYXIuY29tWDVPIVAlQEFQWzRcUFpYNTQoUF4pN0NDKTd9JEVJQ0FSLVNUQU5EQVJELUFOVElWSVJVUy1URVNULUZJTEUhJEgrSCpQSwECFAAKAAAAAADgmLgoPM9RaEQAAABEAAAACQAAAAAAAAABACAA/4EAAAAAZWljYXIuY29tUEsFBgAAAAABAAEANwAAAGsAAAAAAA=="
  ],
  "tx_id": [
    0
  ],
  "http.length": [
    0
  ],
  "timestamp_detected": [
    "2023-10-09T08:51:17.976Z"
  ],
  "http.hostname": [
    "eicar.com"
  ],
  "flow.bytes_toserver": [
    421
  ],
  "dest_ip": [
    ""X.X.X.X""
  ],
  "proto": [
    "TCP"
  ],
  "gcap": [
    "gcap-xxxxxxxxx.domain.local"
  ],
  "timestamp_analyzed": [
    "2023-10-09T08:51:50.124Z"
  ],
  "alert.metadata.deployment": [
    "Perimeter"
  ],
  "http.http_user_agent": [
    "Wget/1.20.1 (linux-gnu)"
  ],
  "http.http_method": [
    "GET"
  ],
  "alert.metadata.attack_target": [
    "Client_Endpoint"
  ],
  "ether.dest_mac": [
    "7e:ef:8d:eb:1a:81"
  ],
  "alert.metadata.affected_product": [
    "Windows_XP_Vista_7_8_10_Server_32_64_Bit"
  ],
  "flow.pkts_toclient": [
    4
  ],
  "http.http_content_type": [
    "application/zip"
  ],
  "src_ip": [
    "X.X.X.X"
  ],
  "community_id": [
    "1:r6LvcE7ltny4a6Y9xt1VroBgcKs="
  ],
  "stream": [
    "1"
  ],
  "alert.rev": [
    2
  ],
  "@version": [
    "1"
  ],
  "alert.signature_id": [
    "2034636"
  ],
  "alert.action": [
    "allowed"
  ],
  "packet_info.linktype": [
    1
  ],
  "severity": [
    3
  ],
  "payload_printable": [
    "HTTP/1.0 200 OK\r\nServer: SimpleHTTP/0.6 Python/3.7.3\r\nDate: Thu, 20 Aug 2020 08:27:52 GMT\r\nContent-type: application/zip\r\nContent-Length: 184\r\nLast-Modified: Thu, 20 Aug 2020 08:26:32 GMT\r\n\r\n"
  ],
  "http.protocol": [
    "HTTP/1.1"
  ],
  "app_proto": [
    "http"
  ],
  "in_iface": [
    "monvirt"
  ],
  "src_port": [
    80
  ],
  "flow.start": [
    "2023-10-09T08:51:17.974Z"
  ],
  "alert.gid": [
    1
  ],
  "@timestamp": [
    "2023-10-09T08:51:50.124Z"
  ],
  "alert.signature": [
    "ET INFO Python SimpleHTTP ServerBanner"
  ],
  "flow.pkts_toserver": [
    4
  ],
  "http.status": [
    "200"
  ],
  "alert.metadata.updated_at": [
    "2021_12_08"
  ]
}
}

2.1.3.6.1.2. Structure of sigflow logs

The logs are composed of different parts:

the leading part
the source part defined by "_source"
the field portion defined by "_fields"

The header part of the sigflow logs

The header section contains:

{
 "_index": "suricata-2023.10.09-000162",
 "_type": "_doc",
 "_id": "lb-iE4sBeBoubSygsnlA",
 "_version": 1,
 "_score": 1,

Table part header of logs sigflow
Field	Required	Description	Values or example
_index	Yes	Internal index	suricata-2023.10.09-000162
_type	Yes	default type	_doc
_id	Yes	internal identifier	lb-iE4sBeBoubSygsnlA
_version	Yes	internal version	1
_score	Yes	relevance of the response to the request	1

The source part of the sigflow logs

The source part defined by "_source" contains:

{
 "_index": "suricata-2023.10.09-000162",
 "_type": "_doc",
 "_id": "lb-iE4sBeBoubSygsnlA",
 "_version": 1,
 "_score": 1,
 "_source": {
   "proto": "TCP",
   "@timestamp": "2023-10-09T08:51:50.124Z",
   "uuid": "1b6e03fa-d325-4f3d-ac54-e2dd6152c8fd",
   "timestamp_analyzed": "2023-10-09T08:51:50.124Z",
   "gcap": "gcap-xxxxxxxxx.domain.local",
   "host": "gcap-xxxxxxxxx.domain.local",
   "flow": {
     "pkts_toclient": 4,
     "pkts_toserver": 4,
     "bytes_toserver": 421,
     "bytes_toclient": 647,
     "start": "2023-10-09T08:51:17.974120+0000"
   },
   "dest_port": 48740,
   "alert": {
     "signature_id": 2034636,
     "rev": 2,
     "action": "allowed",
     "severity": 3,
     "metadata": {
       "updated_at": [
         "2021_12_08"
       ],
       "attack_target": [
         "Client_Endpoint"
       ],
       "former_category": [
         "INFO"
       ],
       "signature_severity": [
         "Minor"
       ],
       "deployment": [
         "Perimeter"
       ],
       "affected_product": [
         "Windows_XP_Vista_7_8_10_Server_32_64_Bit"
       ],
       "created_at": [
         "2021_12_08"
       ]
     },
     "signature": "ET INFO Python SimpleHTTP ServerBanner",
     "gid": 1,
     "category": "Misc activity"
   },
   "type": "suricata",
   "packet": "ivOZFqBXfu+N6xq...AqAABwKgAAwBQvmQfXmk2N+fYu4AYAfzRowAAAQEIClfrLQ32MlIpUEsDBAoAAAAAAOCYuCg8z1FoRAAAAEQAAAAJAAAAZWljYXIuY29tWDVPIVAlQEFQWzRcUFpYNTQoUF4pN0NDKTd9JEVJQ0FSLVNUQU5EQVJELUFOVElWSVJVUy1URVNULUZJTEUhJEgrSCpQSwECFAAKAAAAAADgmLgoPM9RaEQAAABEAAAACQAAAAAAAAABACAA/4EAAAAAZWljYXIuY29tUEsFBgAAAAABAAEANwAAAGsAAAAAAA==",
   "timestamp_detected": "2023-10-09T08:51:17.976Z",
   "dest_ip": "X.X.X.X",
   "src_port": 80,
   "ether": {
     "dest_mac": "7e:ef:8d:eb:1a:81",
     "src_mac": "8a:f3:99:16:a0:57"
   },
   "community_id": "1:r6LvcE7ltny4a6Y9xt1VroBgcKs=",
   "event_type": "alert",
   "@version": "1",
   "severity": 3,
   "stream": 1,
   "flow_id": 1666858537573672,
   "gcenter": "gcenter-xxx.domain.local",
   "payload_printable": "HTTP/1.0 200 OK\r\nServer: SimpleHTTP/0.6 Python/3.7.3\r\nDate: Thu, 20 Aug 2020 08:27:52 GMT\r\nContent-type: application/zip\r\nContent-Length: 184\r\nLast-Modified: Thu, 20 Aug 2020 08:26:32 GMT\r\n\r\n",
   "src_ip": "X.X.X.X",
   "in_iface": "monvirt",
   "http": {
     "hostname": "eicar.com",
     "status": 200,
     "http_user_agent": "Wget/1.20.1 (linux-gnu)",
     "url": "/eicar_com.zip",
     "http_content_type": "application/zip",
     "length": 0,
     "http_method": "GET",
     "protocol": "HTTP/1.1"
   },
   "payload": "SFRUUC8xLjAgMjAwIE9LDQpTZXJ2ZXI6IFNpbXBsZUhUVFAvMC42IFB5dGhvbi8zLjcuMw0KRGF0ZTogVGh1LCAyMCBBdWcgMjAyMCAwODoyNzo1MiBHTVQNCkNvbnRlbnQtdHlwZTogYXBwbGljYXRpb24vemlwDQpDb250ZW50LUxlbmd0aDogMTg0DQpMYXN0LU1vZGlmaWVkOiBUaHUsIDIwIEF1ZyAyMDIwIDA4OjI2OjMyIEdNVA0KDQo=",
   "tx_id": 0,
   "app_proto": "http",
   "packet_info": {
     "linktype": 1
   }
 },

The fields part of the sigflow logs

The field part defined by "fields" contains the same counters as in the source part: refer to the source part section

2.1.3.6.2. Events of type "alert"

2.1.3.6.2.1. Example of "alert" Sigflow events in the webui

The counters are detailed in Structure of sigflow logs.

2.1.3.6.2.2. Log data of type "alert"

The list of protocols that can generate alerts is indicated by the parameter Default settings for existing profiles available (app_proto field)`.

If a protocol changes along the way (for example if SMTP is upgraded to TLS via STARTTLS) or if the protocols used are not the same in both directions of the flow, the following fields may appear:

app_proto_tc (to client)
app_proto_ts (to server)
app_proto_orig

Note

The protocol actually recognized by Sigflow is defined in the GCap profile in the `Alerting and logging` tab in the `Alerting` column.

2.1.3.6.2.3. Counters of the source part of Sigflow logs of alert type

Note

This summary table shows which counters are not protocol dependent.

Table part source of Sigflow logs of alert type
Field	Required	Description	Values or example
@timestamp	Yes	Timestamp of the processing of the alert by the GCenter (corresponds to the passage in logstash)	2023-10-09T08:51:50.124Z
@version	yes	version of document	1
app_proto	No	File source flow application protocol	http
community_Id	Yes	Unique id to correlate the rise between the different security equipment	1:r6LvcE7ltny4a6Y9xt1Vr...
dest_ip	Yes	Destination IP address	"X.X.X.X"
dest_mac	Yes	Destination MAC address	"7e:ef:8d:eb:1a:81"
dest_port	No	Destination port. Present only when proto is udp or tcp	48740
event_type	Yes	Type of event: alert	Default alert
flow_id	Yes	Flow identifier	1666858537573672
flow.bytes_toclient	Yes	Size of flow to customer	647
flow.bytes_toserver	Yes	Size of flow to server	421
flow.pkts_toclient	Yes	Number of packets to client	4
flow.pkts_toserver	Yes	Number of packets to server	4
flow.reason	No	Mechanism that caused the flow between (“timeout”, “forced”, “shutdown” to stop processing)
flow.start	Yes	Date and time of first package seen by suricata	2023-10-09T08:51:17.974120+0000
gcap	Yes	Name of the gcap associated with the alert	gcap-xxxxxxxxx.domain.local
gcenter	Yes	GCenter name associated with alert	gcenter-xxx.domain.local
host	Yes	Name of the gcap associated with the alert	gcap-xxxxxxxxx.domain.local
in_iface	No	Capture interface on gcap	monvirt
packet	Yes	packet that triggered the alert registered in base64 (only for UDP)	ivOZFqBXfu+N6xq...
packet_info.linktype	Yes	Type of link-layer header	1
payload	No	Payload of the base64 package Present only if the payload option of the gcap "variable bases" menu is enabled	"SFRUUC8xLjAgMjAwIE9...
payload_printable	No	Payload of the package in a readable format. Present only if the printable payload option of the gcap «variable bases» menu is activated.	"HTTP/1.0 200 OK r nServer: SimpleHTTP/0.6 Python/3.7.3 r ..."
proto	Yes	Layer 4 protocol used	TCP
severity	Yes	Level of alert severity	3
src_ip	Yes	Source IP address	"X.X.X.X"
src_mac	Yes	Source MAC address	"8a:f3:99:16:a0:57"
src_port	No	Source port. Present only when proto is udp or tcp	80
stream	Yes		1
Timestamp of the processing of the alert by the GCenter (corresponds to the passage in logstash)	Yes	Date and time of alert analysis by logstash	2023-10-09T08:51:50.124Z
timestamp detected	Yes	Date and time of alert generation by suricata	2023-10-09T08:51:17.976Z
tx_id	yes	transaction identification (query/response pair)	0
type	Yes	Event type: default suricata	suricata
uuid	Yes	Unique identifier of the alert	1b6e03fa-d325-4f3d...

Summary table of counters in category "Alert"
Field	Required	Description	Values or example
alert.action	yes	Allowed if alert or pass is used and blocked if drop or reject is used.	alert, drop, reject, pass "action": "allowed",
alert.category	Yes	Description of alert classification	classtype. example Misc activity
alert.gid	Yes	Identifier of an alert group	gid
alert.metadata	No	Alert metadata. Field specification is free.	metadata: key value
alert.rev	Yes	Alert Revision Number	2
alert.severity	Yes	Level of alert severity	3
alert.signature	Yes	Description of the alert	AND INFO Python SimpleHTTP ServerBanner
alert.signature_id	Yes	Alert ID. Must be unique.	sid. example 2034636

List of metadata used in ETPRO and CTI source alerts (alert.metadata object in ES):

alert.metadata.affected_product
alert.metadata.attack_target
alert.metadata.created_at
alert.metadata.deployment
alert.metadata.former_category
alert.metadata.impact_flag
alert.metadata.malware_family

alert.metadata.performance_impact
alert.metadata.ruleset
alert.metadata.service
alert.metadata.signature_severity
alert.metadata.tag
alert.metadata.updated_at

Here is an example of an alert that uses metadata affected_product, attack_target, created_at, deployment, signature_severity, tag et updated_at:

alert tcp $EXTERNAL_NET any -> $SQL_SERVERS 1433 (
msg:"ET EXPLOIT MS-SQL SQL Injection closing string plus line comment";
flow: to_server,established;
content:"'|00|";
content:"-|00|-|00|";
reference:url,doc.emergingthreats.net/bin/view/Main/2000488;
classtype:attempted-user;
sid:2000488;
rev:7;
metadata:affected_product Web_Server_Applications, attack_target Web_Server, created_at 2010_07_30, deployment Datacenter, signature_severity Major, tag SQL_Injection, updated_at 2016_07_01;
)

2.1.3.6.3. Events of type "fileinfo"

Note

This summary table shows which counters are not protocol dependent.

Field	Required	Description
app_proto	Yes	Application protocol of the source stream of the file. example http
dest_ip	Yes	Destination IP address
dest_port	Yes	Destination port. Present only when proto is udp or tcp
event_type	Yes	Type of event
fileinfo.file_id	No	File ID
fileinfo.filename	Yes	File name
fileinfo.gaps	Yes
fileinfo.magic	No	File type identifier
fileinfo.md5	No	MD5 hash of file
fileinfo.sha1	No	SHA1 hash from file
fileinfo.sha256	No	SHA256 hash from file
fileinfo.size	Yes	File size
fileinfo.state	Yes	Completeness of the analyzed file (CLOSED) otherwise TRUNCATED. The file-store.stream-depth variable of suricata defines the size of the reconstructed files. The file is TRUNCATED if its size is > File-store stream depth (10 MB) by default.
fileinfo.stored	Yes	True if the file is reconstructed and False otherwise.
fileinfo.tx_id	Yes	transaction identification (query/response pair)
flow_id	Yes	Flow identifier
gcap	Yes	Name of the gcap associated with the alert
gcenter	Yes	GCenter name associated with alert
host	Yes	Name of the gcap associated with the alert
in_iface	No	Capture interface on gcap
proto	Yes	Layer 4 protocol used
src_ip	Yes	Destination IP address
src_port	No	Destination port. Present only when proto is udp or tcp
timestamp analyzed	Yes	Date and time of alert analysis by logstash
timestamp detected	Yes	Date and time of alert generation by suricata
type	Yes	Event type. Default on
uuid	Yes	Unique identifier of the alert
vlan	No	Identifier of the flow vlan

2.1.3.6.4. Events of type "meta-data"

2.1.3.6.4.1. List of fields present in all alerts with event_type!= ["alert", "fileinfo", "stats"]

@timestamp

@version

dest_ip

event_type

flow_id

gcap

GCenter

host

proto

src_ip

timestamp_analyzed

timestamp_detected

type

uuid

2.1.3.6.4.2. List of protocols compatible with logging (event_type field)

dhcp:

dhcp.assigned_ip
dhcp.client_ip
dhcp.client_mac
dhcp.dhcp_type
dhcp.dns_servers
dhcp.hostname

dhcp.id
dhcp.lease_time
dhcp.next_server_ip
dhcp.params
dhcp.rebinding_time
dhcp.relay_ip

dhcp.renewal_time
dhcp.requested_ip
dhcp.routers
dhcp.subnet_mask
dhcp.type

dnp3
dns:

body.proba_dga
body.severity
dga_probability
dns.aa
dns.answers.rdata
dns.answers.rrname
dns.answers.rrtype
dns.answers.ttl
dns.authorities.rrname
dns.authorities.rrtype

dns.authorities.ttl
dns.flags
dns.grouped.A
dns.grouped.AAAA
dns.grouped.CNAME
dns.id
dns.qr
dns.ra
dns.rcode

dns.rd
dns.rrname
dns.rrtype
dns.tx_id
dns.type
dns.version
headers.content-length
headers.content-type
tags

ftp
http:

http.accept
http.accept-charset
http.accept-datetime
http.accept_encoding
http.accept_language
http.accept-range
http.age
http.allow
http.authorization
http.cache_control
http.connection
http.content_encoding
http.content-language
http.content-length
http.content-location
http.content-md5
http.content-range
http.content_type
http.content-type
http.cookie

http.date
http.dnt
http.etags
http.from
http.hostname
http.http_content_type
http.http_method
http.http_port
http.http_refer
http.http_user_agent
http.last-modified
http.length
http.link
http.location
http.max-forwards
http.origin
http.pragma
http.proxy-authenticate
http.proxy-authorization
http.range

http.redirect
http.referrer
http.refresh
http.retry-after
http.server
http.set-cookie
http.status
http.te
http.trailer
http.transfer-encoding
http.upgrade
http.url
http.vary
http.via
http.warning
http.www-authenticate
http.x-authenticated-user
http.x-flash-version
http.x-forwarded-proto
http.x-requested-with

ikev2:

ikev2.alg_auth

ikev2.alg_dh

ikev2.alg_enc

ikev2.alg_esn

ikev2.alg_prf

ikev2.errors

ikev2.exchange_type

ikev2.init_spi

ikev2.message_id

ikev2.notify

ikev2.payload

ikev2.resp_spi

ikev2.role

ikev2.version_major

ikev2.version_minor

krb5:

krb5.cname

krb5.encryption

krb5.error_code

krb5.failed_request

krb5.msg_type

krb5.realm

krb5.sname

krb5.weak_encryption

netflow:

icmp_code

icmp_type

metadata.flowbits

netflow.age

netflow.bytes

netflow.end

netflow.max_ttl

netflow.min_ttl

netflow.pkts

netflow.start

parent_id

tcp.ack

tcp.cwr

tcp.ecn

tcp.fin

tcp.psh

tcp.rst

tcp.syn

tcp.tcp_flags

nfs:

nfs.file_tx

nfs.filename

nfs.hhashl

rpc.auth_type

rpc.creds.gid

rpc.creds.machine_name

rpc.creds.uid

rpc.status

rpc.xid

smb:

smb.access

smb.accessed

smb.changed

smb.client_dialects

smb.client_guid

smb.command

smb.created

smb.dcerpc.call_id

smb.dcerpc.interfaces.ack_reason

smb.dcerpc.interfaces.ack_result

smb.dcerpc.interfaces.uuid

smb.dcerpc.interfaces.version

smb.dcerpc.opnum

smb.dcerpc.req.frag_cnt

smb.dcerpc.req.stub_data_size

smb.dcerpc.request

smb.dcerpc.res.frag_cnt

smb.dcerpc.res.stub_data_size

smb.dcerpc.response

smb.dialect

smb.directory

smb.disposition

smb.filename

smb.fuid

smb.function

smb.id

smb.modified

smb.named_pipe

smb.ntlmssp.domain

smb.ntlmssp.host

smb.ntlmssp.user

smb.request.native_lm

smb.request.native_os

smb.response.native_lm

smb.response.native_os

smb.server_guid

smb.service.request

smb.service.response

smb.session_id

smb.share

smb.share_type

smb.size

smb.status

smb.status_code

smb.tree_id

smtp:

email.attachment

email.body_md5

email.from

email.status

email.subject

email.subject_md5

email.to

smtp.helo

smtp.mail_from

smtp.rcpt_to

ssh:
- ssh.client.proto_version
- ssh.client.software_version
- ssh.server.proto_version
- ssh.server.software_version
tftp:
- tftp.file
- tftp.mode
- tftp.packet
tls:
- tls.chain
- tls.fingerprint
- tls.issuerdn
- tls.notafter
- tls.notbefore
- tls.sni
- tls.subject
- tls.version

2.1.3.6.4.3. Metadata counters

Summary table of fields that do not depend on the protocols:

Field

Required

Description

app_proto

No

Application protocol of the flow from which the file originates

dest_ip

Yes

Destination's IP address

dest_port

No

Destination port. Only present when the value of proto is udp or tcp

event_type

Yes

Type of event. Alert by default.

flow_id

Yes

Flow identifier

gcap

Yes

Name of the gcap assigned to the alert

gcenter

Yes

Name of the GCenter assigned to the alert

host

Yes

Name of the gcap assigned to the alert

in_iface

No

Capture interface on the gcap

proto

Yes

Layer 4 protocol used

src_ip

Yes

Destination IP address

src_port

No

Destination port. Only present when the value of proto is udp or tcp

timestamp_analyzed

Yes

Date and time of the alert analysis by logstash

timestamp_detected

Yes

Date and time suricata generated the alert

type

Yes

Type of event. Suricata by default

uuid

Yes

Unique alert identifier

vlan

No

Vlan identifier of the flow

2.1.3.7. View the status of Sigflow

The current motor status is displayed in the Web UI `Health checks` screen.

2.1.3.8. Sigflow update

There are updates (Updates) for the Sigflow engine.
These updates can be done manually or scheduled via GUM.
See Presentation of GUM: dedicated module for managing updates and in particular the part Update signatures and/or engines (update).

2.1.3.9. Sigflow Setup

The engine is configured by profiles.
For more information, see paragraph GCAP Profiles.
For implementation, see the Configuring GCaps.