2.1.6. CTI engine, RetroHunt engine and ActiveHunt engine

2.1.6.1. Presentation

The CTI engine consists of the parts:

CTI module
Configuring the CTI engine
RetroHunt engine
ActiveHunt engine

The configuration interface is described in the paragraph `Admin-GCenter- CTI Configuration` screen of the legacy web UI.

2.1.6.2. CTI module

The solution's CTI module uses LastInfoSec's compromise indices to generate alerts.

The CTI module enables:

Searching the metadata after the fact to see if the compromise indices in the CTI database correspond to malicious movements
Generating Suricata rules in the Sigflow module on the basis of compromise indices in order to raise alerts

Note

An additional license is required to activate this module. It is therefore not automatically activated in the solution.

2.1.6.3. Configuring the CTI engine

Ideally, the index database should be updated on a daily basis in order to obtain the latest indices added by the R&D teams.
The default retention time is seven days, which is also the maximum possible value.
The CTI engine must be enabled to generate alerts for Advanced Persistent Threat (APT) threats (see `Admin-GCenter- CTI Configuration` screen of the legacy web UI).

2.1.6.4. RetroHunt engine

This engine will enable searching among all the metadata present in the solution to determine whether or not they correspond to indications of compromise.
If this is the case, then an alert will be raised in the various alert display dashboards: NDR and Kibana.
The idea being that if a malicious file was not detected as such by Malcore during its analysis, because it was too recent for the Malcore antivirus database for example, then if one of the indices matches the hash of the file in question in the metadata, an alert will be raised.

Note

The correlation of indices and metadata will depend on the data retention time configured on the GCenter.

Match analysis between indices and metadata is triggered when updating the index database.
It is therefore only possible to trigger the match analysis manually by updating the compromise indices manually.
There are 3 different ways to update the indices of compromise:

Manual update with cti.gwp package
Online update: the recovery of compromise indices is performed every hour based on the package posted by Gatewatcher
Local update: the recovery of compromise indices is performed every hour based on the package in the local repository.

Note

In order to optimize the implementation of the update of the indices of compromises in Local` mode, it is necessary that the local repository retrieves the package cti.gwp every hour.

Otherwise the update as well as the match search will only be performed according to the recovery frequency of the package on the local deposit.

2.1.6.5. ActiveHunt engine

This engine will enable generating a source of Suricata rules available to the Sigflow module based on the compromise indices.
This source can then be added to the security policy (ruleset) assigned to the GCap in order to raise alerts on the analysed traffic.
This engine is positioned in real time, unlike the Retrohunt engine above, which scans for matches in the past.
The rules generated are updated at the same time as the database of compromise indicators, every day if possible.

Note

Unlike the RetroHunt engine, the alerts generated are of the Suricata type. They will therefore be available in the Sigflow dashboards.

2.1.6.6. Events generated by the RetroHunt engine

Events generated by the RetroHunt engine are alerts.

These are displayed:

In the main interface named WEB UI of the GCenter in the `Alerts` screen (the main interface named WEB UI is described in:doc:../../05_GUI_presentation/00_00_interface_presentation).

To view the alerts, select the filter `APT` and thus view the list of alerts: see the presentation of the Web UI `Alerts` screen.

By clicking on an alert, the detailed information of this alert is displayed: see Example of RetroHunt alert in the webui.
In the Kibana UI interface

To view the alerts, select the filter `C&C` and thus view the list of alerts: see the presentation of the Web UI `Alerts` screen.

By clicking on an alert, select on the command `Alert details` then select the arrow to the left of the alert.

The interface displayed is the interface named Kibana UI (described in Overview of the Kibana GUI).

The detailed information of this alert can be viewed in table or jason format (see Example of Machine Learning log).

2.1.6.6.1. Example of RetroHunt alert in the webui

The counters are detailed in RetroHunt log data structure.

2.1.6.7. Example of a RetroHunt event

 {
 "_index": "retrohunt-2023.10.18-000171",
 "_type": "_doc",
 "_id": "6BESQ4sBeBoubSygpp1s",
 "_version": 1,
 "_score": 1,
 "_source": {
   "flow_id": 1540796205479447,
   "@timestamp": "2023-10-18T13:56:14.789Z",
   "kill_chain_phases": [],
   "gcenter": "gcenter-xxx.domain.local"
   "signature": "RetroHunt - Host - malware/Unknown - Hajime - GW Lab Test - 00135350-1810-2023-34db-1319151da1fd",
   "src_ip": "X.X.X.X",
   "event_type": "retrohunt",
   "case_id": "00135350-1810-2023-edb7-7f8f1e4fccb9",
   "ioc_tags": [
     "trojan.generickd.34055387 (b)",
     "linux/hajime.a trojan",
     "e32/agent.cd",
     "linux.hajime.bc",
     "backdoor.hajime.linux.129",
     "linux/hajime.75930",
     "unix.malware.agent-6626471-0",
     "linux/hajime.nsnlw",
     "hajime",
     "elf.mirai.43048.gc",
     "trojan.elfarm32.hajime.fbhtfi",
     "trojan.linux.hajime",
     "trojan.generickd.34055387"
   ],
   "families": [
     "Hajime"
   ],
   "targeted_platforms": [
     "linux"
   ],
   "risk": "Suspicious",
   "categories": [
     "malware"
   ],
   "campaigns": [],
   "@version": "1",
   "threat_actor": [
     "GW Lab Test"
   ],
   "timestamp_detected": "2023-10-18T08:08:31.112Z",
   "ioc_value": "im.a.very.bad.doma.in",
   "external_links": [
     {
       "source_name": "URLHaus Abuse.ch",
       "url": "https://urlhaus.abuse.ch/url/2269068/"
     }
   ],
   "gcap": "gcap-xxxxxxxxx.domain.local",
   "uuid": "19fe0b3d-05fb-433a-ada0-f246e284d9bd",
   "dest_port": 80,
   "ioc_id": "00135350-1810-2023-34db-1319151da1fd",
   "ttp": [],
   "targeted_sectors": [],
   "meta_data": {
     "cwe": [],
     "ssdeep": "1536:87vbq1lGAXSEYQjbChaAU2yU23M51DjZgSQAvcYkFtZTjzBht5:8D+CAXFYQChaAUk5ljnQssL",
     "descriptions": [],
     "usageMode": "hunting",
     "filetype": "ELF 32-bit LSB executable, ARM, EABI5 version 1 (GNU/Linux)",
     "size": 78.3984375,
     "tslh": "T16D7312E017B517CC1371A8353BED205E9128223972AE35302E97528DF957703BAB2DBE"
   },
   "type": "cti",
   "ioc_creation_date": "2023-10-18T13:53:50+00:00",
   "timestamp_analyzed": "2023-10-18T13:56:14.789Z",
   "targeted_organizations": [],
   "matched_event_type": "http",
   "ioc_updated_date": "2023-10-18T13:53:50+00:00",
   "severity": 1,
   "matched_event": "cf7cf312-883b-4b84-a530-fea8d49b294c",
   "community_id": "1:oPgJrwIH53r44+0TfDB+7uhzL50=",
   "vulnerabilities": [],
   "targeted_countries": [],
   "timestamp_package": "2023-10-18T13:53:50.696659+0000",
   "description": "IOC matching first tests",
   "relations": [
     "0e3cc27b-7999-48ce-8484-dc12b325a355"
   ],
   "": 0.5,
   "dest_ip": "X.X.X.X",
   "src_port": 59338,
   "tlp": "green",
   "usage_mode": "hunting",
   "ioc_type": "Host"
 },
 "fields": {
   "signature": [
     "RetroHunt - Host - malware/Unknown - Hajime - GW Lab Test - 00135350-1810-2023-34db-1319151da1fd"
   ],
   "usage_mode": [
     "hunting"
   ],
   "description": [
     "IOC matching first tests"
   ],
   "type": [
     "cti"
   ],
   "uuid": [
     "19fe0b3d-05fb-433a-ada0-f246e284d9bd"
   ],
   "meta_data.ssdeep": [
     "1536:87vbq1lGAXSEYQjbChaAU2yU23M51DjZgSQAvcYkFtZTjzBht5:8D+CAXFYQChaAUk5ljnQssL"
   ],
   "src_ip": [
     "172.17.0.6"
   ],
   "ioc_updated_date": [
     "2023-10-18T13:53:50.000Z"
   ],
   "community_id": [
     "1:oPgJrwIH53r44+0TfDB+7uhzL50="
   ],
   "event_type": [
     "retrohunt"
   ],
   "ioc_tags": [
     "trojan.generickd.34055387 (b)",
     "linux/hajime.a trojan",
     "e32/agent.cd",
     "linux.hajime.bc",
     "backdoor.hajime.linux.129",
     "linux/hajime.75930",
     "unix.malware.agent-6626471-0",
     "linux/hajime.nsnlw",
     "hajime",
     "elf.mirai.43048.gc",
     "trojan.elfarm32.hajime.fbhtfi",
     "trojan.linux.hajime",
     "trojan.generickd.34055387"
   ],
   "flow_id": [
     1540796205479447
   ],
   "case_id": [
     "00135350-1810-2023-edb7-7f8f1e4fccb9"
   ],
   "@version": [
     "1"
   ],
   "external_links.url": [
     "https://urlhaus.abuse.ch/url/2269068/"
   ],
   "categories": [
     "malware"
   ],
   "meta_data.usageMode": [
     "hunting"
   ],
   "matched_event_type": [
     "http"
   ],
   "dest_port": [
     80
   ],
   "severity": [
     1
   ],
   "targeted_platforms": [
     "linux"
   ],
   "meta_data.filetype": [
     "ELF 32-bit LSB executable, ARM, EABI5 version 1 (GNU/Linux)"
   ],
   "": [
     0.5
   ],
   "meta_data.size": [
     78.39844
   ],
   "gcenter": [
     "gcenter-int-128-dag.gatewatcher.com"
   ],
   "meta_data.tslh": [
     "T16D7312E017B517CC1371A8353BED205E9128223972AE35302E97528DF957703BAB2DBE"
   ],
   "matched_event": [
     "cf7cf312-883b-4b84-a530-fea8d49b294c"
   ],
   "ioc_value": [
     "im.a.very.bad.doma.in"
   ],
   "ioc_id": [
     "00135350-1810-2023-34db-1319151da1fd"
   ],
   "ioc_type": [
     "Host"
   ],
   "families": [
     "Hajime"
   ],
   "timestamp_detected": [
     "2023-10-18T08:08:31.112Z"
   ],
   "external_links.source_name": [
     "URLHaus Abuse.ch"
   ],
   "src_port": [
     59338
   ],
   "threat_actor": [
     "GW Lab Test"
   ],
   "@timestamp": [
     "2023-10-18T13:56:14.789Z"
   ],
   "ioc_creation_date": [
     "2023-10-18T13:53:50.000Z"
   ],
   "dest_ip": [
     "172.17.0.4"
   ],
   "tlp": [
     "green"
   ],
   "risk": [
     "Suspicious"
   ],
   "gcap": [
     "gcap-int-129-dag.gatewatcher.com"
   ],
   "timestamp_analyzed": [
     "2023-10-18T13:56:14.789Z"
   ],
   "timestamp_package": [
     "2023-10-18T13:53:50.696Z"
   ],
   "relations": [
     "0e3cc27b-7999-48ce-8484-dc12b325a355"
   ],
   "description.keyword": [
     "IOC matching first tests"
   ]
 }
}

2.1.6.7.1. RetroHunt log data structure

The logs are composed of different parts:

The leading part
The source part defined by "_source";
The field portion defined by "_fields"

2.1.6.7.1.1. The header part of RetroHunt logs

The header section contains:

{
 "_index": "retrohunt-2023.10.18-000171",
 "_type": "_doc",
 "_id": "6BESQ4sBeBoubSygpp1s",
 "_version": 1,
 "_score": 1,

Table header part of Machine learning logs
Field	Required	Description	Values or example
_index	Yes	Internal index	retrohunt-2023.10.18-000171
_type	Yes	default type	_doc
_id	Yes	internal identifier	6BESQ4sBeBoubSygpp1s
_version	Yes	internal version	1
_score	Yes	relevance of the response to the request	1

2.1.6.7.1.2. The source part of the Machine learning logs

The source part defined by "_source" contains:

  "flow_id": 1540796205479447,
  "@timestamp": "2023-10-18T13:56:14.789Z",
  "kill_chain_phases": [],
  "gcenter": "gcenter-xxx.domain.local"
  "signature": "RetroHunt - Host - malware/Unknown - Hajime - GW Lab Test - 00135350-1810-2023-34db-1319151da1fd",
  "src_ip": "X.X.X.X",
  "event_type": "retrohunt",
  "case_id": "00135350-1810-2023-edb7-7f8f1e4fccb9",
  "ioc_tags": [
    "trojan.generickd.34055387 (b)",
    "linux/hajime.a trojan",
    "e32/agent.cd",
    "linux.hajime.bc",
    "backdoor.hajime.linux.129",
    "linux/hajime.75930",
    "unix.malware.agent-6626471-0",
    "linux/hajime.nsnlw",
    "hajime",
    "elf.mirai.43048.gc",
    "trojan.elfarm32.hajime.fbhtfi",
    "trojan.linux.hajime",
    "trojan.generickd.34055387"
  ],
  "families": [
    "Hajime"
  ],
  "targeted_platforms": [
    "linux"
  ],
  "risk": "Suspicious",
  "categories": [
    "malware"
  ],
  "campaigns": [],
  "@version": "1",
  "threat_actor": [
    "GW Lab Test"
  ],
  "timestamp_detected": "2023-10-18T08:08:31.112Z",
  "ioc_value": "im.a.very.bad.doma.in",
  "external_links": [
    {
      "source_name": "URLHaus Abuse.ch",
      "url": "https://urlhaus.abuse.ch/url/2269068/"
    }
  ],
  "gcap": "gcap-xxxxxxxxx.domain.local",
  "uuid": "19fe0b3d-05fb-433a-ada0-f246e284d9bd",
  "dest_port": 80,
  "ioc_id": "00135350-1810-2023-34db-1319151da1fd",
  "ttp": [],
  "targeted_sectors": [],
  "meta_data": {
    "cwe": [],
    "ssdeep": "1536:87vbq1lGAXSEYQjbChaAU2yU23M51DjZgSQAvcYkFtZTjzBht5:8D+CAXFYQChaAUk5ljnQssL",
    "descriptions": [],
    "usageMode": "hunting",
    "filetype": "ELF 32-bit LSB executable, ARM, EABI5 version 1 (GNU/Linux)",
    "size": 78.3984375,
    "tslh": "T16D7312E017B517CC1371A8353BED205E9128223972AE35302E97528DF957703BAB2DBE"
  },
  "type": "cti",
  "ioc_creation_date": "2023-10-18T13:53:50+00:00",
  "timestamp_analyzed": "2023-10-18T13:56:14.789Z",
  "targeted_organizations": [],
  "matched_event_type": "http",
  "ioc_updated_date": "2023-10-18T13:53:50+00:00",
  "severity": 1,
  "matched_event": "cf7cf312-883b-4b84-a530-fea8d49b294c",
  "community_id": "1:oPgJrwIH53r44+0TfDB+7uhzL50=",
  "vulnerabilities": [],
  "targeted_countries": [],
  "timestamp_package": "2023-10-18T13:53:50.696659+0000",
  "description": "IOC matching first tests",
  "relations": [
    "0e3cc27b-7999-48ce-8484-dc12b325a355"
  ],
  "": 0.5,
  "dest_ip": "X.X.X.X",
  "src_port": 59338,
  "tlp": "green",
  "usage_mode": "hunting",
  "ioc_type": "Host"
},

Table part source
Field	Required	Description	Values or example
@timestamp	Yes	Timestamp of the processing of the alert by the GCenter (corresponds to the passage in logstash)	"2023-10-18T13:56:14.789Z"
"@version"	yes	version of document	1
@"case_id"	yes	Internal identification number	"00135350-1810-..."
Alert type in webui	Yes	Threat Type	APT
"campaigns"	yes	Campaign name
"categories"	yes	threat category	malware
"community_id"	yes	Unique id to correlate the rise between the different security equipment	1:oPgJrwIH53r44+0TfDB+7uhzL50=
"description"	yes	Threat description field	IOC matching first test
"dest_ip"	Yes	Destination IP address	x.x.x.x
"dest_port"	No	Port of destination	80
"event_type"	Yes	type of event	retrohunt
"external_links"	No	See the summary table of the "external_links" category counters
"families"	yes	Threat family	Hajime
"flow_id"	Yes	Unique identifier of the flow. Allows to find the associated fileinfo	1540796205479447
"gcap"	Yes	Name of the gcap associated with the alert	gcap-xxx.domain.local
"gcenter"	yes	GCenter name associated with alert.	gcenter-xxx.domain.local
Hostname (webui)	yes	Host name of the threat originator	If the hostname is not present, its IP or domain name is displayed
"ioc_creation_date"	yes	Index of Compromission; creation date in the database	"2023-10-18T13:53:50+00:00"
"ioc_id"	yes	Indice of Compromission: identifier	"00135350-1810-2023-34db-1319151da1fd"
"ioc_tags"	Yes	Compromise index: label	"trojan.generickd.34055387 (b)" "linux/hajime. a trojan" "e32/agent.cd"
"ioc_type"	yes	Compromise index: type	"Host"
"ioc_updated_date"	yes	Compromission index: update date	"2023-10-18T13:53:50+00:00"
"ioc_value"	yes	Compromise index: value	"im.a.very.bad.doma.in"
"kill_chain_phases"	yes	Phases of the strike chain;
"matched_event"	yes	Corresponding event	cf7cf312-883b-4b84...
"matched_event_type"	yes	Type of event that matched	http
"meta_data"	yes	See Summary table of counters: category "meta_data"	NA
"probability"	yes	Probability	0.5
"relations"	yes	Relations	0e3cc27b-7999-...
"risk"	yes	Threat risk assessment outcome	Suspicious
"severity"	Yes	Analysis result code	Between 0 and 3 0=clean, 1=infected, 2=suspicious, 3=Other
"signature" (or Signature or Description in Webui)	yes	Title of the threat	"RetroHunt - Host - malware/Unknown - Hajime - GW Lab Test - 00135350-1810-2023-34db-1319151da1fd"
"src_ip"	Yes	Source IP address	X.X.X.X
"src_port"	Yes	Source port	59338
"targeted_countries"	yes	Target countries
"targeted_organizations"	yes	Targeted organisations;
"targeted_platforms"	yes	Target platforms	linux
"targeted_sectors"	yes	Targeted sectors of activity
"threat_actor"	yes	Actors of this threat
"timestamp analyzed"	Yes	Date and time of last file scan	2023-10-18T13:56:14.789Z
"timestamp detected"	Yes	Date and time of first file capture	2022-09-08T09:21:22.223Z
"timestamp_package"	Yes	Date and time of update of CTI sources	2023-10-18T13:53:50.696659+0000
"tlp"	yes	Traffic Light Protocol (4 colours depending on disclosure limitation)	green. this means "limited disclosure, beneficiaries can disseminate it within their community."
"ttp"	yes	Trusted Third Party
"type"	Yes	Type of event	"cti"
"use_mode"	yes	Mode of use	hunting
"uuid"	Yes	Unique identifier of the alert	19fe0b3d-05fb-433a...
"vulnerabilities"	yes	Vulnerabilities

Summary table of counters: category "external_links"
Field	Required	Description	Values or example
"source_name"	yes	Name of source	"URLHaus Abuse.ch"
"descriptions"	yes	Description
"url"	yes	URL	"https://urlhaus.abuse.ch/url/2269068/

Summary table of counters: category "meta_data"
Field	Required	Description	Values or example
"cwe"	yes	Common weakness enumeration
"Descriptions"	yes	description	yes
"usageMode"	yes	Use of this IOC	hunting
"threadype":	yes	File type	ELF 32-bit LSB executable, ARM, EABI5 version 1 (GNU/Linux)
"size"	yes	Pruning	78.3984375
"ssdeep"	yes	Hash calculated by ssdeep	1536:87vbq1lGAXSEYQjbChaAU..
"s"	yes		T16D7312E017B517CC1371A8...

2.1.6.7.1.3. The fields part of the RetroHunt logs

The field part defined by "fields" contains the same counters as in the source part: refer to the source part section.

2.1.6.8. Viewing the CTI Status

The current motor status is displayed in the Web UI `Health checks` screen.

2.1.6.9. CTI Update

There are updates (Updates) for the CTI engine.
These updates can be done manually or scheduled via GUM.
See section Presentation of GUM: dedicated module for managing updates and in particular part Update signatures and/or engines (update).