Processing Pipelines

This documentation page describes the concepts and classes of pySigma that can be used for transformation of Sigma rules.

Sigma rules are tranformed to take care of differences between the Sigma rule and the target data model. Examples are differences in field naming schemes or value representation.

Resolvers

Pipeline resolvers resolve identifiers and file names into a consolidated processing pipeline and take care of the appropriate ordering via the priority property that should be contained in a processing pipeline.

A processing pipeline resolver is a sigma.processing.resolver.ProcessingPipelineResolver object. It is initialized with an mapping between identifiers and sigma.processing.pipeline.ProcessingPipeline objects or callables that return such objects.

The method sigma.processing.resolver.ProcessingPipelineResolver.resolve_pipeline() returns a ProcessingPipeline object corresponsing with the given identifier or contained in the specified YAML file. sigma.processing.resolver.ProcessingPipelineResolver.resolve() returns a consolidated pipeline with the appropriate ordering as specified by the priority property of the specified pipelines.

Processing Pipeline

Classes

Specifying Processing Pipelines as YAML

A processing pipeline can be specified as YAML file that can be loaded with ProcessingPipeline.from_yaml(yaml) or by specifying a filename to ProcessingPipelineResolver.resolve() or ProcessingPipelineResolver.resolve_pipeline().

The following items are expected on the root level of the YAML file:

  • name: the name of the pipeline.

  • priority: specifies the ordering of the pipeline in case multiple pipelines are concatenated. Lower priorities are used first.

  • transformations: contains a list of transformation items.

Common used priorities are:

  • 10: log source pipelines like for Sysmon.

Pipelines with the same priority are applied in the order they were provided. Pipelines without a priority are assumed to have the priority 0.

Transformation items are defined as a map as follows:

  • id: the identifier of the item. This is also tracked at detection item or condition level and can be used in future conditions.

  • type: the type of the transformation as specified in the identifier to class mappings below: Transformations

  • Arbitrary transformation parameters are specified at the samle level.

  • rule_conditions or detection_item_conditions: conditions of the type corresponding to the name.

Conditions are specified as follows:

  • type: defines the condition type. It must be one of the identifiers that are defined in Conditions

  • Arbitrary conditions parameters are specified on the same level.

Example:

name: Custom Sysmon field naming
priority: 100
transformations:
- id: field_mapping
    type: field_name_mapping
    mapping:
        CommandLine: command_line
    rule_conditions:
    - type: logsource
        service: sysmon

Conditions

There are two types of conditions: rule conditions which are evaluated to the whole rule and detection item conditions that are evaluated for each detection item.

Rule Conditions

Detection Item Identifiers

Identifier

Class

logosurce

LogsourceCondition

Detection Item Conditions

Detection Item Identifiers

Identifier

Class

include_fields

IncludeFieldCondition

exclude_fields

ExcludeFieldCondition

match_string

MatchStringCondition

Base Classes

Base classes must be overridden to implement new conditions that can be used in processing pipelines. In addition, the new class should be mapped to an identifier. This allows to use the condition from processing pipelines defined in YAML files. The mapping is done in the dict rule_conditions or detection_item_conditions in the sigma.processing.conditions package for the respective condition types. This is not necessary for conditions that should be uses privately and not be distributed via the main pySigma distribution.

Transformations

Implemented Transformations

The following transformations with their corresponding identifiers for usage in YAML-based pipeline definitions are available:

Detection Item Identifiers

Identifier

Class

field_name_mapping

FieldMappingTransformation

field_name_suffix

AddFieldnameSuffixTransformation

field_name_prefix

AddFieldnamePrefixTransformation

wildcard_placeholders

WildcardPlaceholderTransformation

value_placeholders

ValueListPlaceholderTransformation

query_expression_placeholders

QueryExpressionPlaceholderTransformation

add_condition

AddConditionTransformation

change_logsource

ChangeLogsourceTransformation

replace_string

ReplaceStringTransformation

rule_failure

RuleFailureTransformation

detection_item_failure

DetectionItemFailureTransformation

Base Classes

There are four transformation base classes that can be derived to implement transformations on particular parts of a Sigma rule or the whole Sigma rule:

Transformation Tracking

tbd