Backends¶

Backends are responsible for conversion of Sigma rules into a target query languages. Mainly, they have to convert the conditions of the Sigma rules with their reference to detection items into equivalent query. Backends should not be used to handle log source types or data models, e.g. field naming or differences in value representation. Use :Processing Pipelines instead.

To implement a conversion for a new query language derive an appropriate backend base class from below and override properties or methods as required.

Use the Cookiecutter template to start a new backend.

Conventions¶

Always implement the default output format in a way that the user does get some directly actionable output if she/he doesn’t explicitely chooses a format.
Don’t do any concatenation of simple queries in the basic default format, the CLI or other tools will take care of this.
Use backend specific options to control behavior of the backend. Backend options are provided as additional keyword arguments to the __init__ constructor of a backend. They can be passed to the CLI with the -O` option. Numbers are automatically converted to int type. If a backend option is specified multiple times in the CLI, all values are passed as arrays.
Don’t print any output to the console or create files from the backend. Return text output as string or file output as bytes. The tools using your backend will take care of the proper handling of the result.

Concepts¶

Conversion Methods¶

Builtin Processing Pipeline¶

Output Formats¶

Rule Finalization¶

Output Finalization¶

Classes¶

Backend¶

The backend base class is generic and can generate arbitrary output, e.g. Python data structures.

class sigma.conversion.base.Backend(processing_pipeline: ProcessingPipeline | None = None, collect_errors: bool = False)¶

Base class for Sigma conversion backends. A backend is made up from the following elements:

Some metadata about the properties of the backend.
A processing pipeline stored in backend_processing_pipeline that is applied to each Sigma rule that is converted by the backend. This is the location where you add generic transformations that should be applied to all Sigma rules before conversion.
An additional processing pipeline can be passed to the constructor and is applied after the backend pipeline. This one is configured by the user to implement transformations required in the environment (e.g. field renaming).
If collect_errors is set to True, exceptions will not be thrown, but collected in (sigma_rule, exception) tuples in the errors property.
The method convert is the entry point for a conversion of a rule set. By default it converts each rule and invokes the finalization step for the whole set of converted rules. There are better locations to implement backend functionality.
convert_rule converts a single rule. By default it converts all conditions and invokes the rule finalization.
convert_condition is the entry point for conversion of a rule condition into a query. It dispatches to the condition element classes.
convert_condition_* methods must be implemented and handle the conversion of condition elements. The result might be an intermediate representation which is finalized by finalize_query.
finalize_query finalizes the conversion result of a converted rule condition. By default it simply passes the generated queries.
finalize_output_<format> finalizes the conversion result of a whole rule set in the specified format. By default finalize_output_default is called and outputs a list of all queries. Further formats can be implemented in similar methods. The defaulf format can be specified in the class variable default_format.

Implementation of a backend:

Implement conversion of condition elements in convert_condition_*. The output can be an intermediate or the final query representation.
If required, implement a per-query finalization step in finalize_query. Each Sigma rule condition results in a query. This can embed the generated query into other structures (e.g. boilerplate code, prefix/postifx query parts) or convert the intermediate into a final query representation.
If required, implement a finalization step working on all generated queries in finalize. This can embed the queries into other data structures (e.g. JSON or XML containers for import into the target system) or perform the conversion of an intermediate to the final query representation.

Some hints and conventions:

Use processing pipelines to apply transformations instead of implementing transformations in the backend itself. Implement generic transformations if they aren’t too backend-specific.
Use TextQueryBackend as base class for backends that output text-based queries.
Use intermediate representations for queries and query sets for formats that require state information, e.g. if the target query language results in a different structure than given by the condition.

convert(rule_collection: SigmaCollection, output_format: str | None = None, correlation_method: str | None = None) → Any¶: Convert a Sigma ruleset into the target data structure. Usually the result are one or multiple queries, but might also be some arbitrary data structure required for further processing.

convert_condition(cond: ConditionOR | ConditionAND | ConditionNOT | ConditionFieldEqualsValueExpression | ConditionValueExpression, state: ConversionState) → Any¶

Convert query of Sigma rule into target data structure (usually query, see above). Dispatches to methods (see above) specialized on specific condition parse tree node objects.

The state mainly contains the deferred list, which is used to collect query parts that are not directly integrated into the generated query, but added at a postponed stage of the conversion process after the conversion of the condition to a query is finished. This is done in the finalize_query method and must be implemented individually.

abstract convert_condition_and(cond: ConditionAND, state: ConversionState) → Any¶: Conversion of AND conditions.

abstract convert_condition_as_in_expression(cond: ConditionOR | ConditionAND, state: ConversionState) → Any¶: Conversion of OR or AND conditions into “field in (value list)” expressions.

abstract convert_condition_field_compare_op_val(cond: ConditionFieldEqualsValueExpression, state: ConversionState) → Any¶: Conversion of field matches regular expression value expressions

convert_condition_field_eq_expansion(cond: ConditionFieldEqualsValueExpression, state: ConversionState) → Any¶: Convert each value of the expansion with the field from the containing condition and OR-link all converted subconditions.

abstract convert_condition_field_eq_field(cond: SigmaFieldReference, state: ConversionState) → Any¶: Conversion of field equals another field expressions.

abstract convert_condition_field_eq_query_expr(cond: ConditionFieldEqualsValueExpression, state: ConversionState) → Any¶: Conversion of query expressions bound to a field.

convert_condition_field_eq_val(cond: ConditionFieldEqualsValueExpression, state: ConversionState) → Any¶: Conversion dispatcher of field = value conditions. Dispatches to value-specific methods.

abstract convert_condition_field_eq_val_bool(cond: ConditionFieldEqualsValueExpression, state: ConversionState) → Any¶: Conversion of field = boolean value expressions

abstract convert_condition_field_eq_val_cidr(cond: ConditionFieldEqualsValueExpression, state: ConversionState) → Any¶: Conversion of field matches CIDR expression value expressions

convert_condition_field_eq_val_exists(cond: ConditionFieldEqualsValueExpression, state: ConversionState) → Any¶: Dispatch conversion of field exists expressions to appropriate method.

abstract convert_condition_field_eq_val_null(cond: ConditionFieldEqualsValueExpression, state: ConversionState) → Any¶: Conversion of field is null expression value expressions

abstract convert_condition_field_eq_val_num(cond: ConditionFieldEqualsValueExpression, state: ConversionState) → Any¶: Conversion of field = number value expressions

abstract convert_condition_field_eq_val_re(cond: ConditionFieldEqualsValueExpression, state: ConversionState) → Any¶: Conversion of field matches regular expression value expressions

abstract convert_condition_field_eq_val_str(cond: ConditionFieldEqualsValueExpression, state: ConversionState) → Any¶: Conversion of field = string value expressions

abstract convert_condition_field_eq_val_str_case_sensitive(cond: ConditionFieldEqualsValueExpression, state: ConversionState) → Any¶: Conversion of field = cased string value expressions

abstract convert_condition_field_exists(cond: ConditionFieldEqualsValueExpression, state: ConversionState) → Any¶: Conversion of field exists expressions

abstract convert_condition_field_not_exists(cond: ConditionFieldEqualsValueExpression, state: ConversionState) → Any¶: Conversion of field not exists expressions

abstract convert_condition_not(cond: ConditionNOT, state: ConversionState) → Any¶: Conversion of NOT conditions.

abstract convert_condition_or(cond: ConditionOR, state: ConversionState) → Any¶: Conversion of OR conditions.

abstract convert_condition_query_expr(cond: ConditionValueExpression, state: ConversionState) → Any¶: Conversion of query expressions without field association.

convert_condition_val(cond: ConditionValueExpression, state: ConversionState) → Any¶: Conversion of value-only conditions.

abstract convert_condition_val_num(cond: ConditionValueExpression, state: ConversionState) → Any¶: Conversion of number-only conditions.

abstract convert_condition_val_re(cond: ConditionValueExpression, state: ConversionState) → Any¶: Conversion of regexp-only conditions.

abstract convert_condition_val_str(cond: ConditionValueExpression, state: ConversionState) → Any¶: Conversion of string-only conditions.

abstract convert_correlation_event_count_rule(rule: SigmaCorrelationRule, output_format: str | None = None, method: str | None = None) → List[Any]¶

Convert an event count correlation rule into the target data structure (usually query).

Args:: rule (SigmaCorrelationRule): The event count correlation rule to be converted. output_format (Optional[str]): The output format for the conversion. Defaults to None. method (Optional[str]): The correlation method to be used. Defaults to None.
Returns:: Any: The converted data structure.

convert_correlation_rule(rule: SigmaCorrelationRule, output_format: str | None = None, method: str | None = None) → List[Any]¶

Convert a correlation rule into the target data structure (usually query).

Args:: rule (SigmaCorrelationRule): The correlation rule to be converted. output_format (Optional[str]): The desired output format. Defaults to None. method (Optional[str]): The correlation method to be used. Defaults to None.
Returns:: Any: The converted data structure.
Raises:: NotImplementedError: If the conversion for the given correlation rule type is not implemented.

abstract convert_correlation_temporal_ordered_rule(rule: SigmaCorrelationRule, output_format: str | None = None, method: str | None = None) → List[Any]¶

Convert an ordered temporal correlation rule into the target data structure (usually query).

Args:: rule (SigmaCorrelationRule): The ordered temporal correlation rule to be converted. output_format (Optional[str]): The output format for the conversion. Defaults to None. method (Optional[str]): The correlation method to be used. Defaults to None.
Returns:: Any: The converted data structure.

abstract convert_correlation_temporal_rule(rule: SigmaCorrelationRule, output_format: str | None = None, method: str | None = None) → List[Any]¶

Convert a temporal correlation rule into the target data structure (usually query).

Args:: rule (SigmaCorrelationRule): The temporal correlation rule to be converted. output_format (Optional[str]): The output format for the conversion. Defaults to None. method (Optional[str]): The correlation method to be used. Defaults to None.
Returns:: Any: The converted data structure.

abstract convert_correlation_value_count_rule(rule: SigmaCorrelationRule, output_format: str | None = None, method: str | None = None) → List[Any]¶

Convert a value count correlation rule into the target data structure (usually query).

Args:: rule (SigmaCorrelationRule): The value count correlation rule to be converted. output_format (Optional[str]): The output format for the conversion. Defaults to None. method (Optional[str]): The correlation method to be used. Defaults to None.
Returns:: Any: The converted data structure.

convert_rule(rule: SigmaRule, output_format: str | None = None) → List[Any]¶: Convert a single Sigma rule into the target data structure (usually query, see above).

decide_convert_condition_as_in_expression(cond: ConditionOR | ConditionAND, state: ConversionState) → bool¶

Decide if an OR or AND expression should be converted as “field in (value list)” or as plain expression.

Parameters:

cond (Union[ConditionOR, ConditionAND]) – Condition that is converted for which the decision has to be made.
state (ConversionState) – Current conversion state.

Returns:

True if in-expression should be generated, else False

Return type:

bool

finalize(queries: List[Any], output_format: str)¶: Finalize output. Dispatches to format-specific method.

finalize_output_default(queries: List[Any]) → Any¶

Default finalization.

This is the place where syntactic elements of the target format for the whole output are added, e.g. putting individual queries into a XML file.

finalize_query(rule: SigmaRule, query: Any, index: int, state: ConversionState, output_format: str)¶

Finalize query. Dispatches to format-specific method. The index parameter enumerates generated queries if the conversion of a Sigma rule results in multiple queries.

This is the place where syntactic elements of the target format for the specific query are added, e.g. adding query metadata.

finalize_query_default(rule: SigmaRule, query: Any, index: int, state: ConversionState) → Any¶: Finalize conversion result of a query. Handling of deferred query parts must be implemented by overriding this method.

TextQueryBackend¶

Backend base class for conversion to text based query languages. In many cases the methods doesn’t have to be overridden but string tokens have to be defined as class variable members (tbd).

class sigma.conversion.base.TextQueryBackend(*args, **kwargs)¶

Backend base for backends generating text-based queries. The behavior can be defined by various class variables. If this is not sufficient, the respective methods can be implemented with more complex transformations.

compare_precedence(outer: ConditionItem, inner: ConditionItem) → bool¶: Compare precedence of outer and inner condition items. Return True if precedence of enclosing condition item (outer) is lower than the contained (inner) condition item. In this case, no additional grouping is required.

convert_condition_and(cond: ConditionAND, state: ConversionState) → str | DeferredQueryExpression¶: Conversion of AND conditions.

convert_condition_as_in_expression(cond: ConditionOR | ConditionAND, state: ConversionState) → str | DeferredQueryExpression¶: Conversion of field in value list conditions.

convert_condition_field_compare_op_val(cond: ConditionFieldEqualsValueExpression, state: ConversionState) → str | DeferredQueryExpression¶: Conversion of numeric comparison operations into queries.

convert_condition_field_eq_field(cond: SigmaFieldReference, state: ConversionState) → str | DeferredQueryExpression¶: Conversion of comparision of two fields.

convert_condition_field_eq_field_escape_and_quote(field1: str, field2: str) → Tuple[str, str]¶: Escape and quote field names of a field-quals-field expression.

convert_condition_field_eq_query_expr(cond: ConditionFieldEqualsValueExpression, state: ConversionState) → str | DeferredQueryExpression¶: Conversion of field is null expression value expressions

convert_condition_field_eq_val_bool(cond: ConditionFieldEqualsValueExpression, state: ConversionState) → str | DeferredQueryExpression¶: Conversion of field = bool value expressions

convert_condition_field_eq_val_cidr(cond: ConditionFieldEqualsValueExpression, state: ConversionState) → str | DeferredQueryExpression¶: Conversion of field matches CIDR value expressions.

convert_condition_field_eq_val_null(cond: ConditionFieldEqualsValueExpression, state: ConversionState) → str | DeferredQueryExpression¶: Conversion of field is null expression value expressions

convert_condition_field_eq_val_num(cond: ConditionFieldEqualsValueExpression, state: ConversionState) → str | DeferredQueryExpression¶: Conversion of field = number value expressions

convert_condition_field_eq_val_re(cond: ConditionFieldEqualsValueExpression, state: ConversionState) → str | DeferredQueryExpression¶: Conversion of field matches regular expression value expressions.

convert_condition_field_eq_val_re_contains(cond: ConditionFieldEqualsValueExpression, state: ConversionState) → str | DeferredQueryExpression¶: Conversion of value-only regular expressions.

convert_condition_field_eq_val_str(cond: ConditionFieldEqualsValueExpression, state: ConversionState) → str | DeferredQueryExpression¶: Conversion of field = string value expressions

convert_condition_field_eq_val_str_case_sensitive(cond: ConditionFieldEqualsValueExpression, state: ConversionState) → str | DeferredQueryExpression¶: Conversion of case-sensitive field = string value expressions

convert_condition_field_exists(cond: ConditionFieldEqualsValueExpression, state: ConversionState) → str | DeferredQueryExpression¶: Conversion of field exists expressions

convert_condition_field_not_exists(cond: ConditionFieldEqualsValueExpression, state: ConversionState) → str | DeferredQueryExpression¶: Conversion of field not exists expressions

convert_condition_group(cond: ConditionItem, state: ConversionState) → str | DeferredQueryExpression¶: Group condition item.

convert_condition_not(cond: ConditionNOT, state: ConversionState) → str | DeferredQueryExpression¶: Conversion of NOT conditions.

convert_condition_or(cond: ConditionOR, state: ConversionState) → str | DeferredQueryExpression¶: Conversion of OR conditions.

convert_condition_query_expr(cond: ConditionValueExpression, state: ConversionState) → str | DeferredQueryExpression¶: Conversion of value-only plain query expressions.

convert_condition_val_num(cond: ConditionValueExpression, state: ConversionState) → str | DeferredQueryExpression¶: Conversion of value-only numbers.

convert_condition_val_re(cond: ConditionValueExpression, state: ConversionState) → str | DeferredQueryExpression¶: Conversion of value-only regular expressions.

convert_condition_val_str(cond: ConditionValueExpression, state: ConversionState) → str | DeferredQueryExpression¶: Conversion of value-only strings.

convert_correlation_event_count_rule(rule: SigmaCorrelationRule, output_format: str | None = None, method: str | None = None) → List[str]¶

Convert an event count correlation rule into the target data structure (usually query).

Args:: rule (SigmaCorrelationRule): The event count correlation rule to be converted. output_format (Optional[str]): The output format for the conversion. Defaults to None. method (Optional[str]): The correlation method to be used. Defaults to None.
Returns:: Any: The converted data structure.

convert_correlation_search_multi_rule_query_postprocess(query: str) → str¶: This function is called for each query in the multi-rule correlation search phase. It can be used to postprocess the query before it is joined with the other queries.

convert_correlation_temporal_ordered_rule(rule: SigmaCorrelationRule, output_format: str | None = None, method: str | None = None) → List[str]¶

Convert an ordered temporal correlation rule into the target data structure (usually query).

Args:: rule (SigmaCorrelationRule): The ordered temporal correlation rule to be converted. output_format (Optional[str]): The output format for the conversion. Defaults to None. method (Optional[str]): The correlation method to be used. Defaults to None.
Returns:: Any: The converted data structure.

convert_correlation_temporal_rule(rule: SigmaCorrelationRule, output_format: str | None = None, method: str | None = None) → List[str]¶

Convert a temporal correlation rule into the target data structure (usually query).

Args:: rule (SigmaCorrelationRule): The temporal correlation rule to be converted. output_format (Optional[str]): The output format for the conversion. Defaults to None. method (Optional[str]): The correlation method to be used. Defaults to None.
Returns:: Any: The converted data structure.

convert_correlation_typing_query_postprocess(query: str) → str¶: This function is called for each query in the typing phase of the correlation query. It can be used to postprocess the query before it is joined with the other queries.

convert_correlation_value_count_rule(rule: SigmaCorrelationRule, output_format: str | None = None, method: str | None = None) → List[str]¶

Convert a value count correlation rule into the target data structure (usually query).

Args:: rule (SigmaCorrelationRule): The value count correlation rule to be converted. output_format (Optional[str]): The output format for the conversion. Defaults to None. method (Optional[str]): The correlation method to be used. Defaults to None.
Returns:: Any: The converted data structure.

convert_value_re(r: SigmaRegularExpression, state: ConversionState) → str | DeferredQueryExpression¶: Convert regular expression into string representation used in query.

convert_value_str(s: SigmaString, state: ConversionState) → str¶: Convert a SigmaString into a plain string which can be used in query.

decide_string_quoting(s: SigmaString) → bool¶: Decide if string is quoted based on the pattern in the class attribute str_quote_pattern. If this matches (or not matches if str_quote_pattern_negation is set to True), the string is quoted.

escape_and_quote_field(field_name: str) → str¶

Escape field name by prepending pattern matches of field_escape_pattern with field_escape string. If field_escape_quote is set to True (default) and field escaping string is defined in field_escape, all instances of the field quoting character are escaped before quoting.

Quote field name with field_quote if field_quote_pattern (doesn’t) matches the original (unescaped) field name. If field_quote_pattern_negation is set to True (default) the pattern matching result is negated, which is the default behavior. In this case the field name is quoted if the pattern doesn’t matches.

finalize_query(rule: SigmaRule, query: str | DeferredQueryExpression, index: int, state: ConversionState, output_format: str) → str | DeferredQueryExpression¶: Finalize query by appending deferred query parts to the main conversion result as specified with deferred_start and deferred_separator.

get_flag_template(r: SigmaRegularExpression) → Dict[str, str]¶: Return the flag_x template variales used for regular expression templates as dict that maps flag_x template variable names to the static template if flag is set in regular expression r or an empty string if flag is not set.

quote_string(s: str) → str¶: Put quotes around string.