mmsigma

An rsyslog message-modification module that evaluates Sigma rules against every parsed log event, on the machine that collected the log. It does not interpret YAML at runtime. A Python compiler (pySigma) turns a rule corpus into a compact binary; a C matcher loads that artifact zero-copy and annotates $!sigma.

Correlation is the same C engine, in the same action. Commutative windows (event_count, value_count, temporal, sum/avg) merge across collocated peers by hashing each (corr_id, group) onto one owner. The module does not open sockets. Transport is ordinary rsyslog output (omfwd / omrelp). On MarlinOS the owner ring is a static list, named by the device overlay.

mmsigma is a stage of the WAKE pipeline. It also stands on its own: the matcher core is libsigma, the Advens Sigma engine, with no dependency on rsyslog or liblognorm, and the same engine drives a CLI and the test harness.

Why match on the collector

The usual path ships every line, indexes it, translates Sigma into the SIEM query language, then waits. Detection latency becomes pipeline latency, WAN bandwidth becomes a prerequisite, and the collector pays interpreter overhead on every event for work that was fixed when the ruleset was built.

On the collector the parsed event is already in memory. Compile once, match per event, ship the verdict. Forwarding raw logs is then a policy choice, not a requirement.

Compiler split

Slow, expressive, and trusting on the management host. Fast, minimal, and paranoid on every collector.

  offline (Python / pySigma)              online (C, each collector)
  --------------------------              -------------------------
  Sigma YAML corpus                       rsyslog
    |                                       |
    v                                       +-- mmnormalize + TurboVM
  taxonomy → ECS fields                     +-- mmsigma
  required-literal CNF                            1. category bucket
  rules.sigmac  ---- binary artifact ------→      2. Teddy SIMD prefilter
  rules.corr.json -- correlations ---------→      3. RPN verify + memo
  rules.sigmac.manifest.json -- ledger ----→      correlation windows
                                                  annotate $!sigma
                                                  emit $!sigma_contribution
                                                    |
                                                    v
                                              hashed omfwd to owner

The compiler accepts a large untrusted corpus and degrades per rule (a rule that fails to compile is recorded and skipped). The loader accepts nothing it cannot fully validate: magic, version, sizes, CRC32C, every internal reference in range. A truncated file mid-reload keeps the previous generation.

Three tiers, each with a soundness contract

1. Logsource / category buckets

A rule is narrowed into a category only when the compiler is confident. An unconstrained rule joins the always-verify set. A wrongly-bucketed rule would be a silent miss; a conservative extra verify is the accepted cost.

2. Required-literal CNF + Teddy SIMD prefilter

Compile-time extraction of needles the event must contain if the rule is to match. The scan is field-scoped text, SSE 4.2 / NEON substring, two-byte fingerprint. A prefilter hit is a candidate, never a decision. The prefilter can only over-select.

3. RPN verify with field memoization

Each surviving rule’s condition is a postfix program over selections. Field lookup runs at most once per distinct field per event. Operators cover EQ / contains / startswith / endswith, numeric compares, CIDR, PCRE2 regex, exists. Negation is boolean. Fieldless keywords compile to event.original (the raw line).

Every stage is checked against a linear-scan oracle: the narrowed hit-set must equal the exhaustive hit-set for every test event.

Bounded correlation

Sigma correlation primitives run next to the matcher, not inside it. State is operator-tunable and strictly bounded, so worst-case memory is never a function of adversary-controlled cardinality. Windows default to parsed event time (@timestamp, then aliases, then the syslog header).

Correlations whose base rule did not compile land in the dropped-rules ledger, which is persisted next to the artifact rather than counted and thrown away: a corpus you cannot see is a corpus you cannot distribute. The intent is that no rule disappears silently, and the compiler is built around it, though the property is only as good as its last audit.

Mesh pattern

One engine, two jobs. Match and local apply stay on the node that saw the log. Commutative partials for a group are applied on exactly one peer: the owner. That is not gossip. It is consistent hashing onto a ring the operator declares in the overlay.

What travels

A contribution is a small JSON object (or an array of them): kind, corr_id, event-time ts, resolved group values, and the partial (count / numeric / seen_mask / value), emitted as an array with one entry per matching correlation rule. omfwd pool.policy="hash" routes it on the same djb2 key; that pooling mode is a MarlinOS-carried rsyslog patch pending upstream, not stock behaviour. It will not be PlumTree-gossiped.

A fire that ingested a remote partial sets $!sigma_correlation.correlations[].remote = true, marking it as cross-instance evidence rather than something one node saw end to end. The flag is written today but not yet read back, so nothing downstream ranks on it.

When a ring member dies

The ring is static, so this is the sharp edge. The module never consults liveness: it hashes onto peercount whatever the state of the peers. What happens next is worth knowing before you size a ring.

Non-owners skip local apply for the groups that hash to the dead index. omfwd with pool.policy="hash" may then linear probe to the next connected target, so the TCP session lands somewhere, but that is transport and not ownership. Ingest recomputes the owner and refuses any contribution this node does not own, counting it on corr.contrib.drop. Applying a mis-routed partial would split-brain the group when the real owner came back, so it is dropped instead.

The result is that a dead owner parks its share of the groups rather than corrupting them: those correlations stop advancing until the ring is rewritten, and everything else is unaffected. Losing a window is recoverable, and a group with two divergent partial states is not, so the trade is deliberate.

Making the ring track liveness is a control-plane job: something outside the module has to watch membership and re-render both peercount and the omfwd target list. Putting that inside mmsigma would cost the property that makes it portable, which is that it opens no sockets.

What it will not do

• Open sockets. mmsigma emits and ingests JSON. rsyslog (or a later immesh unicast) moves the bytes. classify_corr stays transport-free.
• A second correlator on the same partials. Python reads $!sigma_correlation. It must not re-count (corr_id, group). The campaign matcher is attack-chain in waked: fixed patterns over signals, not another window on these partials, and not the pheromone table.
• WAN-wide short windows. The ring is meant to be collocated, and high RTT forbids the same event_count across sites. A coupling in TAC is same-entity memory in shadow. It is not a cross-site kill chain.
• Runtime YAML. Modifiers (|base64, |wide, |cidr, …) are materialized at compile time. The C engine sees only concrete operands.
• A global automaton over the raw line. Sigma predicates are field-scoped comparisons. Flattening the parse back into bytes for Hyperscan-style matching discards the structure the parser produced. The automaton-shaped part (which of thousands of literals appear) is the Teddy prefilter, and nowhere else.

In WAKE, and without it

On a MarlinOS edge, per-source rulesets run mmenrich then mmwake. mmsigma runs after that, once, in the shared logstore ruleset. Parsed fields come from TurboVM. Matches feed entity scoring and optional OCSF export to a downstream SIEM. The pheromone mesh is the scoring gossip plane; this page is the Sigma reduce plane.

Proposed rsyslog contribution (contrib/mmsigma). Until that lands upstream, it ships in the MarlinOS rsyslog port. The upstream module does not require SWIM, Fleet, or .prev. Those are the WAKE wrapper. TAC is not on this plane.