Backtesting

BlazeRules uses the same evaluation engine for live and historical data. A backtest evaluates recorded batches with an active ruleset and a candidate ruleset, then compares their decisions, scores, and match behavior before promotion.

📘

One engine for stream and backtest

Backtesting uses the standard evaluate_batch and evaluate_ndjson methods. The input source is historical Parquet or Arrow rather than a live stream.

Comparison workflow

A typical ruleset comparison has three steps:

1. Read historical records as Arrow RecordBatch objects.
2. Evaluate each batch through two engines: one loaded with the active rules and one with the candidate rules.
3. Compare per-record decisions, scores, and match counts. Differences identify records that the candidate would route differently.

The comparison uses the standard result fields documented in Observability: decisions, decision_codes, scores, risk_bands, winning_rule_ids, and match_counts. Grouped decision indices avoid per-record Python routing loops:

import blazerules

current = blazerules.RuleEngine()
current.load_rules("rules.yaml")

candidate = blazerules.RuleEngine()
candidate.load_rules("rules-candidate.yaml")

# For each historical batch:
cur = current.evaluate_batch(batch)
cand = candidate.evaluate_batch(batch)

# Records the candidate would route differently:
cur_groups = cur.grouped_decision_indices()
cand_groups = cand.grouped_decision_indices()
# Compare cur_groups vs cand_groups, or diff cur.decisions vs cand.decisions.

Reading historical data

Any source that produces typed Arrow batches can drive a backtest. Full builds provide blazerules_io.read_record_batches(path, batch_size=...); custom builds can read Parquet with pyarrow and pass each batch to the engine.

import blazerules
import blazerules_io

engine = blazerules.RuleEngine()
engine.load_rules("rules-candidate.yaml")

for batch in blazerules_io.read_record_batches("history.parquet", batch_size=16384):
    result = engine.evaluate_batch(batch)
    # accumulate result.decisions / result.scores for comparison

import pyarrow.parquet as pq
import blazerules

engine = blazerules.RuleEngine()
engine.load_rules("rules-candidate.yaml")

table = pq.read_table("history.parquet")
for batch in table.to_batches(max_chunksize=16384):
    result = engine.evaluate_batch(batch)
    # accumulate result.decisions / result.scores for comparison

Native backtest API

For Parquet history, use the built-in A/B comparison API.

report = engine.backtest(
    parquet_path=["history/day-1.parquet", "history/day-2.parquet"],
    rules_a="rules.yaml",
    rules_b="rules-candidate.yaml",
    label_column="fraud_label",
)

print(report.total_records)
print(report.fire_rate_a, report.fire_rate_b)
print(report.new_positives, report.lost_positives)
print(report.agreement_rate)
print(report.precision_a, report.recall_a)
print(report.precision_b, report.recall_b)

The C++ overloads are backtest(const BacktestConfig&) and backtest(parquet_paths, rules_a, rules_b, label_column). BacktestConfig contains parquet_paths, rules_file_a, rules_file_b, label_column, and batch_size.

The same comparison is available from the blazerules CLI over Parquet inputs (pass --path once per file):

blazerules backtest \
  --rules-a rules.yaml \
  --rules-b rules-candidate.yaml \
  --path history/day-1.parquet \
  --path history/day-2.parquet \
  --label-column fraud_label

The CLI output JSON reports total_records, fire_rate_a, fire_rate_b, new_positives, lost_positives, and agreement_rate. Passing --label-column additionally emits precision_a, recall_a, precision_b, and recall_b — the same label-based metrics as backtest(..., label_column=...) in Python/C++.

📘

Shadow rules

A rule can carry a shadow field. Shadow rules fire and contribute observability data without affecting the final decision. match_counts, winning_rule_ids, and grouped decision indices show the candidate fire rate and the records that would change after promotion.

🚧

Replay window rules in chronological order

Window rules read prior-batch history, inject derived window columns, evaluate the current batch, then commit that batch for future batches. Window backtests require chronological batch order and stable entity affinity. Same-batch repeated entity rows do not see earlier rows from the same batch by default. See Windows for the ordering contract.

Related documentation