Recipe: Handle Dirty JSON

Ruleset loading is strict, but ingest can be tolerant. For streams with malformed records or occasional type drift, configure the engine to skip and count rather than fail the whole batch.

Malformed JSON lines in an NDJSON stream are recorded and skipped by both the Python API and the CLI (they no longer abort the batch), so a bad line in the middle of a file does not stop the good records around it.

import blazerules

cfg = blazerules.EngineConfig()
cfg.output_detail = blazerules.OutputDetail.DECISIONS
cfg.ingest_error_mode = blazerules.IngestErrorMode.SKIP_AND_COUNT
cfg.type_mismatch_mode = blazerules.TypeMismatchMode.NULL_ON_TYPE_ERROR
cfg.max_error_samples = 16

engine = blazerules.RuleEngine(cfg)
engine.load_rules("rules.yaml")

result = engine.evaluate_ndjson(payload)
print(result.messages_processed, result.messages_skipped)
print(result.error_counts)
print(result.error_samples)

CLI equivalent: tolerant ingest through the agent

cat events.ndjson | \
  blazerules_agent \
    --rules rules.yaml \
    --input stdin \
    --output ndjson \
    --output-path decisions.ndjson

The agent keeps processing valid records. For explicit dead-letter file policy, configure EngineConfig in Python or C++.

Use COERCE when source systems sometimes send numeric values as strings:

cfg.type_mismatch_mode = blazerules.TypeMismatchMode.COERCE

CLI equivalent: use COERCE from the shell

cat events.ndjson | blazerules eval --rules rules.yaml --input ndjson --stdin --type-mismatch-mode coerce --output summary

Use hard-fail modes for strict offline jobs:

cfg.ingest_error_mode = blazerules.IngestErrorMode.HARD_FAIL
cfg.type_mismatch_mode = blazerules.TypeMismatchMode.HARD_FAIL_TYPE

To capture bad rows for inspection, set a dead-letter path:

cfg.ingest_error_mode = blazerules.IngestErrorMode.SKIP_TO_DEAD_LETTER
cfg.dead_letter_path = "bad_records.ndjson"

Normal comparisons on null or failed-coercion fields return false. Use explicit is_null, is_not_null, is_empty, and is_not_empty rules when null state itself is meaningful.