Ingestion Overview

BlazeRules evaluates batches. Every input path either already is a batch, or collects records until a batch is ready.

flowchart LR
  json["JSON / NDJSON"] --> engine["RuleEngine"]
  arrow["Arrow / PyArrow"] --> engine
  kafka["Kafka"] --> io["blazerules_io"]
  ipc["Arrow IPC / Avro / Protobuf"] --> io
  s3["S3 / local files"] --> io
  http["HTTP /v1/logs"] --> agent["blazerules_agent"]
  stdin["stdin"] --> agent
  tail["file_tail"] --> agent
  k8s["Kubernetes pod logs"] --> agent
  io --> engine
  agent --> engine
  engine --> decisions["decisions / scores / rule counts"]
  engine --> dlq["dead-letter records"]

The blazerules CLI (blazerules eval --input ...) is an equivalent entry point to the same engine for JSON/Arrow and the binary formats, taking files, stdin, or Kafka instead of in-memory objects.

Input Matrix

Input	API or component	Best for
NDJSON bytes	`engine.evaluate_ndjson(bytes)`, or `blazerules eval --input ndjson`	Raw event streams, JSON logs, HTTP batches.
Top-level JSON arrays	`engine.evaluate_json_array(bytes)`, or `blazerules eval --input json-array`	Batch APIs/files that already group events as one JSON array.
JSON string list	`engine.evaluate_messages(list[str])`, or `blazerules eval --input ndjson`	Small scripts and simple tests.
PyArrow batch	`engine.evaluate_batch(batch)`, or `blazerules eval --input arrow-ipc\|parquet`	Typed pipelines and fastest non-JSON path.
Kafka	`blazerules stream kafka`, `blazerules_io.KafkaConsumer`, `run_stream(...)`	Brokered JSON, Arrow IPC, Avro, Protobuf, or Debezium streams with offset handling.
HTTP logs/events	`blazerules_agent --input http`	Apps POST NDJSON to a local listener.
stdin	`blazerules_agent --input stdin`	Terminal output, process logs, shell pipelines.
File tail	`blazerules_agent --input file_tail`	Pod stdout/stderr files and node-local logs.
Plain text logs	wrap lines as JSON first	Unstructured terminal/stdout/stderr text.
Kubernetes	Helm chart DaemonSet	Node-local log collection and dashboard sidecar/deployment.
Debezium CDC	`blazerules eval --input debezium`, `blazerules_io.unwrap_debezium(...)`	Database change streams.
Arrow IPC	`blazerules eval --input arrow-ipc`, `blazerules_io.ArrowIpcDecoder`	Binary Arrow frames.
Avro	`blazerules eval --input avro`, `blazerules_io.AvroDecoder`/`decode_avro_ocf_file_each`	Schema-based binary events; Object Container Files decode every record via auto-detected magic bytes.
Protobuf	`blazerules eval --input protobuf\|protobuf-delimited`, `blazerules_io.ProtobufDecoder`	Descriptor-backed binary events; `protobuf-delimited` decodes every varint-length-delimited message in a file.
S3 / local files	`read_ndjson_bytes`, `read_record_batches`	Backtests, batch files, rules, lookups, and models.

Record Shape

JSON records can be flat or nested. Rules use dotted paths:

conditions:
  and:
    - field: merchant.risk.score
      op: gt
      value: 50
    - array_any:
        path: items
        where:
          and:
            - field: price
              op: gt
              value: 100
            - field: category
              op: eq
              value: electronics

Arrow, Avro, Protobuf, and PyArrow inputs use the same logical field names. Nested structs are projected into the same dotted namespace, so a rule does not care which wire format produced the batch.