---
updatedAt: 2026-07-14T22:49:00.000Z
agentTools:
  projectIndex: https://docs.blazerules.dev/llms.txt
---

# Ingestion Overview

All supported input paths and how they reach the same batch evaluation engine.

BlazeRules evaluates batches. Every input path either already is a batch, or collects records until a batch is ready.

```mermaid
flowchart LR
  json["JSON / NDJSON"] --> engine["RuleEngine"]
  arrow["Arrow / PyArrow"] --> engine
  kafka["Kafka"] --> io["blazerules_io"]
  ipc["Arrow IPC / Avro / Protobuf"] --> io
  s3["S3 / local files"] --> io
  http["HTTP /v1/logs"] --> agent["blazerules_agent"]
  stdin["stdin"] --> agent
  tail["file_tail"] --> agent
  k8s["Kubernetes pod logs"] --> agent
  io --> engine
  agent --> engine
  engine --> decisions["decisions / scores / rule counts"]
  engine --> dlq["dead-letter records"]
```

The `blazerules` CLI (`blazerules eval --input ...`) is an equivalent entry point to the same engine for JSON/Arrow and the binary formats, taking files, stdin, or Kafka instead of in-memory objects.

## Input Matrix

| Input                 | API or component                                                                        | Best for                                                                                                       |
| --------------------- | --------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| NDJSON bytes          | `engine.evaluate_ndjson(bytes)`, or `blazerules eval --input ndjson`                    | Raw event streams, JSON logs, HTTP batches.                                                                    |
| Top-level JSON arrays | `engine.evaluate_json_array(bytes)`, or `blazerules eval --input json-array`            | Batch APIs/files that already group events as one JSON array.                                                  |
| JSON string list      | `engine.evaluate_messages(list[str])`, or `blazerules eval --input ndjson`              | Small scripts and simple tests.                                                                                |
| PyArrow batch         | `engine.evaluate_batch(batch)`, or `blazerules eval --input arrow-ipc\|parquet`         | Typed pipelines and fastest non-JSON path.                                                                     |
| Kafka                 | `blazerules stream kafka`, `blazerules_io.KafkaConsumer`, `run_stream(...)`             | Brokered JSON, Arrow IPC, Avro, Protobuf, or Debezium streams with offset handling.                            |
| HTTP logs/events      | `blazerules_agent --input http`                                                         | Apps POST NDJSON to a local listener.                                                                          |
| stdin                 | `blazerules_agent --input stdin`                                                        | Terminal output, process logs, shell pipelines.                                                                |
| File tail             | `blazerules_agent --input file_tail`                                                    | Pod stdout/stderr files and node-local logs.                                                                   |
| Plain text logs       | wrap lines as JSON first                                                                | Unstructured terminal/stdout/stderr text.                                                                      |
| Kubernetes            | Helm chart DaemonSet                                                                    | Node-local log collection and dashboard sidecar/deployment.                                                    |
| Debezium CDC          | `blazerules eval --input debezium`, `blazerules_io.unwrap_debezium(...)`                | Database change streams.                                                                                       |
| Arrow IPC             | `blazerules eval --input arrow-ipc`, `blazerules_io.ArrowIpcDecoder`                    | Binary Arrow frames.                                                                                           |
| Avro                  | `blazerules eval --input avro`, `blazerules_io.AvroDecoder`/`decode_avro_ocf_file_each` | Schema-based binary events; Object Container Files decode every record via auto-detected magic bytes.          |
| Protobuf              | `blazerules eval --input protobuf\|protobuf-delimited`, `blazerules_io.ProtobufDecoder` | Descriptor-backed binary events; `protobuf-delimited` decodes every varint-length-delimited message in a file. |
| S3 / local files      | `read_ndjson_bytes`, `read_record_batches`                                              | Backtests, batch files, rules, lookups, and models.                                                            |

## Record Shape

JSON records can be flat or nested. Rules use dotted paths:

```yaml
conditions:
  and:
    - field: merchant.risk.score
      op: gt
      value: 50
    - array_any:
        path: items
        where:
          and:
            - field: price
              op: gt
              value: 100
            - field: category
              op: eq
              value: electronics
```

Arrow, Avro, Protobuf, and PyArrow inputs use the same logical field names. Nested structs are projected into the same dotted namespace, so a rule does not care which wire format produced the batch.

## Error Handling

Rules and lookups fail fast at load time. Ingest can be tolerant:

* `SKIP_AND_COUNT`: skip malformed records and count/sample errors.
* `SKIP_TO_DEAD_LETTER`: write bad rows to an NDJSON dead-letter file.
* `HARD_FAIL`: throw on the first malformed row.

See [Decision and DLQ Logs](decision-and-dlq-logs) and [Handle Dirty Data](handle-dirty-data).

## Recipes

<Cards columns={2}>
  <Card title="HTTP Logs" href="http-log-ingestion" icon="fa-globe">POST NDJSON batches to `/v1/logs`.</Card>
  <Card title="stdin Logs" href="stdin-log-ingestion" icon="fa-terminal">Pipe terminal or service logs into the agent.</Card>
  <Card title="File Tail" href="file-tail-ingestion" icon="fa-file-lines">Tail application or pod log files.</Card>
  <Card title="Plain Text Logs" href="plain-text-log-ingestion" icon="fa-align-left">Wrap unstructured text into JSON records.</Card>
  <Card title="Kubernetes Logs" href="kubernetes-log-ingestion" icon="fa-dharmachakra">Run the agent as a DaemonSet.</Card>
  <Card title="Kafka" href="kafka-streaming" icon="fa-tower-broadcast">Consume, evaluate, and publish decisions.</Card>
  <Card title="Binary Formats" href="binary-format-ingestion" icon="fa-code">Decode Arrow IPC, Avro, or Protobuf to Arrow batches.</Card>
</Cards>