Recipe: Load Rules, Lookups, and Models from S3

BlazeRules resolves exact-object s3://bucket/key URIs through the AWS CLI cache path. Use a named profile when possible.

import blazerules

blazerules.set_aws_profile("personal")
blazerules.set_aws_region("us-east-1")

engine = blazerules.RuleEngine()
engine.register_model("fraud_logreg", "s3://my-bucket/models/fraud_logreg.onnx")
engine.load_rules("s3://my-bucket/rules/rules.yaml")

export BLAZERULES_AWS_PROFILE=personal
export BLAZERULES_AWS_REGION=us-east-1

blazerules_agent \
  --rules s3://my-bucket/rules/rules.yaml \
  --input stdin \
  --output stdout

For MinIO, LocalStack, Cloudflare R2, or another S3-compatible endpoint:

blazerules.set_aws_endpoint_url("http://127.0.0.1:9000")

export BLAZERULES_AWS_ENDPOINT_URL=http://127.0.0.1:9000
blazerules_agent --rules s3://local-bucket/rules/rules.yaml --input stdin --output stdout

Environment variables are also supported:

export BLAZERULES_AWS_PROFILE=personal
export BLAZERULES_AWS_REGION=us-east-1
export BLAZERULES_AWS_ENDPOINT_URL=http://127.0.0.1:9000

import blazerules

blazerules.set_aws_profile("personal")
blazerules.set_aws_region("us-east-1")
blazerules.set_aws_endpoint_url("http://127.0.0.1:9000")

Avoid embedding credentials in rules, source code, or docs. If you must set credentials from Python in a short-lived local process:

blazerules.set_aws_credentials(
    access_key_id="...",
    secret_access_key="...",
    session_token="",
    region="us-east-1",
)

export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_SESSION_TOKEN=...
export AWS_REGION=us-east-1

Relative lookup paths in a rules file resolve relative to that rules file. For S3-hosted rules, keep lookup CSV paths either absolute s3://... URIs or relative keys under the same prefix.

Decision logs on S3 (stateless deployment)

Inputs are not the only thing that can live on S3 — the agent's decision-log output and the dashboard's input can too, so a pod holds no durable local state. This uses the same AWS CLI mechanism (and honors a custom endpoint), so it works against AWS or any S3-compatible store.

Point the agent's --output-path at an s3://…/prefix/. It writes rolled part objects locally and uploads them to the prefix on a background thread; each part is capped by --s3-roll-mb (default 64) and rolled/synced every --s3-flush-seconds (default 10). Each Arrow part is a complete, independently-readable IPC stream; each NDJSON part is standalone.

export AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... AWS_REGION=eu-central-1

blazerules_agent --name checkout --rules s3://my-bucket/rules/checkout.yaml --input http \
  --output arrow --output-path s3://my-bucket/decisions/checkout/ \
  --s3-roll-mb 64 --s3-flush-seconds 10

Point the dashboard at the same prefix (or a parent prefix with one sub-prefix per instance). It syncs new parts to a local cache each poll and feeds them into the same fast filter index used for local files:

blazerules_dashboard --decision-log-dir s3://my-bucket/decisions/ --rules s3://my-bucket/rules/checkout.yaml

Region and endpoint come from AWS_REGION/AWS_ENDPOINT_URL or the explicit --aws-region / --aws-endpoint-url flags on both binaries. On SIGINT/SIGTERM the agent flushes its final part before exiting, so a rolling deploy loses no decisions. The dead-letter file stays local NDJSON — mirror it separately if you need it centralized.

Reading s3:// input data files (native streaming)

Everything above resolves rules, lookups, models, and decision-log parts through the AWS CLI cache/sync path — that is unchanged. Reading s3:// input data files — the NDJSON/JSON, Arrow IPC, Parquet, or CSV records you evaluate — goes through a separate blazerules_io code path, and as of 0.5.3 it streams directly instead of downloading the whole object first.

By default (native_s3=True), an s3:// input file streams straight off Arrow's own S3 filesystem instead of shelling out to aws s3 cp to pull the whole object to local disk before reading a single row. Region, endpoint, and profile still come from the same set_aws_profile / set_aws_region / set_aws_endpoint_url calls (or BLAZERULES_AWS_* / AWS_* env vars) shown above — there is no separate auth configuration for the streaming path.

import blazerules, blazerules_io

blazerules.set_aws_profile("personal")
blazerules.set_aws_region("us-east-1")

engine = blazerules.RuleEngine()
engine.load_rules("rules.yaml")

def handle_batch(batch):
    engine.evaluate_batch(batch)
    return True  # return False to stop reading early

# Streams Parquet/CSV/Arrow-IPC batches straight off S3, one batch at a time.
options = blazerules_io.FileReadOptions()
options.native_s3 = True              # default: direct Arrow S3 streaming
options.allow_s3_cli_fallback = True  # default: retry via AWS CLI if native streaming fails first
blazerules_io.for_each_record_batch(
    "s3://my-bucket/events/day.parquet", "parquet", handle_batch, options,
)

# Simpler one-shot form for smaller NDJSON objects; also streams natively (no options to pass).
payload = blazerules_io.read_ndjson_bytes("s3://my-bucket/events/day.ndjson")
engine.evaluate_ndjson(payload)

If the native read fails before any batch has reached the caller — a permissions error, an unreachable custom endpoint, or a build without S3 support compiled in — BlazeRules automatically retries the same object through the old AWS-CLI-download path, so one flaky attempt does not require caller-side retry logic. Once at least one batch has been delivered, no fallback is attempted, so a partially-streamed file is never silently re-delivered from the top.

blazerules eval --rules rules.yaml --input parquet --path s3://my-bucket/events/day.parquet --output summary

📘

Two independent knobs, and an NDJSON note

FileReadOptions.native_s3 (default True) turns native streaming on or off; FileReadOptions.allow_s3_cli_fallback (default True) controls whether a failed native attempt retries via the AWS CLI path. Setting native_s3=False always uses the AWS-CLI path directly (the pre-0.5.3 behavior). None of this touches how rules/lookups/models or decision-log parts are resolved above. For NDJSON specifically, the raw-chunk streaming visitor the CLI uses internally (for_each_ndjson_chunk) is currently C++-only and not yet exposed to Python — from Python, stream NDJSON as Arrow batches instead with for_each_record_batch(path, "ndjson", callback, options), or use the one-shot read_ndjson_bytes shown above.