Ingestion Guide

Learn how to send log entries to Krafter using single and batch ingestion, structure your log data for effective searching, and follow best practices for production use.

Authentication

All ingest endpoints use your project ingest token (Authorization: Bearer <ingest_token>), not your team API key. Ingest tokens are generated when you create a project and can be rotated at any time.

Single Log Entry

Send one log entry at a time using JSON:

curl -X POST https://app.krafter.dev/api/v1/logs/ingest \
  -H "Authorization: Bearer krl_live_kp7tx2bm4qrs6vwy3jnh5zfd7acmeg2j" \
  -H "Content-Type: application/json" \
  -d '{
    "level": "error",
    "message": "Failed to connect to database",
    "stream": "backend",
    "service": "api-server",
    "trace_id": "abc123def456",
    "span_id": "span_789",
    "metadata": {
      "host": "db-primary.internal",
      "port": 5432,
      "retry_count": 3
    }
  }'

// 202 Accepted
{
  "ok": true
}

Batch Ingestion

Send multiple log entries in a single request using NDJSON (Newline-Delimited JSON). Each line is a separate JSON object. Maximum request size is 5 MB.

curl -X POST https://app.krafter.dev/api/v1/logs/ingest/batch \
  -H "Authorization: Bearer krl_live_kp7tx2bm4qrs6vwy3jnh5zfd7acmeg2j" \
  -H "Content-Type: application/x-ndjson" \
  -d '{"level":"info","message":"Request started","stream":"backend","service":"api-server","trace_id":"abc123"}
{"level":"info","message":"Query executed in 42ms","stream":"backend","service":"api-server","trace_id":"abc123"}
{"level":"info","message":"Response sent","stream":"backend","service":"api-server","trace_id":"abc123"}'

// 202 Accepted
{
  "ok": true,
  "count": 3
}

TIP

NDJSON format means one JSON object per line with no trailing commas or wrapping array. Each line must be valid JSON on its own.

Log Entry Schema

Field	Type	Required	Description
message	string	Yes	The log message text.
level	string	No	Log severity level. Defaults to "info".
stream	string	No	Logical grouping for logs (e.g., "backend", "worker", "cron").
service	string	No	Name of the service that produced the log.
trace_id	string	No	Distributed trace identifier for correlating logs across services.
span_id	string	No	Span identifier within a trace.
metadata	object	No	Arbitrary key-value pairs for structured data.

Log Levels

Level	Description
debug	Detailed diagnostic information for development.
info	General operational events. Default level.
warn	Potential issues that are not errors.
error	Errors that need attention but the service can continue.
fatal	Critical failures that may cause the service to stop.

Streams

Streams are logical groupings for your logs. They are auto-created from the stream field in your log entries -- there is no need to create them in advance.

Common stream naming patterns:

• By component: backend, frontend, worker, cron
• By feature: auth, payments, notifications
• By environment: staging, production

You can list all streams for a project using the List Streams endpoint.

Distributed Tracing

Use trace_id and span_id to correlate logs across multiple services in a distributed system:

{"level": "info", "message": "API request received", "service": "gateway", "trace_id": "tr_abc123", "span_id": "sp_001"}
{"level": "info", "message": "Fetching user data", "service": "user-service", "trace_id": "tr_abc123", "span_id": "sp_002"}
{"level": "info", "message": "Query completed", "service": "user-service", "trace_id": "tr_abc123", "span_id": "sp_003"}
{"level": "info", "message": "Response sent", "service": "gateway", "trace_id": "tr_abc123", "span_id": "sp_004"}

Search for all logs in a trace by filtering on trace_id in the Search API.

Ingest Token Rotation

If an ingest token is compromised, rotate it immediately. The old token stops working as soon as the new one is generated.

curl -X POST https://app.krafter.dev/api/v1/logs/projects/a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d/rotate_token \
  -H "Authorization: Bearer kr_live_abc123def456"

{
  "data": {
    "id": "a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
    "name": "My App",
    "ingest_token_prefix": "krl_live_xn4",
    "ingest_token": "krl_live_xn4w7bm2tk6qrs3vwy5jnh7zfdcameg2p",
    "..."
  }
}

WARNING

Token rotation uses your team API key, not the ingest token. Update all services sending logs with the new ingest token after rotation.

Best Practices

1. Use structured metadata -- Put searchable data in the metadata object instead of embedding it in the message string. This makes filtering and analysis much easier.

2. Set meaningful stream names -- Use consistent, descriptive stream names to organize logs by component or feature. This helps when filtering in the dashboard and search API.

3. Always include a service name -- When running multiple services, the service field lets you quickly isolate logs from a specific service.

4. Use batch ingestion in production -- Buffer log entries and send them in batches to reduce HTTP overhead and improve throughput. Aim for batches of 100--1,000 entries.

5. Include trace IDs for distributed systems -- Propagate trace_id across service boundaries to correlate related log entries.

6. Set appropriate log levels -- Use debug for development details, info for normal operations, warn for concerning but non-critical events, error for failures, and fatal for unrecoverable situations.

Operational Caveats

A few aspects of how Krafter handles ingest are worth knowing before scaling out — they show up only in multi-pod deployments or under sustained high-throughput load.

Stream auto-discovery is per-pod

Krafter caches "this stream already exists" lookups in a per-pod ETS table (Krafter.Logs.StreamCache, 5-minute TTL) to avoid hitting Postgres on every ingest. Cache state is not shared across pods: when a new stream name lands on a pod that has not seen it yet, that pod issues a Postgres INSERT … ON CONFLICT DO NOTHING and primes its local cache.

This is safe — on_conflict: :nothing keeps the operation idempotent — but it means that on a fresh deploy or after a pod restart you may see one extra DB round-trip per (pod, stream_name) pair until the cache warms up. Stream creation never produces duplicates regardless of pod count.

Ingest rate limit is per-pod

The log_ingest bucket (300 req / 60s per ingest token) is enforced via in-memory ETS in KrafterWeb.Plugs.RateLimit, which is not shared across pods. On a deploy with N pods behind the load balancer, the effective per-token throughput cap is 300 × N requests per minute — a request is only rejected when a single pod sees more than 300 requests for the same token in a 60-second window.

For an SDK that pins to one pod (e.g. via session affinity) the per-pod cap and the documented limit coincide. For round-robin load balancing assume the higher real-world ceiling and size your batches accordingly. If you need a true global cap for billing or abuse-prevention purposes, post-aggregate from the log_ingestion_usage table rather than relying on the rate-limit response.