Telemetry Pipeline Delivery Semantics

This document describes the delivery guarantees of the Pavri telemetry pipeline, covering event ingestion, storage, and the handling of duplicates, timestamps, and ordering.

Delivery Guarantee: At-Least-Once

The Pavri telemetry pipeline provides at-least-once delivery semantics.

Why At-Least-Once

The pipeline architecture is:

SDK → gRPC Gateway → NATS JetStream → Store Consumer → ClickHouse

• NATS JetStream provides at-least-once delivery by default. Messages are persisted to disk and redelivered if the consumer does not acknowledge them.
• ClickHouse MergeTree does not deduplicate rows by default. Each INSERT is appended to the table. Deduplication would require ReplacingMergeTree with explicit OPTIMIZE FINAL or query-time dedup (FINAL modifier).

This means the same event may be stored more than once if:

• The store consumer crashes after writing but before acknowledging to JetStream.
• Network retries between the SDK and gateway cause the same event to be published twice.
• The SDK retries a failed PublishEvents call.

Implications

Scenario	Behavior
Normal operation	Each event stored exactly once
Consumer restart during processing	Events redelivered, may produce duplicates
SDK retry on timeout	Same event_id stored again
JetStream redelivery	Duplicate row in ClickHouse

Recommended Practices

1. Assign unique event_id values in the SDK (the default behavior). This enables query-time deduplication using GROUP BY event_id or argMax().
2. Use query-time dedup when exact counts matter:

   SELECT DISTINCT event_id, * FROM telemetry_events WHERE agent_id = '...'

3. Do not rely on strict ordering across different sessions or agents. Events within a single PublishEvents batch maintain relative order, but interleaving between concurrent publishers is non-deterministic.

Timestamp Handling

• occurred_at is set by the SDK at the time the event happens and preserved through the pipeline without modification.
• sent_at (in the request context) records when the SDK sent the batch.
• ClickHouse stores timestamps in UTC. The pipeline does not shift or truncate sub-second precision.

Event Ordering

• Events within a single PublishEvents gRPC call are written in order.
• Events across different calls or from different agents may interleave due to concurrent NATS consumers.
• Query results should be sorted by occurred_at to reconstruct timeline order.

Future Improvements

• Idempotent inserts: Migrate telemetry tables to ReplacingMergeTree(occurred_at) keyed by event_id to enable background deduplication.
• Exactly-once consumer: Use JetStream's AckSync with atomic ClickHouse batch commits to prevent redelivery-caused duplicates.
• Dedup metrics: Expose a metric counting duplicate event_id insertions so operators can monitor pipeline health.