Webhook outbox Live

Outbound fanout to the siblings · HMAC signed, one immutable idempotency key per subscriber, retry ladder, dead letters with an owner

Register the subscription before switching on the first topic. ops2's emit() writes nothing at all when no subscriber is registered for an event type, and returns 0 silently. Enable a topic first and the first hours of events vanish with no trace on either side. i3 makes the order a database constraint, so the mistake cannot be made by hand. Ruling L18. Inbound side →
Live 24
DLQ 3
Subscribers 6
Retry policy
Signing

Per-subscriber health

The sequencing checklist is the L18 guard made visible: a topic cannot be enabled until the sibling has confirmed the subscription

ops2activeFulfillment Dashboard
$I3_OPS2_WEBHOOK_URL · secret $I3_OPS2_WEBHOOK_SECRET
✓Subscription row created12-07 09:14
✓Sibling confirmed the handshake12-07 09:22
✓Topics enabled after confirmation12-07 10:01
✓First event delivered12-07 10:04
Sent 24h
612
Success
99.8%
p95
204ms
3 topics · i3.balance.changed, i3.claim.decided, i3.stockout.raised
csdactiveCustomer Service
$I3_CSD_WEBHOOK_URL · secret $I3_CSD_WEBHOOK_SECRET
✓Subscription row created14-07 11:02
✓Sibling confirmed the handshake14-07 11:08
✓Topics enabled after confirmation14-07 11:30
✓First event delivered14-07 11:31
Sent 24h
318
Success
99.7%
p95
168ms
2 topics · i3.reservation.confirmed, i3.balance.changed
rfidactiveRFID app
$I3_RFID_WEBHOOK_URL · secret $I3_RFID_WEBHOOK_SECRET
✓Subscription row created18-07 08:40
✓Sibling confirmed the handshake18-07 08:44
✓Topics enabled after confirmation18-07 09:00
✓First event delivered18-07 09:02
Sent 24h
204
Success
100%
p95
142ms
1 topic · i3.unit.state_changed
skuintelpausedSKU Intelligence
$I3_SKUINTEL_WEBHOOK_URL · secret $I3_SKUINTEL_WEBHOOK_SECRET
✓Subscription row created20-07 14:10
✓Sibling confirmed the handshake20-07 14:18
✓Topics enabled after confirmation20-07 15:00
⏸Paused for their PG migration29-07 11:20
Backlog
113
Oldest
3h 21m
p95
418ms
113 messages held at pending and nothing marked delivered. ops2's own drain does the opposite: a deactivated subscriber has its whole queue stamped delivered_at and counted as delivered.
grnawaiting confirmGRN app
$I3_GRN_WEBHOOK_URL · secret $I3_GRN_WEBHOOK_SECRET
✓Subscription row created28-07 16:44
…Sibling confirmation outstanding1d 22h
×Topics blocked by ck_subs_sequence-
×No event delivered-
Queued
0
Success
-
p95
-
Nothing is enqueued while the handshake is outstanding, so there is no backlog to lose and no half-registered window.
shopifygate-blockedAdmin API write-back
$I3_SHOPIFY_ADMIN_URL · token $I3_SHOPIFY_ADMIN_TOKEN
✓Subscription row created22-07 10:00
✓Endpoint reachable, signature accepted22-07 10:06
🔒Topic held until the Phase 9 cutover gate-
×No inventory level written yet-
Held
4,182
Oldest
7d
Dry-run OK
100%
The rows are created and held rather than discarded, so the fanout is provable before the flag flips. A held row is evidence; a discarded one is not.

Live deliveries

One row per (message, subscriber) · the idempotency key is generated once at enqueue and is immutable, so attempt 7 carries the same key as attempt 1

EnqueuedTopicSubscriberIdempotency key AttHTTPLatencyNext attemptStatus
14:41:22i3.balance.changedops2i3:i3.balance.changed:8a11…ff02:ops21202118ms-delivered
14:41:22i3.balance.changedcsdi3:i3.balance.changed:8a11…ff02:csd120296ms-delivered
14:41:22i3.balance.changedskuinteli3:i3.balance.changed:8a11…ff02:skuintel350310.0s15:11:22retrying
14:39:04i3.unit.state_changedrfidi3:i3.unit.state_changed:c4d2…0091:rfid1200142ms-delivered
14:34:19i3.reservation.confirmedcsdi3:i3.reservation.confirmed:71bd…4a10:csd1202184ms-delivered
14:31:07i3.claim.decidedops2i3:i3.claim.decided:d902…1f77:ops2120296ms-delivered
14:28:44i3.stockout.raisedops2i3:i3.stockout.raised:4e18…bb31:ops21202128ms-delivered
14:22:10i3.balance.changedskuinteli3:i3.balance.changed:2c74…8e05:skuintel45028.4s16:22:10retrying
14:18:33i3.receipt.postedrfidi3:i3.receipt.posted:9fa1…7c22:rfid1200210ms-delivered
14:16:08i3.balance.changedops2i3:i3.balance.changed:6bd0…03c9:ops22202308ms-delivered on 2
14:12:51shopify.inventory_level.setshopifyi3:shopify.inventory_level.set:aa03…5512:shopify0--GATE-9held by gate
14:09:12i3.stockout.clearedcsdi3:i3.stockout.cleared:1d55…9902:csd1202104ms-delivered
11:20:04i3.balance.changedskuinteli3:i3.balance.changed:70ae…c418:skuintel0--on resumepending, paused
27-07 03:14i3.claim.decidedops2i3:i3.claim.decided:5502…ae71:ops2742288ms-dead letter
13:58:41i3.unit.state_changedrfidi3:i3.unit.state_changed:e770…2d84:rfid1200156ms-delivered
13:44:02i3.reservation.rejectedcsdi3:i3.reservation.rejected:b311…6ff0:csd1202122ms-delivered
13:31:19i3.balance.changedops2i3:i3.balance.changed:cd47…7b19:ops21202142ms-delivered
1,247 deliveries today · 17 shownThe key column is the whole point. i2 called uuid.uuid4() inside deliver_one, so every retry arrived at the sibling as a brand new event.

Retry ladder in flight

Six backoff steps then a dead letter · the ladder is per subscriber and editable

8 retrying
skuintel · i3.balance.changed · 8a11…ff02step 3 of 6 · next 15:11
1m5m30m2h6h24hDLQ
skuintel · i3.balance.changed · 2c74…8e05step 4 of 6 · next 16:22
1m5m30m2h6h24hDLQ
ops2 · i3.claim.decided · 5502…ae71exhausted · dead letter
1m5m30m2h6h24hDLQ
A 4xx dead-letters immediately without walking the ladder, because a rejected body will be rejected identically in six hours. A 5xx, a timeout or a network error walks the ladder. A 401 is treated as 5xx on the first occurrence only, since a rotating secret produces a transient 401.
00:00 · 34 delivered 02:00 · 28 delivered 04:00 · 22 delivered 06:00 · 31 delivered 08:00 · 46 delivered 09:00 · 54 delivered 10:00 · 58 delivered 11:00 · 56 delivered 11:00 · 6 retrying, skuintel paused 12:00 · 52 delivered 12:00 · 6 retrying 13:00 · 50 delivered 13:00 · 5 retrying 14:00 · 57 delivered 14:00 · 8 retrying 15:00 · 39 delivered 16:00 · 35 delivered 17:00 · 30 delivered 18:00 · 38 delivered 19:00 · 33 delivered 20:00 · 27 delivered 21:00 · 21 delivered 22:00 · 17 delivered 23:00 · 13 delivered 00:00 · 9 delivered 01:00 · 6 delivered 02:00 · 4 delivered 02:00 · 1 dead letter
00:0008:0016:0002:00
delivered retrying dead letter

Dead letter queue

Nothing is auto-discarded. A dead letter needs an owner, then either a successful replay or an explicit discard with a reason. Both are audited.

3
Failed atTopicSubscriberAttLast errorClaimed byAge
27-07 03:14i3.claim.decidedops27422 unknown anchor_key formatahmad2d 11h
25-07 21:40i3.balance.changedskuintel7timeout after 10.0s ×7fatima3d 17h
29-07 10:52i3.receipt.postedgrn1404 endpoint not foundunclaimed 4h4h 12m
Unclaimed beyond 2h raises fanout.dlq_unclaimed at P2The GRN 404 is the sequencing rule proving itself: the topic was never enabled, so this row is a manual enqueue from a test, not a lost event.
Replay contract
POST /api/outbox/1204/replay
→ re-sends the ORIGINAL idempotency key
→ i3:i3.claim.decided:5502…ae71:ops2
→ attempts counter continues at 8, not reset

A replay is not a new message. It reuses the key so that if ops2 processed attempt 4 and lost the ACK, the replay is a no-op on their side. Resetting the counter would hide how many times i3 has already knocked.

Retry policy and signing, per subscriber

Endpoint comes from an environment variable name. There is no default URL anywhere in i3, and a subscriber with no secret reference cannot be active.

ops2Fulfillment Dashboard

URL from
$I3_OPS2_WEBHOOK_URL
Secret from
$I3_OPS2_WEBHOOK_SECRET
Signature
X-I3-Signature sha256=
Timestamp
X-I3-Timestamp · 300s window
Idempotency
X-I3-Idempotency-Key
Timeout
10s
Max attempts
7
1m5m30m2h6h24hDLQ

csdCustomer Service

URL from
$I3_CSD_WEBHOOK_URL
Secret from
$I3_CSD_WEBHOOK_SECRET
Signature
X-I3-Signature sha256=
Timestamp
X-I3-Timestamp · 300s window
Idempotency
X-I3-Idempotency-Key
Timeout
8s
Max attempts
7
1m5m30m2h6h24hDLQ

rfidRFID app

URL from
$I3_RFID_WEBHOOK_URL
Secret from
$I3_RFID_WEBHOOK_SECRET
Signature
X-I3-Signature sha256=
Timestamp
X-I3-Timestamp · 300s window
Idempotency
X-I3-Idempotency-Key
Timeout
6s
Max attempts
5
1m5m30m2h6hDLQ

skuintelSKU Intelligence

URL from
$I3_SKUINTEL_WEBHOOK_URL
Secret from
$I3_SKUINTEL_WEBHOOK_SECRET
Signature
X-I3-Signature sha256=
Timestamp
X-I3-Timestamp · 300s window
Idempotency
X-I3-Idempotency-Key
Timeout
15s
Max attempts
7
2m8m30m2h6h24hDLQ

grnGRN app

URL from
$I3_GRN_WEBHOOK_URL
Secret from
$I3_GRN_WEBHOOK_SECRET
Signature
X-I3-Signature sha256=
Timestamp
X-I3-Timestamp · 300s window
Idempotency
X-I3-Idempotency-Key
Timeout
10s
Max attempts
7
1m5m30m2h6h24hDLQ

shopifyAdmin API write-back

URL from
$I3_SHOPIFY_ADMIN_URL
Token from
$I3_SHOPIFY_ADMIN_TOKEN
Auth
X-Shopify-Access-Token
Budget
shopify_rest bucket
Idempotency
key held, not sent
Timeout
15s
Max attempts
5
1m5m30m2h6hDLQ
i3 never sends an unsigned request. i2's fanout computed signature = _sign(body, secret) if secret else "", so an unset environment variable produced an empty signature header and the POST went out anyway. Combined with the receiving side's if secret and not verify(...), an unconfigured pair of apps ran completely unsigned in both directions and reported healthy. Here a null secret reference makes the subscriber row fail its CHECK.
Loading

Subscriber cards keep their heights, so the sequencing checklist does not shift as counts arrive.

Empty and zero-filter
Nothing outbound yet.

Six subscribers registered, four confirmed, and no message enqueued. Before Phase 2 that is the expected state, not a fault. Zero-filter names the chips doing the hiding.

Error
Outbox query failed.

The fanout worker is a separate process group and keeps draining. This page failing does not pause delivery.

wo-51ab7c · 29-07-2026 14:32 GST
Permission

Claiming or discarding a dead letter needs role >= ops_manager plus i3.outbox_admin. Payload bodies and request headers are gated the same way, since a payload can carry customer detail.

Maintenance

A sibling deploy window is a pause, not a stop. Messages accumulate at pending with a visible backlog age, and the fanout.backlog_age condition fires at P2 past one hour.

Gate-blocked

4,182 Shopify write-back rows are held at pending with the topic disabled until the Phase 9 cutover gate. The card shows the gate, the count and the oldest age rather than an empty tab.

The rules this page enforces

R-14.20, R-14.22, R-14.23

The key never changes, and the order cannot be got wrong. Two constraints and one trigger carry it:

CONSTRAINT ck_subs_sequence CHECK (
  enabled = FALSE
  OR (registration_confirmed_at IS NOT NULL
    AND enabled_at >= registration_confirmed_at))

TRIGGER od_key_immutable BEFORE UPDATE
  -- raises if idempotency_key changes

A paused subscriber keeps its backlog. ops2's drain stamps delivered_at on every pending row of a deactivated subscriber and counts them as delivered, so a ten-minute maintenance pause there destroys the queue and the counter says it went fine. That is the second half of ruling L18, and it matters more than the unregistered case because it is the likelier path during the Phase 5 cutover.

Inbound side Outbound budgets
Keyboard J K moveEnter open X selectR replay C claimP pause subscriber 1-5 tab⌘K palette
Connected to Sibling events Event inbox Event detail Rate limits Audit log Alert rules Classifier ops2 events CSD events GRN events RFID Live inventory Settings