Ops health

Wednesday · 29 July 2026 · 16:42 GST · i3-minimalist FRA · poll 10s · uptime 14d 03h

Live Phase 1 of 9 · next gate GATE-1 day 6/7 Ingestion 7/9 healthy · 2 never seen DLQ 41 across 3 topics Error budget worst 3.24x · drift resolution Refresh R · Jump gate G
GATE-6 streak reset yesterday. The 3-way reconcile metric returned NULL (RFID collector down 04:00-06:40), and a NULL metric counts as a failing day, never a skipped one. Streak went 4 -> 0. Best streak so far 4. 3-way reconcile →

Phase-gate tracker

A gate passes only when every blocking metric passes on the same day, seven consecutive days. One failing day resets the streak to zero.

pass fail no data = fail not yet measuring
Gate Blocking metrics · measured vs threshold Last 7 days Streak Best State Blocks
GATE-0
Opening balance
signoff_coverage 10/10 (=10) · chain_valid true · shopify_total_gap 0.62% (≤1.00)
1818 passed Phase 1 open
GATE-1
Sales and cancels
shopify_divergence 0.04% (≤0.10) · classifier_match 99.7% (≥99.5) · unexplained_negatives 0 (=0)
66 holding Phase 2 · unlocks 30-07
GATE-2
Reverse logistics
reverse_drift 0.41% (≤0.20) · unmatched_returns_scans 17 (=0) · replacement_claims_paired 100%
01 failing Phase 3 blocked
GATE-3
Inbound and cost
grn_post_latency_p95 - (≤300s) · po_expected_vs_received - (≤0.5%) · cost_confidence_layered no data (≥95%)
00 not started Valuation blocked
GATE-4
Transfers and retail
transit_nets_to_zero - (=0) · store_count_gap - (≤0.5%) · half_pairs_open - (=0)
00 not started Phase 5
GATE-5
IWMS physical
carton_sum_vs_ledger - (≤0.5%) · opened_carton_double_count - (=0) · unregistered_event_types - (=0)
00 not started Phase 6
GATE-6
RFID unit ledger
unit_vs_qty_HQ_319 NULL (=0) · gate_catch_rate 99.2% (≥99.0) · sample_size 0 (>0)
04 regressed Serial reports
GATE-7
Programs
influencer_pool_match no source (=0) · damage_incident_coverage - (=100%)
00 not started Phase 8
GATE-8
Close reproducible
close_residual - (=0) · pack_hash_stable - (=true) · netsuite_unexplained - (≤tolerance)
00 not started Phase 9 cutover
A NULL metric is a failing day, not a skipped one, and a metric computed over zero rows never counts as a pass. Both rules exist because a broken collector must not manufacture a green streak.

Ingestion lag by source

Silence is never read as zero. A source past its critical interval raises an alert and stamps every dependent report as stale.

5 ok2 warn2 blocked
SourceTransport Last event Lag Warn Crit Events 1h Rejected DLQ Contract Owner
shopifywebhook16:41:528s120s600s1,28400liveplatform@minimalist.ae
ops2 · inboundwebhook16:39:043m 03s300s900s312412liveops@minimalist.ae
ops2 · reversewebhook--300s900s000not emittableops2 PR pending
csdwebhook16:28:1113m 56s600s1800s4613livecs@minimalist.ae
grnwebhook15:12:401h 29m3600s14400s000livewh@minimalist.ae
iwmswebhook16:35:226m 45s600s1800s8829shadowwh@minimalist.ae
rfid · gateswebhook16:41:1255s180s600s2,103717shadow · G2 downplatform@minimalist.ae
rfid · handheldwebhook14:02:382h 39m1800s7200s000staleretail@minimalist.ae
influencer hubwebhooknevernever seen3600s86400s000plannedmarketing@minimalist.ae
sku inteln8n pull04:05:0012h 37m90000s172800s000live · dailyplatform@minimalist.ae
netsuite extractmanual01-07 09:4028dmonthly45d000file dropfinance@minimalist.ae
shopify · levelspoll 15m16:30:0012m1800s3600s100live · cross-check onlyplatform@minimalist.ae
ops2 reverse is registered but not emittable: iwms/webhooks.py::ALL_EVENT_TYPES carries six inbound-warehouse topics and none of the reverse ones. The subscription is registered first, deliberately, so the first hour of events is not lost when the topic is switched on.

Queue depth · last 24h

sampled 5 min · 288 points · peak 14:32
webhook inbox 847 worker default 28 worker high 2 DLQ 41
1,200 pending 800 pending 400 pending 0 pending Webhook inbox pending, peak 1,248 at 14:32 Worker default queue Worker high-priority queue, near zero all day DLQ depth, monotonically rising: nothing is draining it Peak 1,248 at 14:32, Shopify bulk order edit
16:4212:0006:0000:0018:00now

Dead letter queue

41 across 3 topics · oldest 3d
TopicDepthOldestTop error
rfid.gate_read173dEPC not encoded, unknown sku_idReplay
ops2.carton_moved1218hlocation code not in registryReplay
csd.case_transition96hmissing line_id on reservationReplay
iwms.qc_outcome32hbucket value outside the 5-bucket setReplay
shopify.orders_edited0--clear
grn.receipt_line0--clear
A DLQ row is never dropped on age. Replay is idempotent, so a duplicate replay reports "suppressed" rather than double-posting.

Services

8 monitored, including RFID hardware, because a dead gate silently stops dispatch verification.

2 degraded
Web · Flask
i3-minimalist · FRA · perf-2x
Healthy
Machines2 / 2
RPS48.3
p95 latency142 ms
Memory1.8 / 4.0 GB
Postgres
i3-minimalist-db · 16.3 · FRA
Pool high
Pool used14 / 20
DB size22.7 GB
Ledger rows4,182,993
Slowest p95812 ms
Worker group
2 machines · default + high
Healthy
Active3
Queued28
Failed 1h1
p95 job2.4 s
Webhook inbox
9 sources · HMAC verified
Lag 4m
Pending847
Oldest4m 12s
Classified/s3.8
Rejected 1h14
Fanout outbox
5 destinations · HMAC signed
Healthy
Pending6
Delivered 1h2,340
DLQ41
p95 delivery218 ms
Tigris S3
i3-artifacts · 84 GB
Healthy
Report artifacts3,182
Errors 24h0
p95 GET84 ms
Lifecycle30-day
RFID gate G2
FXR90 · HQ_319 exit door
Offline 2h
Last heartbeat14:38:02
Reads today0
Dispatch verifiedunverified since 14:38
ImpactGATE-6 NULL
RFID gate G1
FXR90 · HQ_319 dock
Healthy
Last heartbeat16:41:12
Reads today2,103
Avg read47 ms
Antenna health4 / 4

Service-level objectives

Two blocks. Machine health, then reconciliation health. A green machine can still be completely wrong.

30-day windows
Machine
Web availability
99.96%
target ≥ 99.9%
burn0.12x
API p95
142 ms
target ≤ 250 ms
burn0.34x
Inbox lag
4m 12s
target ≤ 2m
burn1.82x
Worker success
99.3%
target ≥ 99%
burn0.61x
Cron on-time
99.4%
target ≥ 98%
burn0.29x
Reconciliation · i3-specific
Balance keys reconciled
99.98%
target ≥ 99.95%
4 keys open0.40x
Drift resolved < 24h
87.3%
target ≥ 95%
burn3.24x
Qty vs value agreement
100.00%
target = 100%
cost ledger0.00x
Unit vs qty agreement
no data
target = 100%
collector downfails day
NetSuite period delta
AED 18.4K
tolerance AED 12.0K
unexplained1.53x

Recurring jobs

Every job in the estate, whoever runs it. A job with no owner cannot be registered.

JobRunnerSchedule Last runTook StatusNext CoversPages
valuation.nightlyi3 worker15 2 * * *02:15:044m 12soktomorrowStable historical valuation#n8n-updatesLogs
costmodel.reconcilei3 worker45 2 * * *02:45:112m 38soktomorrowcost_ledger vs cost read models#n8n-updatesLogs
gates.evaluatei3 worker0 3 * * *03:00:0258spartialtomorrow7-day gate streaks#n8n-updatesLogs
balance.full_verifyi3 worker30 3 * * *03:30:006m 51soktomorrowBalance cache vs balance_after#n8n-updatesLogs
chain.verifyi3 worker0 4 * * *04:00:013m 09soktomorrowHash chain, 4 journals#n8n-updatesLogs
shopify.divergencei3 worker0 5 * * *05:00:001m 44soktomorrowGATE-1 metric#n8n-updatesLogs
rfid.three_wayi3 worker30 5 * * *05:30:00-failedtomorrowGATE-6 metric#n8n-updatesLogs
netsuite.reconn8n0 6 * * *06:00:0022sskippedtomorrowPeriod delta, 4 bucketsfinance@Logs
cash.position_dailyi3 worker30 6 * * *06:30:0751soktomorrowCash tied up by stage#n8n-updatesLogs
reports.daily_packn8n0 7 * * *07:00:001m 18soktomorrowScheduled deliveries#n8n-updatesLogs
secrets.rotation_sweepi3 worker0 8 * * 1Mon 08:004s1 overdueMonOverdue secret detectionplatform@Logs
dlq.age_reporti3 worker0 9 * * *09:00:037soktomorrowDLQ aging, never auto-dropsplatform@Logs
velocity.recomputei3 worker0 1 * * *01:00:002m 02soktomorrowCover with NULL, never 999#n8n-updatesLogs
ops2.orphan_drainn8n*/30 * * * *16:30:0014sok17:00Half-pair transfersops@Logs
influencer.pool_syncn8n0 10 * * *never-never ran10:00GATE-7 metricunownedLogs
shrinkage.weeklyn8n0 8 * * 0Sun 08:0033spausedpausedAwaiting GATE-6finance@Logs
close.propose_dryruni3 worker0 23 28-31 * *30-06 23:003m 40sresidual31-07Close residual must be zerofinance@Logs
fixture.staging_seedi3 worker0 4 * * 1Mon 04:0018spausedpausedStaging only, never prodplatform@Logs
1 unowned influencer.pool_sync has never run and has no owner. An unowned job is an outage with no assignee, so it blocks GATE-7 rather than quietly reporting zero.

Recent errors

last 6 · click to expand · retained 30d
16:41:23rfid-gateHeartbeatMissing: FXR90 G2 silent 2h 03m, dispatch verification unavailableexpand
HeartbeatMissing at i3/ops/collect.py:212
  device: FXR90-G2 · location: HQ_319 · antenna_count: 4
  last_heartbeat: 2026-07-29T14:38:02+04:00
  expected_interval_s: 300 · observed_silence_s: 7401
  consequence: gate_metrics(GATE-6, unit_vs_qty_HQ_319) -> NULL
  rule: a NULL gate metric counts as a FAILING day (R-15.20)
  correlation_id: 01K1B7Q4ZC9MDF3P2A6TXHY8WR
16:38:02workerJobTimeout: rfid.three_way exceeded 180s waiting on unit ledger snapshotexpand
JobTimeoutException: task exceeded 180s
  job_code: rfid.three_way · queue: default
  blocked_on: SELECT ... FROM unit_ledger WHERE location_id = 1 AND state <> 'SOLD'
  rows_scanned: 139,561 · no index on (location_id, state)
  retries: 0 (timeouts are not retried)
  fix: partial index, and raise max_duration_s to 600
16:24:55classifierValidationError: shopify/orders/create missing line_items[2].variant_idexpand
ValidationError at /webhooks/shopify/orders/create
  payload.id: 6234712394876 · order_name: #MIN10482
  missing: line_items[2].variant_id (product deleted after order)
  classifier_rule: shopify_order_v2 · confidence: n/a
  routed: dlq:shopify:orders · requeue in 15m
  note: keyed on variant.id per the SKU-rename rule, so no SKU fallback is attempted
15:58:11outboxDeliveryBounce: reports.daily_pack to finance@minimalist.ae (554 mailbox full), attempt 3 of 3expand
DeliveryBounce at i3/reports/deliver.py:88
  run_id: 41207 · report_code: valuation_by_location · format: pdf (218 KB)
  target: finance@minimalist.ae · consecutive_bounces: 3
  action: subscription paused, alert.raise emitted at P2
  rule: three bounces pauses the subscription; it is never silently retried forever
14:12:40closeCloseResidual: July dry-run left 41 units uncategorised across 2 source typesexpand
CloseResidual at i3/finance/close.py:164
  period: 2026-07 · location: HQ_319 · residual_qty: 41
  uncategorised source_types: ['consignment_partner_sale', 'rfid_found_unexpected']
  opening 128,904 + categories 4,182 = 133,086, closing = 133,127, delta 41
  rule: a close with uncategorised_qty <> 0 cannot be frozen (R-15.11)
  this is the exact failure INV1 absorbs silently into next month's opening
09:02:18netsuiteExtractMissing: no NetSuite extract loaded for period 2026-07, recon skippedexpand
ExtractMissing at i3/finance/ns_recon.py:41
  period: 2026-07 · last_extract: 2026-06 loaded 01-07 09:40 by finance@
  effect: recon status = 'no_extract'. The period may be FROZEN, not FILED.
  rule: filing without a reconciliation would make i3 the accounting record,
        which ruling L16 forbids.

Why this page has two SLO blocks

The incumbent ops dashboard measures thirteen things and every one of them is a property of the machines: availability, latency, freshness, job success, cron punctuality. Not one of them measures whether the ledger agrees with anything.

For a system whose founding doctrine is that a phase closes only when its ledger agrees with external truth for seven consecutive days, that is the wrong instrumentation. i3 could be perfectly available, perfectly fast, perfectly on schedule, and completely wrong, and every tile would be green.

So the reconciliation block is a peer of the machine block, not a footnote. It is the block that goes red while the hardware is fine.

Gate rule
All blocking metrics pass on the same day
NULL metric
Counts as a failing day
Zero-sample metric
Never counts as a pass
Threshold change
Frozen per check, history never rewritten
Waiver
Named human, reason, expiry. Never a default
Alerting
Emits alert.raise, never Slack directly

Nothing to report

All nine sources inside their intervals, no failing jobs, no open incidents, and every measuring gate holding. This is what a quiet day looks like. It is not the same as no data: the collectors ran at 16:42:07 and returned values.

Could not load platform health

GET /api/ops/health returned 503, last success 4m ago. The gate tracker below is showing the last successful evaluation from 03:00:02 and is labelled stale, not current.

Viewer access

You can read every panel on this page. Run now, Pause and Waive require ops_manager or above. Your derived persona is store manager · MCC, from (role=manager, features=[retail], rota=MCC). Personas are derived at login, never granted.

Scheduled maintenance

Postgres minor upgrade, 02:00 to 02:40 GST. Ingestion continues into the inbox and is classified on resume, so no event is lost. Gate evaluation for tonight is deferred to 03:40 and will still record a value for today.

Reconciliation SLOs are gate-blocked

The reconciliation block needs GATE-1 to have held for 7 days before its numbers mean anything. It has held 6. Until then the tiles show shadow measurements marked not authoritative, rather than being hidden, so you can watch them converge.

Connected to Dev console Settings + flags Reports Cash + valuation Drift Sibling events 3-way reconcile Notifications Audit log Dashboard
R refresh · G gates · J jobs · D DLQ · / filter jobs · ⌘K palette All times Asia/Dubai · every action on this page is written to the audit log before it executes