Six tools that answer "why is this number this number" without opening a psql session. Read-only by default. Every action is written to the audit log before it executes.
-- compiled from query_object 9f2a1b0d · engine forbids any statement other than SELECT SELECT st.code AS stock_type, le.source_type, SUM(le.quantity_delta) AS qty_delta, COUNT(*) AS entries FROM ledger_entries le JOIN skus s ON s.id = le.sku_id JOIN stock_types st ON st.id = le.stock_type_id WHERE s.sku_code = 'MST-CRM-42-BLK' AND le.location_id IN (1, 3) -- HQ_319, MCC AND le.occurred_at >= '2026-07-01' AND le.id <= 4182993 -- cut id, so a re-run is reproducible GROUP BY st.code, le.source_type ORDER BY st.code, le.source_type;
| Stock type | Source type | Qty delta | Entries | First seen | Last seen |
|---|---|---|---|---|---|
| STOCK | order_sale | -284 | 284 | 01-07 06:12 | 29-07 15:48 |
| STOCK | refund_restock | +41 | 38 | 02-07 11:04 | 28-07 09:22 |
| STOCK | grn_receipt | +420 | 3 | 08-07 07:40 | 24-07 08:02 |
| STOCK | transfer_out | -96 | 8 | 05-07 14:10 | 26-07 13:38 |
| STOCK | transfer_in | +96 | 8 | 06-07 09:22 | 27-07 10:14 |
| STOCK | stock_count_adjust | -6 | 2 | 14-07 18:02 | 28-07 17:44 |
| IN_TRANSIT_CUSTOMER | fulfillment_create | +284 | 284 | 01-07 06:12 | 29-07 15:48 |
| IN_TRANSIT_CUSTOMER | delivery_confirmed | -268 | 268 | 03-07 11:20 | 29-07 12:04 |
| IN_TRANSIT_INTERNAL | transfer_ship | +96 | 8 | 05-07 14:10 | 26-07 13:38 |
| IN_TRANSIT_INTERNAL | transfer_receive | -96 | 8 | 06-07 09:22 | 27-07 10:14 |
| IN_TRANSIT_RETURN | return_carrier_confirmed | +52 | 49 | 02-07 10:44 | 28-07 08:18 |
| QUARANTINE | return_quarantine | +52 | 49 | 02-07 10:44 | 28-07 08:18 |
| QUARANTINE | quarantine_release | -41 | 38 | 02-07 11:04 | 28-07 09:22 |
| GIT_INBOUND | pi_shipped | +420 | 3 | 01-07 04:00 | 18-07 04:00 |
| RECEIVED_NOT_INVOICED | grn_receipt | +420 | 3 | 08-07 07:40 | 24-07 08:02 |
| ON_TRIAL | trial_out | +4 | 4 | 11-07 16:30 | 25-07 12:08 |
| When | Event | Target | Result |
|---|---|---|---|
| 29-07 16:28 | evt_shopify_order_paid_01K1BC7X | staging | 200 |
| 29-07 16:11 | evt_ops2_carton_moved_01K1BAF2 | prod | suppressed |
| 29-07 15:52 | evt_grn_receipt_01K1B9Y0A8 | staging | 200 |
| 29-07 14:14 | evt_rfid_gate_read_01K1B7T0W9 | prod | 429 |
| 28-07 22:03 | evt_csd_case_transition_01K19XZ1 | shadow | 200 |
| 28-07 19:37 | evt_iwms_qc_outcome_01K19R2G | prod | 500 |
| 28-07 11:48 | evt_shopify_refund_01K18WSP00 | prod | 200 |
| Seq | Received | Qty | Outcome | Ledger row |
|---|---|---|---|---|
| 1 | 08-07 07:40 | 108 | posted | L-3,918,204 |
| 2 | 14-07 06:12 | 402 | suppressed | none · dedup to L-3,918,204 |
| 3 | 19-07 08:22 | 388 | suppressed | none · dedup to L-3,918,204 |
| 4 | 24-07 08:02 | 282 | suppressed | none · dedup to L-3,918,204 |
-- incumbent, keyed on the PO LINE: the second tranche is a replay idem_key = f"receipt:{po_code}:{line['id']}" -- i3, keyed on the RECEIPT identity: each GRN is its own event idem_key = f"grn:{grn_number}:{line_id}" -- so the four rows above become four postings: grn:GRN-2026-0708:4812 -> +108 grn:GRN-2026-0714:4812 -> +402 grn:GRN-2026-0719:4812 -> +388 grn:GRN-2026-0724:4812 -> +282
| Journal | Rows | Took | Valid | First bad id | Note |
|---|---|---|---|---|---|
| ledger_entries | 4,182,993 | 3m 09s | yes | - | full chain, genesis to cut |
| ledger_entries · i2 replay | 587,412 | 42s | yes | - | balance_after computed at import, ordered by (occurred_at, id) |
| cost_ledger | 118,402 | 11s | yes | - | value journal, 4 entry kinds |
| unit_ledger | 1,842,118 | 1m 18s | yes | - | EPC serial hops |
| audit_log | 248,904 | 14s | yes | - | infinite retention |
| ledger_entries · corruption test | 1,000 | 0.2s | no | L-742 | deliberately corrupted fixture, detected at the right row |
| cost_ledger · corruption test | 500 | 0.1s | no | C-118 | deliberately corrupted fixture, detected |
| unit_ledger · gap test | 2,000 | 0.3s | no | U-1,204 | a skipped lifecycle hop, detected as a chain break |
-- fixture FIX-2026-07-29-004, row 742 mutated by one unit journal : ledger_entries first_bad_id : 742 prev_hash : 9c1f4a2e7b8d0c35 (matches row 741 row_hash, so the link is intact) expected_hash : "4b7e1c9a2f0d8365" sha256(prev_hash || core fields) actual_hash : "4b7e1c9a2f0d8365" stored value agrees with itself... recomputed : "e2a5f81b6c704d92" ...but not with the row's own columns mutated_column : quantity_delta -2 -> -3 verdict : row_hash was NOT recomputed after the mutation, so the tamper is visible. Rows 743+ still chain to 742's stored hash, which is why the FIRST bad id is the useful answer.
| Ledger id | Occurred | Source type | Reference | Delta | Running | balance_after | Cache | Agree | Note |
|---|---|---|---|---|---|---|---|---|---|
| L-4,178,204 | 22-07 09:12 | opening_carry | close 2026-06 | - | 61 | 61 | 61 | yes | carried from the frozen June close |
| L-4,178,441 | 22-07 11:04 | order_sale | #MIN10402 line 1 | -2 | 59 | 59 | 59 | yes | - |
| L-4,178,902 | 23-07 08:22 | order_sale | #MIN10411 line 2 | -1 | 58 | 58 | 58 | yes | - |
| L-4,179,118 | 23-07 14:38 | transfer_receive | TR-0921 HQ_319 | +12 | 70 | 70 | 70 | yes | IN half of a movement pair |
| L-4,179,402 | 24-07 10:14 | order_sale | #MIN10428 line 1 | -3 | 67 | 67 | 67 | yes | - |
| L-4,179,688 | 24-07 16:02 | display_allocate | MCC window | -4 | 63 | 63 | 63 | yes | paired DISPLAY_GOOD +4 |
| L-4,179,912 | 25-07 09:44 | trial_out | ON_TRIAL · staff | -1 | 62 | 62 | 62 | yes | ON_TRIAL is owned, not sellable |
| L-4,180,204 | 25-07 15:20 | refund_restock | CSD-4802 line 1 | +2 | 64 | 64 | 64 | yes | out of QUARANTINE after a physical scan |
| L-4,180,441 | 26-07 08:18 | order_sale | #MIN10444 line 3 | -6 | 58 | 58 | 58 | yes | - |
| L-4,180,602 | 26-07 13:38 | transfer_ship | TR-0937 to YAS | -8 | 50 | 50 | 50 | yes | OUT half, pair id MP-2841 |
| L-4,180,744 | 27-07 10:02 | order_sale | #MIN10461 line 1 | -4 | 46 | 46 | 46 | yes | - |
| L-4,180,882 | 27-07 17:44 | stock_count_adjust | CNT-0188 MCC | -2 | 44 | 44 | 42 | no | cache double-applied the variance. First divergent row |
| L-4,181,204 | 28-07 09:22 | refund_restock | CSD-4821 line 3 | +1 | 45 | 45 | 43 | no | offset persists, exactly 2 |
| L-4,181,602 | 28-07 14:10 | order_sale | #MIN10474 line 2 | -1 | 44 | 44 | 42 | no | offset persists |
| L-4,182,118 | 29-07 11:48 | order_sale | #MIN10481 line 1 | -2 | 42 | 42 | 40 | no | offset persists |
| L-4,182,904 | 29-07 15:48 | display_return | MCC window rotate | +2 | 44 | 44 | 42 | no | current state · delta -2 |
The incumbent backbone has no running-balance column at all. A grep for balance_after across every migration and every Python file in i2 returns zero occurrences. The cache did not drift away from a check that existed and was skipped: there was never anything to check against.
So balance_after is a new invariant in i3, not a restored one. Nobody has ever seen it work, and a column nobody trusts is a column nobody uses. This tool exists to make it inspectable: pick any key, see every row that built it, and watch the two numbers agree or not.
It also fixes nothing, deliberately. An unreconciled key drops out of every total and stays out until a compensating pair closes it. A console that could quietly write the cache back into agreement would recreate the exact failure it is here to detect.
| Batch | Scenario | Target | Events | Seed | State | What it proves |
|---|---|---|---|---|---|---|
| FIX-2026-07-29-004 | composite_sale_then_partial_refund | staging | 24 | 88104 | live | refund replays the pinned recipe, not the live one |
| FIX-2026-07-29-003 | split_delivery_four_tranches | staging | 8 | 41022 | live | all four tranches post, none deduplicated away |
| FIX-2026-07-29-002 | clean_count_no_variance | staging | 12 | 70118 | live | a clean count writes a verification, never a zero-delta row |
| FIX-2026-07-28-006 | cost_revision_after_close | shadow | 6 | 10488 | reverted | frozen period unchanged, adjustment lands in the open one |
| FIX-2026-07-28-005 | transfer_half_pair_orphan | staging | 4 | 22841 | reverted | a half-move cannot commit |
| FIX-2026-07-28-004 | exchange_pair_size_swap | staging | 10 | 64012 | reverted | one anchor key, two legs, no double deduction |
| FIX-2026-07-28-003 | sku_rename_orphans_tags | staging | 18 | 33907 | reverted | unit ledger survives a rename because it keys on sku_id |
| FIX-2026-07-27-002 | chain_corruption_single_row | staging | 1000 | 74210 | reverted | verifier names the right first bad id |
| FIX-2026-07-27-001 | balance_cache_double_apply | staging | 16 | 18088 | reverted | divergent key is excluded, not repaired |
| FIX-2026-07-26-003 | rni_ownership_monotonic | shadow | 9 | 50204 | reverted | owned_qty does not dip when a pallet lands |
| FIX-2026-07-26-002 | carrier_status_cannot_credit | shadow | 7 | 91104 | reverted | a returned string moves units to QUARANTINE, never STOCK |
| FIX-2026-07-26-001 | composite_parent_never_holds | staging | 14 | 42188 | reverted | balance(composite) = 0 at every location and type |
| Header | Rendered as |
|---|---|
| Authorization | Bearer <SSO session> |
| X-I3-Signature | <computed server-side> |
| I3_WEBHOOK_SECRET | [see ~/.env.master] |
| SSO_JWT_SECRET | [see ~/.env.master] |
A masked value with a real prefix and a real last-four is still a disclosure: it confirms the key family, the environment and enough of the tail to correlate against a leaked log. The secret registry has no value column, so there is nothing here to leak by accident.
| When | Actor | Action | Target | Latency | Read | Wrote | Result |
|---|---|---|---|---|---|---|---|
| 16:42:18 | ahmad@minimalist.ae | ledger_query | ledger_entries · MST-CRM-42-BLK | 214ms | 16 | 0 | ok |
| 16:41:04 | ahmad@minimalist.ae | projection_debug | MST-CRM-42-BLK / MCC / STOCK | 88ms | 16 | 0 | divergent |
| 16:28:52 | ahmad@minimalist.ae | event_replay | evt_shopify_order_paid_01K1BC7X to staging | 164ms | 1 | 2 | 200 |
| 16:22:04 | i3-worker@system | chain_verify | ledger_entries 1 to 4,182,993 | 3m 09s | 4.18M | 0 | valid |
| 16:11:18 | ahmad@minimalist.ae | event_replay | evt_ops2_carton_moved_01K1BAF2 to prod | 98ms | 1 | 0 | suppressed |
| 15:58:30 | fatima@minimalist.ae | idem_lookup | inv1 · receipt:PO-2026-118:4812 | 42ms | 4 | 0 | ok |
| 15:52:11 | ahmad@minimalist.ae | fixture_generate | FIX-2026-07-29-004 to staging | 1.2s | 0 | 24 | ok |
| 15:14:44 | omar@minimalist.ae | ledger_query | close_lines · 2026-07 residual | 308ms | 88 | 0 | ok |
| 14:14:40 | ahmad@minimalist.ae | event_replay | evt_rfid_gate_read_01K1B7T0W9 to prod | 78ms | 1 | 0 | 429 |
| 13:51:09 | omar@minimalist.ae | flag_flip | RFID_UNIT_LEDGER shadow requested authoritative | - | 0 | 0 | blocked · gate |
| 12:38:02 | fatima@minimalist.ae | projection_debug | MIN-2159 / HQ_319 / STOCK | 104ms | 42 | 0 | agrees |
| 11:04:22 | ahmad@minimalist.ae | fixture_revert | FIX-2026-07-28-006 · 6 compensating rows | 640ms | 6 | 6 | ok |
| 09:22:14 | i3-worker@system | chain_verify | cost_ledger 1 to 118,402 | 11s | 118K | 0 | valid |
| 08:14:51 | rania@minimalist.ae | ledger_query | balance_verifications · MCC last 30d | 62ms | 28 | 0 | ok |
You have opened the console but run nothing. Every tool here is read-only until you request write access, so there is no risk in starting: pick a SKU and location in the projection debugger, or paste an event id into replay. The activity log will fill from your first query.
POST /api/dev/ledger-query returned 503. The console reads the replica by design and will not fall back to the primary, because an unbounded exploratory query against the write path is how a debugging session becomes an incident. Retry, or narrow the range and try again in a minute.
The console needs role = admin plus the dev_console feature. You have ops_manager, so the projection debugger and the idempotency lookup are available read-only and every other tool is hidden. Those two are safe for any operator because neither can write and both only explain numbers you can already see. Role mapping →
Postgres minor upgrade, 02:00 to 02:40 GST. Read tools stay available against the replica. Replay and fixture generation are paused because an event landing mid-failover could commit its ledger row and lose its outbox row, which is the split-write i3 exists to prevent.
The unit_ledger entity in the query console and the serial branch of the projection debugger need GATE-6 to have held for 7 days. It has held 0. Both are still listed and still queryable in shadow, and every result carries a not authoritative stamp, so you can watch the unit and quantity ledgers converge instead of waiting blind. Gate tracker →