Rate limits and budgets 2 buckets at risk

Upstream and inbound budgets, current burn, live backoff state, and the priority ladder that decides which work sheds first · window rolling · currency AED

Backing off right now. shopify_rest returned 429 at 14:40:58 with Retry-After: 2. i3 is holding for 1.4s more and will not retry sooner than asked. Two job classes are shed: backfill and cost_pull. ledger_write is rank 1 and still has its reserved 40% floor. Affected fanout →
poll 15s · last refresh 14:41:07

Buckets

Percent of capacity, absolute burn, reset ETA and cost · the tick marks on each bar are that bucket's slowdown, hold and stop thresholds

12 registered
Shopify REST
leaky · 2.00 calls/s
backing off
89%1.78 / 2.00 calls/s
429 · hold 1.4s2,104 calls todayAED 0 · free
80%slow down90%hold95%hold, never drop
Shopify GraphQL
cost · 1,000 pts/min
healthy
42%418 / 1,000 pts/min
safe9,842 pts todayAED 0 · free
80%slow down90%hold95%hold, never drop
Shopify Bulk Ops
concurrency · 1 job
1 running
100%1 / 1 job
ETA 6m 12s3 jobs todayAED 0 · free
-queue only1queue-never parallel
Postgres write pool
concurrency · 25 conn
high
84%21 / 25 conn
shed at 85%p95 wait 42msAED 320 · fixed
70%slow down85%shed rank 5+95%shed rank 3+
ops2 API
fixed window · 60 req/min
healthy
31%19 / 60 req/min
safe612 calls todayAED 0 · internal
75%slow down90%hold100%hold
CSD API
fixed window · 120 req/min
healthy
18%22 / 120 req/min
safe318 calls todayAED 0 · internal
80%slow down90%hold100%hold
RFID app API
leaky · 300 req/min
healthy
24%72 / 300 req/min
safe1,204 calls todayAED 0 · internal
80%slow down92%hold100%hold · gate reads exempt
SKU Intel API
fixed window · 60 req/min
paused
0%0 / 60 req/min
subscriber paused44 calls todayAED 0 · internal
80%slow down90%hold100%hold
GRN app API
fixed window · 30 req/min
idle
7%2 / 30 req/min
safe64 calls todayAED 0 · internal
80%slow down90%hold100%hold
Slack chat.postMessage
leaky · 1 msg/s per channel
healthy
12%0.12 / 1.00 msg/s
safe41 posts todayAED 0 · free
70%slow down85%digest instead95%digest only
Inbound admission
leaky · 400 req/min
healthy
36%144 / 400 req/min
safe1,924 deliveries todayAED 0 · internal
75%defer classify90%defer all classify100%still 202, never 5xx
DHL tracking
fixed window · 100 req/hour
watch
72%72 / 100 req/hour
resets in 18m412 calls todayAED 88 · MTD
75%slow down90%hold100%hold

Priority ladder · which work sheds first

Rank 1 is never paused while any budget remains and holds a reserved floor. This is the piece v2 has no answer for: it shows the burn and never says who loses.

9 job classes
RankJob classWhat it doesReserved floorSheds atShed 24hEffect when shed
1ledger_writePost a classified event to the ledger40%never0never shed · stock truth stops otherwise
2classifyRun the classifier on an accepted event20%98%0events queue at pending, lag rises, nothing lost
2alert_evalEvaluate the 44 live alert conditions5%98%0tick skipped, gap recorded, never silently missed
3fanoutDeliver outbound webhooks to siblings10%95%18next_attempt_at pushed, ladder unaffected
4reconcileNightly balance and chain verification0%92%2window slips, result still recorded as late
5shopify_crosscheckDaily Shopify levels divergence pull0%88%6the day's cross-check is marked incomplete
6backfillHistorical replay from the opening balance date0%85%402pauses and resumes from its cursor, never restarts
7cost_pullLanded cost and unit cost sourcing0%80%744cost stays stale, flagged as of yesterday
8reportReports hub queries and CSV exports0%78%96user sees a queued message with a position
9backtestRule and alert backtests over history0%70%16run parked, resumes when the bucket clears
Total shed 24h: 1,284 calls · zero at rank 1 or 2Shedding defers work; it never discards it. A shed call is a call that will happen later, and every job class states what "later" costs.

Backoff state

Live, with the countdown

1 active
shopify_rest
Trigger
HTTP 429 at 14:40:58
Retry-After
2s, honoured exactly
Hold remaining
1.4s
Occurrence
3rd today
Shed classes
backfill, cost_pull
Protected
ledger_write 40% floor

i3 never retries faster than the upstream asked, and the honoured delay is visible while it is in effect rather than only in a log line afterwards.

Monthly spend · AED
Postgres320fixed
Fly compute · workers486fixed
Tigris payload archive21264%
DHL tracking8882%
Sentry156fixed
Cloudflare92fixed
Total MTDAED 1,35467%

Shopify, ops2, CSD, GRN and the RFID app cost nothing per call; their budgets exist to protect the upstream, not the invoice. Only three buckets carry a real AED cost.

Shopify REST burn · 24 hours

Percent of the 2 calls/s leaky bucket, minute samples · slowdown, hold and stop thresholds drawn as reference lines · shaded bands are when a job class was shed

13:20 to 14:10 · cost_pull shed 14:10 to 14:41 · cost_pull and backfill shed, 3 × 429 stop 95% hold 90% slowdown 80% 14:12 · 429 · Retry-After 1s 14:31 · 429 · Retry-After 2s 14:40 · 429 · Retry-After 2s 14:41 · 89% after shedding two classes
15:0019:0023:0004:0009:0014:41
burn % slowdown 80% hold 90% · stop 95% one class shed two classes shed The rise from 09:00 is the Phase 1 backfill catching up. Shedding it at 14:10 is what pulled the line back under 95%.

Recent throttle decisions

Every deferral and every 429, with the job class that paid for it

TimeBucketJob classRankMethod and pathBurnHTTPRetry-AfterDecision
14:40:58shopify_restcost_pull7GET /admin/api/inventory_items96%4292sheld, honoured
14:40:41shopify_restbackfill6GET /admin/api/orders91%--shed, requeued
14:38:12pg_write_poolreport8SELECT reports.valuation_month86%--queued, position 2
14:36:04shopify_restledger_write1POST /admin/api/inventory_levels/set88%200-passed · floor honoured
14:31:22shopify_restcost_pull7GET /admin/api/inventory_items95%4292sheld, honoured
14:28:47dhl_trackingshopify_crosscheck5GET /track/shipments89%--shed until reset
14:22:10skuintel_apifanout3POST /webhooks/i3/balance0%503-ladder step 4
14:16:33shopify_restbackfill6GET /admin/api/orders87%--shed, requeued
14:12:04shopify_restcost_pull7GET /admin/api/inventory_items94%4291sheld, honoured
14:08:51inbound_admissionclassify2POST /webhooks/shopify/orders/create58%202-accepted, classify queued
13:58:19pg_write_poolbacktest9SELECT classifier.replay_window78%--parked
13:44:07shopify_graphqlshopify_crosscheck5POST /admin/api/graphql · 214 pts44%200-passed
13:31:02slack_postalert_eval2POST chat.postMessage #n8n-updates14%200-passed · module 01
13:12:44rfid_apireconcile4GET /api/rfid/units?location=HQ_31926%200-passed
12:48:16ops2_apireconcile4GET /api/orders?since=2026-07-2832%200-passed
1,284 throttle decisions today · 15 shownRow 4 is the point of the whole page: at 88% burn a rank-1 ledger write still went through, because rank 1 holds a reserved 40% floor no other class can consume.
Loading

Bucket cards keep their heights so the grid does not reflow as each bucket reports.

Empty and zero-filter
All 12 buckets idle.

No sample in the last 5 minutes on any bucket. Either the workers are paused or the sampler is dead, and those need different responses, so the empty state names both. Zero-filter lists the chips.

Error
Bucket state unreachable.

The limiter fails closed: if it cannot read a bucket it assumes the bucket is full and sheds everything below rank 3. A limiter that fails open is how you earn a multi-hour ban.

Permission

Every operator sees burn and spend. Editing a ladder or a reserved floor needs role >= admin plus i3.ratelimit_admin, because lowering rank 1's floor is how you break the ledger quietly.

Maintenance

During the nightly verify, reconcile is temporarily promoted to rank 2 so the chain walk is not shed by a backfill. The promotion is shown here rather than hidden in a config file.

Gate-blocked

The Shopify write-back share of shopify_rest is reserved but unused until the Phase 9 gate. The card shows the reservation so the eventual burn is not a surprise on cutover day.

The rules this page enforces

R-14.26, R-14.27, R-14.28

Three steps, and rank 1 never loses. The ladder is a constraint, not a convention:

CONSTRAINT ck_rlb_ladder CHECK (
  slowdown_pct < hold_pct
  AND hold_pct <= stop_pct)

CONSTRAINT ck_rlp_rank CHECK (
  rank BETWEEN 1 AND 9)

A repo-wide search of i2 for rate limiting, 429 handling, Retry-After, token buckets or the Shopify call-limit header returns nothing. The only matches for the word "bucket" are histogram bins. So i3 is the first system here to have a budget at all, which also means it gets no historical data and needs its own bootstrap week.

Inbound admission is itself a bucket, and it is the one place where the stop action is not a stop: i3 still returns 202 and still persists the raw body, and only defers classification. A 5xx to Shopify becomes a 48-hour retry storm that amplifies the very burst that caused it.

Ingest Fanout
Keyboard R refreshF filter by bucket L edit ladderP priority table B burn chart⌘K palette
Connected to Webhook outbox Event inbox Sibling events Alert rules Audit log Classifier Reports RFID In transit Settings