Payment operations and recovery
Use Payment operations for scoped alerts and unresolved initiation. A payment's operational trace connects its operation, attempts, provider reference and normalized observations; the delivery panel connects signed merchant events and HTTP attempts. Raw provider payloads, credentials and customer contact details are excluded from diagnostics.
Alert delivery and scope
The dashboard evaluates thresholds when refreshed. An authorized external monitor can poll GET /v1/internal/payments/health with the deployment's monitoring token. HTTP 503 means active alerts or unavailable diagnostics; inspect the response before attributing the cause. The endpoint itself does not send email, page an on-call team or measure availability. Configure the monitor and escalation recipient for each deployment.
Rates measure attempts, not unique merchant orders or causal uplift. Retries can contribute multiple attempts. Initiation and final outcomes use separate denominators. Unresolved initiations and webhook backlog include old work; completion alerts use a 24-hour creation cohort. Use payment history and delivery records alongside alerts to follow up on individual payments.
Unresolved initiation
Threshold: At least one keyed operation remains executing or unknown after 15 minutes.
Open Payment Operations, then the operation. Inspect the existing provider reference and recovery reason. Use Resolve with provider to query the original attempt. Keep the original reservation and idempotency key; never initiate another collection to clear uncertainty. Escalate unsupported or exhausted recovery to an administrator with a provider evidence reference.
Unresolved completion
Threshold: At least one accepted attempt is pending or unknown after 15 minutes in the last 24-hour creation cohort.
Open the payment operational trace, verify with its provider and review callback or query observations. Successful initiation does not prove payment completion. Older unresolved completion and conflicts remain in the existing investigation workflow; these cohort alerts do not replace that workflow.
Provider-interface failures
Threshold: At least five timeout or provider-down failures, representing at least 20% of at least 20 settled initiation attempts in a 24-hour creation cohort.
Inspect the affected app, provider credentials, eligibility and provider status. A timeout does not prove that the provider declined or did not accept the collection. Multiple connectors may share a card network, bank or mobile-money rail; switching connectors is not independent-rail redundancy.
Unclassified initiation failures
Threshold: At least five unclassified failures, representing at least 20% of at least 20 settled initiations in 24 hours.
Investigate normalized attempt codes and operation evidence before attributing responsibility. Validation errors and insufficient funds are excluded from this infrastructure alert. Unknown codes are not labelled as provider or Tinker Payments failures.
Verified final failures
Threshold: At least five failed, expired or cancelled results, representing at least 20% of at least 20 verified terminal outcomes in a 24-hour creation cohort. Conflicted results are excluded.
Review the payment timeline and provider evidence. These are final payment outcomes, which can include customer or issuer decisions; they do not directly measure provider availability or Tinker Payments uptime.
Merchant webhook backlog
Threshold: At least one queued event lacks a durable acknowledgement after 15 minutes, across all event sources and creation dates.
Open payment delivery evidence and inspect the event ID, last result and next eligible retry. Restore the recipient endpoint or delivery worker. The existing worker retries retained events; a restart recovers expired leases. Do not delete the queue or re-create events. A 2xx delivery log without an outbox acknowledgement can be delivered again.
Workers disabled
Threshold: A required worker is disabled by deployment configuration. Visible to administrators only.
Check TP_MANAGED_WORKERS_ENABLED and TP_PAYMENT_RECOVERY_WORKERS_ENABLED. Restore the intended worker configuration and restart the process. Read retained work and the next successful heartbeat before closing the incident.
Worker freshness
Threshold: No successful worker pass in five minutes, including startup before the first heartbeat. Visible to administrators only.
Inspect process health, database access and queue progress. A long-running or deferred pass requires investigation; this threshold does not identify the failing dependency. Restore the worker and confirm a fresh successful heartbeat.
Worker pass deferred
Threshold: The latest worker pass deferred work. Visible to administrators only.
Correlate the worker with per-payment and delivery evidence. Recipient outages, provider lookup failures and local failures have different remedies. The pass alert alone does not attribute fault to Tinker Payments or a provider.
Routing rollback
Use Routing Control to roll back the affected app's rollout with a specific reason. Confirm rolled_back, zero treatment traffic and the audit event. New traffic uses the existing app route; accepted sessions and in-flight payment evidence remain intact. Repeated or concurrent actions must reload the latest state. Never roll back by editing historical payment outcomes.
Delivery and restart semantics
Core payment outcome events and managed billing outbox events survive worker restarts. Claims lease for three minutes. Durable retries become eligible after 2, 4, 8 minutes and continue exponentially to a maximum delay of 1,024 minutes; the worker polls approximately once per minute. Processing or an outage can delay this further. There is no finite attempt cap or automatic expiry for this outbox.
Retries retain the event ID, signed payload and queued destination. Receivers must verify the signature in the JSON security field and atomically deduplicate event IDs with business effects. A recipient that accepts an event before the sender loses its acknowledgement can receive that event again. Events may arrive out of order. Existing historical goroutine deliveries are not backfilled into durable events; other legacy event publishers can retain their own bounded retry behavior.
Configured destination or signing-secret changes affect newly queued events. Existing events retain their original destination and signature; coordinate receiver key overlap and endpoint migrations. A malformed payload or invalid destination remains queued for investigation. Updating a merchant setting does not silently redirect retained financial events.
Business-hours support and incident handling
Support operates during business hours. Each support agreement defines the timezone, hours, contacts, acknowledgement and update targets. There is no blanket 24/7 response or uptime guarantee on this page.
- Critical: suspected duplicate collection, incorrect financial attribution, credential exposure or broad inability to collect. Contain unsafe execution, preserve evidence and escalate through your agreed urgent contact.
- High: persistent unknown outcomes, material provider-interface failures or aged webhook backlog. Investigate the affected scope, recover retained work and use the agreed update cadence.
- Routine: isolated noncritical integration or reporting questions. Use business-hours support and your agreed response target.
Share app, operation, payment, attempt and event IDs, UTC timestamps and normalized error codes. Keep credentials, full provider payloads and customer contact details out of incident messages.
Reliability measurement boundaries
Measure API availability, initiation acceptance, provider-confirmed completion, unresolved-age distribution, event acknowledgement latency and worker freshness separately. Define environment, probe, denominator, time window and exclusions in the support agreement before publishing a percentage. Use your agreed monitoring setup to track these measures consistently.