> ## Documentation Index
> Fetch the complete documentation index at: https://docs.skinloop.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Safe fulfillment

> Deliver each completed order once across retries, duplicates, and crashes.

## The required guarantees

Skinloop delivers webhooks at least once, and status polling can observe the
same completed checkout concurrently. Never call an external fulfillment
service directly from either handler.

Use two independent safeguards:

1. Store one durable fulfillment job for each checkout in your database.
2. Send the same stable idempotency key to the fulfillment destination on every
   attempt.

The database constraint prevents polling and webhook workers from creating
different jobs. The destination's idempotency support prevents a retry after an
ambiguous response from delivering twice.

<Warning>
  Exactly-once external delivery cannot be guaranteed if the fulfillment
  destination does not support idempotency or lookup by a stable key. After an
  ambiguous timeout, stop automatic retries and reconcile manually rather than
  risk delivering twice.
</Warning>

## Store a durable job

Adapt this PostgreSQL schema to your order system:

```sql theme={null}
CREATE TABLE fulfillment_jobs (
  id BIGSERIAL PRIMARY KEY,
  checkout_id TEXT NOT NULL,
  merchant_order_id TEXT NOT NULL,
  idempotency_key TEXT NOT NULL,
  status TEXT NOT NULL CHECK (
    status IN ('queued', 'processing', 'succeeded', 'needs_reconciliation')
  ),
  attempt_count INTEGER NOT NULL DEFAULT 0,
  lease_token UUID,
  lease_expires_at TIMESTAMPTZ,
  destination_reference TEXT,
  last_error_code TEXT,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  UNIQUE (checkout_id),
  UNIQUE (merchant_order_id),
  UNIQUE (idempotency_key)
);
```

Use a deterministic key that remains unchanged across webhook deliveries,
polling, worker restarts, and deployments:

```ts theme={null}
function fulfillmentKey(merchantOrderId: string, checkoutId: string) {
  return `skinloop:fulfillment:v1:${merchantOrderId}:${checkoutId}`;
}
```

Do not use a webhook event ID as the fulfillment key. Different events or a
status poll can report the same completed checkout.

## Use one enqueue operation

Both the signed webhook handler and status poller must call this operation. It
records authoritative completion and creates the job in one local transaction:

```ts theme={null}
type CompletedCheckout = {
  checkoutId: string;
  merchantOrderId: string;
};

async function recordCompletionAndEnqueue(input: CompletedCheckout) {
  const key = fulfillmentKey(input.merchantOrderId, input.checkoutId);

  await db.transaction(async (tx) => {
    const order = await tx.oneOrNone(
      `UPDATE orders
          SET payment_status = 'completed',
              checkout_id = $2,
              updated_at = now()
        WHERE merchant_order_id = $1
          AND (checkout_id IS NULL OR checkout_id = $2)
      RETURNING id`,
      [input.merchantOrderId, input.checkoutId]
    );
    if (!order) throw new Error("checkout does not match the merchant order");

    await tx.execute(
      `INSERT INTO fulfillment_jobs
         (checkout_id, merchant_order_id, idempotency_key, status)
       VALUES ($1, $2, $3, 'queued')
       ON CONFLICT DO NOTHING`,
      [input.checkoutId, input.merchantOrderId, key]
    );

    const job = await tx.one(
      `SELECT checkout_id, merchant_order_id, idempotency_key
         FROM fulfillment_jobs
        WHERE checkout_id = $1 OR merchant_order_id = $2
        FOR UPDATE`,
      [input.checkoutId, input.merchantOrderId]
    );
    if (
      job.checkout_id !== input.checkoutId ||
      job.merchant_order_id !== input.merchantOrderId ||
      job.idempotency_key !== key
    ) {
      throw new Error("fulfillment identity conflict");
    }
  });
}
```

Validate that the completed checkout belongs to the expected order, amount, and
currency before enqueueing. The unique constraints make repeated webhook
deliveries and concurrent polling harmless.

## Claim work with a lease

A worker claims one job in a short database transaction, then commits before
making the external request:

```ts theme={null}
async function claimFulfillmentJob() {
  const leaseToken = crypto.randomUUID();
  return db.transaction(async (tx) => tx.oneOrNone(
    `WITH candidate AS (
       SELECT id
         FROM fulfillment_jobs
        WHERE status = 'queued'
           OR (status = 'processing' AND lease_expires_at < now())
        ORDER BY created_at
        FOR UPDATE SKIP LOCKED
        LIMIT 1
     )
     UPDATE fulfillment_jobs AS job
        SET status = 'processing',
            attempt_count = attempt_count + 1,
            lease_token = $1,
            lease_expires_at = now() + interval '2 minutes',
            updated_at = now()
       FROM candidate
      WHERE job.id = candidate.id
    RETURNING job.*`,
    [leaseToken]
  ));
}
```

The lease allows another worker to recover a job if the current worker crashes
before finishing. Size the lease above the normal fulfillment request timeout
and renew it for longer operations. The unique `lease_token` prevents a stale
worker from acknowledging a job after another worker has reclaimed it.

## Deliver outside the transaction

Send the stored key to a destination that guarantees repeated requests with
that key return the original delivery result:

```ts theme={null}
async function processFulfillmentJob(job: FulfillmentJob) {
  try {
    const delivery = await fulfillmentProvider.deliver({
      merchantOrderId: job.merchant_order_id,
      idempotencyKey: job.idempotency_key
    });

    await db.transaction(async (tx) => {
      const acknowledged = await tx.oneOrNone(
        `UPDATE fulfillment_jobs
            SET status = 'succeeded',
                destination_reference = $2,
                lease_token = NULL,
                lease_expires_at = NULL,
                updated_at = now()
          WHERE id = $1
            AND status = 'processing'
            AND lease_token = $3
      RETURNING id`,
        [job.id, delivery.reference, job.lease_token]
      );
      if (!acknowledged) return;

      await tx.execute(
        `UPDATE orders
            SET fulfillment_status = 'fulfilled',
                updated_at = now()
          WHERE merchant_order_id = $1`,
        [job.merchant_order_id]
      );
    });
  } catch (error) {
    await handleFulfillmentFailure(job, error);
  }
}
```

Never hold a database transaction open during the network request.

## Recover every crash point

| Failure point                                    | Safe recovery                                                                                 |
| ------------------------------------------------ | --------------------------------------------------------------------------------------------- |
| Before the enqueue transaction commits           | Skinloop retries the webhook or the status poller observes completion again                   |
| After enqueue, before a worker claims the job    | The committed `queued` job remains available                                                  |
| After claim, before external delivery            | The lease expires and another worker retries with the same key                                |
| After delivery, before local success is recorded | Retry or query the destination with the same key; it must return the original delivery        |
| Network timeout with unknown delivery result     | Query by the same key; if unsupported, set `needs_reconciliation` and stop automatic delivery |
| Duplicate webhook or simultaneous status poll    | The unique `checkout_id` constraint returns the existing job                                  |

For an ambiguous result without destination idempotency, mark the job for
reconciliation:

```sql theme={null}
UPDATE fulfillment_jobs
   SET status = 'needs_reconciliation',
       lease_token = NULL,
       lease_expires_at = NULL,
       last_error_code = 'ambiguous_delivery_result',
       updated_at = now()
 WHERE id = $1 AND lease_token = $2;
```

An operator must then check the destination before deciding whether to mark the
job succeeded or return it to `queued`.

## Production checklist

* Verify Skinloop signature and timestamp before recording a webhook.
* Deduplicate webhook storage by event `id`.
* Re-read or validate authoritative completed status before enqueueing.
* Validate merchant order ID, checkout ID, amount, and currency.
* Enforce unique checkout, order, and fulfillment keys in the database.
* Commit the job before making any external request.
* Use bounded timeouts and leases.
* Require the matching lease token for success and failure updates.
* Reuse the same fulfillment key forever for that checkout.
* Alert on `needs_reconciliation` and expired processing leases.
* Record the destination reference before marking the merchant order fulfilled.

**Expected result:** webhook duplicates, concurrent polling, worker restarts, and
crashes do not create a second delivery. Ambiguous results are either resolved
through the destination's idempotency key or stopped for manual reconciliation.