Jobs that can run twice
A queue will deliver the same message again. The work should survive that.
Celery, Sidekiq, and every other queue will retry. The worker dies, the broker redelivers, and your function runs a second time with the same arguments. If the second run sends another payment or inserts another row, the bug is in the job, not the queue.
Make the operation idempotent. Give the work a key that already means something in the domain: a distribution run id, an upload id, a webhook event id. The first run records that key. The second run sees it and returns.
A practical shape:
- Write the intent first. Insert a row with a unique key and a status of
started. - Do the side effect.
- Mark the row
finished.
If step 2 fails, the retry finds started and can continue or bail out with a clear error. If step 3 never happens, you can see the stuck row. If the whole job succeeds and is delivered again, the unique key stops a duplicate.
Timestamps are a poor key. "Today" changes, and two workers can agree on it at the same moment. An id the caller already has does not.