Workflow automation: manage retries and duplicates

An automated workflow rarely fails while you are watching. It fails in the middle: the request is received, the document prepared, the validation obtained, and then the network drops right before the external write. The question is not "did it work?" but "what actually happened?". This guide addresses that uncertainty: idempotency, queues, intermediate states and human recovery.

Why a timeout tells you nothing

A timeout only means the response did not arrive within the delay. The remote operation may have failed, succeeded without the response reaching you, or still be running. These three cases call for different reactions, yet many automation tools treat them the same way: by replaying the call.

Blindly replaying a send is precisely what you must avoid. If the first write succeeded, you have just created a duplicate: two invoices, two bookings, two messages to the same client. The opposite is just as bad: if you abandon without checking, you lose an operation that may have gone through. The transport connection, the provider's authorization and the proof that the operation occurred are three different things; a timeout only tells you about the first.

That is why designing a reliable workflow starts before the error: with mapping the steps and their side effects. If you start from a poorly bounded process, no recovery mechanism will fix the ambiguity. Our article on business process automation and mapping first details this preparatory work.

Idempotency: the structural answer to duplicates

An idempotent operation produces the same effect whether executed once or five times. This is the property that turns an ambiguous timeout into a manageable problem: instead of asking "did the operation succeed?", you ask "is the expected effect present?".

Concretely, you attach a unique key to each business operation — an order number, a request identifier, a document reference — and the remote tool rejects or recognizes repeats carrying the same key. Some systems support this natively; others require you to do the check yourself before writing. Two levels to distinguish:

• Key-based idempotency: you send the same business identifier on every attempt, and the remote system deduplicates.
• Check before write: before creating, you look for an existing object with that reference, then create or reuse it.

Neither mechanism is universal. Claiming a workflow is "exactly-once" because it uses a queue is a common design mistake: the queue guarantees message delivery, not the unique effect of each external write. What matters is that every side-effecting step has a deduplication key and an explicit verification rule.

Duplicate triggers: the most mundane source of duplicates

Before network failures even enter the picture, the most common duplicate is born from the trigger itself. A form submitted twice, a webhook recalled by the provider, an email handled by two overlapping rules, a file imported twice into a watched folder. The workflow works perfectly — twice.

The remedy is the same as for retries: a stable business key and an existence check at the start of the workflow. A request email with the same sender and subject within a short window, an order number already present in your registry: these signals must short-circuit the execution, not launch a second parallel pass. Our guide on automatic email management and threads shows how to spot these repetitions on the messaging side; the same reasoning applies to any inbound trigger.

Queues and states: giving the workflow a memory

A workflow that holds everything in memory loses everything at the first interruption. The queue sequences operations and makes them replayable; states tell you where you were. Four states are enough in practice for each side-effecting step:

1. Pending: the operation is ready, nothing has been sent.
2. Sent, unconfirmed: the call went out, the response is missing — the uncertainty zone.
3. Confirmed: the remote effect has been verified.
4. Needs recovery: a human or a rule must decide.

The "sent, unconfirmed" transition deserves all your attention. It is what triggers the existence check rather than a new write. A queue without states produces blind retries; states without a queue produce workflows that stop at the first incident. The two go together.

Main example: request received, document prepared, validation, external write

Let's take a hypothetical, deliberately concrete scenario. A services SME receives quote requests by email. The automated workflow has four steps:

Step 1 — Reception. Each request email gets an internal reference, for example QT-2026-0417, derived from the date and a counter. The workflow records the request in a database with the state "received" and this reference. If the same email arrives twice — the trigger was recalled — a lookup by reference and by sender/subject pair within a thirty-minute window (a hypothetical value, to adjust to your volume) blocks the second execution.

Step 2 — Document preparation. A quote template is filled with the request's details. The document carries the reference QT-2026-0417 in its title and in a dedicated field. This step is naturally idempotent: regenerating it with the same data produces the same file, overwritten with no side effect. The state moves to "document prepared".

Step 3 — Human validation. A manager receives a validation request with a link to the document. They can approve, reject or request changes. Until a decision is recorded, the state stays "in validation". If the workflow restarts after an outage at this stage, it does not regenerate the document nor resend the request: it first checks the recorded state.

Step 4 — External write. Once approved, the workflow creates the quote in the remote billing tool. This is where the timeout can strike. The rule is strict: the call goes out with QT-2026-0417 as the deduplication key. On timeout, the workflow moves to "sent, unconfirmed" and runs a search in the remote tool for that reference. Three outcomes:

• The quote already exists: the state moves to "confirmed", the workflow notifies the team, nothing is rewritten.
• The quote does not exist: the write is replayed once with the same key. If the second call also fails, the state moves to "needs recovery".
• The search itself fails: the state moves to "needs recovery" with no new write attempt.

That last case is the most neglected. When the check fails, the temptation is to rewrite "to keep moving". That is exactly the path to the duplicate. An item in "needs recovery" waits for a human, with the necessary context: reference, attempt history, timestamps.

Incident table and the evidence to look for

Each incident calls for different evidence. Looking for the wrong evidence leads to the wrong decision.

Observed incident	What it does not prove	Evidence to look for	Default decision
Timeout on the external write	That the write failed	Search for the business reference in the remote tool	Verify before replaying
Webhook received twice	That two requests exist	Deduplication key at workflow start	Short-circuit the second execution
Provider authorization error	That the operation was attempted successfully	Call logs and error code	Fix access, then resume from the recorded state
Response lost after remote success	That the operation failed	Remote identifier returned or found	Mark "confirmed" without rewriting
Verification search failing	That the object exists or not	No evidence available	"Needs recovery" state, human decision
Workflow interrupted during validation	That the validation is lost	State recorded in the database	Resume from the last confirmed state

Adapt this table to your tools, but the logic holds: each ambiguous state maps to a verifiable piece of evidence, and absent evidence, a human recovery rather than an additional write.

Human recovery: a state, not a failure

Many teams treat human recovery as an admission of failure and try to drive it to zero. That is the wrong goal. The right indicator is not "zero human recoveries" but "no duplicate writes, no lost operations". An item awaiting a human decision is a system that works: it detected its own uncertainty and refuses to invent an answer.

For this recovery to be genuinely usable, the "needs recovery" item must expose three things: the business reference, the timestamped attempt history, and the precise action expected ("check whether quote QT-2026-0417 exists in the billing tool, then confirm or replay"). Without context, the human will replay blindly — reproducing the very defect you wanted to avoid. Our article on repetitive task automation stresses this point: automation should make the residual human work simpler, not more opaque.

What tools guarantee — and what they don't

Automation platforms and tooling protocols like MCP make it easier to connect your tools and workflows: they exchange tools, context and capabilities. But the protocol does not choose your business rules or permissions for you: the deduplication key, the states and the recovery thresholds remain yours to define. The documentation on MCP architecture describes that exchange framework; it does not replace your recovery policy.

Likewise, some providers offer explicit identifiers that make idempotency easier — event IDs in calendar tools are a case in point, where creating with a chosen ID prevents double bookings. Our guide on calendar automation and double bookings shows this mechanism in detail. But even there, there is no guaranteed global transaction between your system and the remote tool: verification after a timeout remains necessary.

Finally, beware of overly broad permissions. In Gmail, for example, the gmail.compose and gmail.modify scopes can also allow sending; restricting your workflow to drafts therefore requires application-level controls, not just a statement of intent. A label is not a permission boundary — and a workflow that writes externally must be designed as if every write could go out twice.

Actionable procedure: audit an existing workflow

You can audit a production workflow in half a day with this procedure:

1. List the side-effecting steps. Every creation, send, external modification. Read and computation steps are out of scope.
2. For each step, identify the deduplication key. If none exists, define one: business reference, stable attribute pair, explicit identifier where the tool allows it.
3. Check the timeout behavior. Does the workflow replay the write without verification? If so, fix it: existence check first, single replay second.
4. Check state persistence. Are the four states (pending, sent unconfirmed, confirmed, needs recovery) recorded in a database, or only in the tool's memory?
5. Test human recovery. Take an item in "needs recovery" and check that a colleague unfamiliar with the workflow can decide with the provided context, without asking for help.
6. Test duplicate triggers. Submit the same form twice, resend the same webhook in a test environment, and verify the second execution is blocked or merged.

This end-to-end test resembles the approach described in our article on testing a conversational AI agent: you don't test only the happy path, but the ambiguous ones — timeout, duplicate, recovery.

Limits and common mistakes

Three mistakes come up constantly. The first: believing a queue solves duplicates. It sequences messages; it does not deduplicate your external writes. The second: replaying a write after a timeout "because it probably failed". Probably is not proof, and the duplicate created costs more to fix than the verification. The third: states recorded without recovery context, which turn the queue into a list of mysteries nobody dares to handle.

One limit to accept as well: key-based idempotency assumes the key is stable and unique on the business side. If your references are reused, or if two request sources generate independent keys, deduplication will fail silently. The quality of your business identification determines the quality of the entire recovery chain.

If you would like to scope a mission around the reliability of your workflows, you can browse our agent profiles.

Frequently asked questions

Does a timeout mean the operation failed?

No. A timeout only means the response did not arrive in time. The operation may have succeeded, failed, or still be running. That is why the rule is to verify the expected effect (a search for the business reference on the remote side) before any new write.

Does a queue guarantee an operation runs only once?

No. A queue guarantees message delivery and ordering, not the uniqueness of an external write's effect. If the remote tool does not respond after processing the operation, resuming the queue may replay the call. Only a deduplication key or an existence check prevents the duplicate.

How do you choose a good deduplication key?

It must be stable, unique and business-borne: order number, quote reference, explicit identifier where the remote tool allows it. Avoid timestamps and identifiers generated at each execution, which change on every retry and make deduplication ineffective.

Should you automatically replay a write after a negative check?

You may replay once, with the same deduplication key, if and only if the check worked properly and confirmed the object's absence. If the check itself fails, no automatic replay: the item moves to "needs recovery" for a human decision, since no evidence is available.

Is human recovery a sign of a poorly designed workflow?

No, quite the opposite. A workflow that detects its own uncertainty and suspends the decision to a properly informed human is more reliable than one that replays blindly. The reasonable goal is not zero human recoveries, but zero duplicates and zero lost operations.