Pattern: Idempotent Consumer

pattern  

---

Context

In an enterprise application, it’s usually a good idea to use a message broker that guarantees at-least once delivery. At-least once delivery guarantees that a message broker will deliver a message to a consumer even if errors occur. One side-effect, however, is that the consumer can be invoked repeatedly for the same message. Consequently, a consumer must be idempotent: the outcome of processing the same message repeatedly must be the same as processing the message once. If a consumer is not idempotent, multiple invocations can cause bugs. For example, a consumer of an AccountDebited message that subtracts the debit amount from the current balance would calculate the incorrect balance.

Problem

How does a message consumer handle duplicate messages correctly?

Solution

Make a consumer idempotent by having it record the IDs of processed messages in the database. When processing a message, a consumer can detect and discard duplicates by querying the database. There are a couple of different places to store the message IDs. One option is for the consumer to use a separate PROCESSED_MESSAGES table. The other option is for the consumer to store the IDs in the business entities that it creates or updates.

Here’s how a consumer can use the PROCESSED_MESSAGES table:

After starting the database transaction, the message handler inserts the message’s ID into the PROCESSED_MESSAGE table. Since the (subscriberId, messageID) is the PROCESSED_MESSAGE table’s primary key the INSERT will fail if the message has been already processed successfully. The message handler can then rollback the transaction and ignore the message.

See also

• The Eventuate framework implements this pattern.
• This blog post about this pattern