[Feature] Inbox Support for "Crash Only" or "Replay"

[iancooper]

Fundamentally with an inbox we store two things: I have received this; I have processed this.

• I have received this => mainly useful for tracking how we got to current state i.e. what have we seen.
• I have processed this => mainly useful for tracking that we should not apply this again because the operation would not be idempotent.

We use the former to recover from outages, the latter is used for coping with "at least once."

Now you do normally choose only to ack once you have processed, so leaving a message in the queue means that you will reprocess it, and typically we let you reprocess unless you have flagged "I have processed this".

But issues do happen. Typically it is "there was a bug, and so I processed it and acked it, but my bug means I did not complete the required actions and now the downstream is incorrect". An option here is to then reprocess from your inbox. Of course to do that, I have to have a model of reading my inbox and reprocessing anything that is not "processed" at start up (this helps with crash only software as well btw) so that I can just wipe out the processed dates of various messages to force them to be replayed through the pipeline.

Now, today, we don't do that out of the box but I have always thought it would be useful to support an option to re-process incomplete inbox commands at startup.

[iancooper]

BTW this would also allow you to recreate the state any entities by replaying your inbox, through simply clearing the processed indicator in the inbox for a date