add instrumentation for non-empty memPool during upgrade

[Chris-Hibbert]

What is the Problem Being Solved?

The application of upgrade-18 to agoric-3 mainnet had some trouble because some Oracle operations were left in the MemPool, and processed after the upgrade. This caused surprising results, and resulted in a fire drill in order to re-enable the oracles to push prices. The fix was simple, but it might have been easier to find a fix if we had realized that those transactions were there.

Description of the Design

Our expectation is that there will seldom be transactions left in the memPool. If it's hard to force them to be processed before the shutdown, it would still be helpful to be warned that this occurred.

Let's see if there's a way to detect this after the chain halt (or during chain startup) and add a warning to the log. We could add alerts for the message, so we'd be aware at some level, and could know to look there if anything untoward happened during upgrade. The expectation would be that most of the time this warning would not fire.

Security Considerations

No particular security implications. Just awareness of unusual circumstances.

Scaling Considerations

Not relevant.

Test Plan

I don't know how hard it would be to inject unprocessed message into the MemPool while the chain is halting, so it might be hard to build a test case.

Upgrade Considerations

The goal is to raise awareness of an unusual condition during chain-halting software upgrades.

[Chris-Hibbert]

FYI, @gibson042

[mhofman]

I don't think this is the right way to think about this. The mempool is a purely local / per validator thing. The block proposer is free to include any transaction it wants in the block.

The problem in this case is that it was possible to "use" the previous price feeds after they had been replaced. My understanding is that even now, nothing prevents an oracle to use the previous invitation it had for the old price feed and push a price to it. There should be no effect on the current price logic by merely pushing to a replaced feed.

[Chris-Hibbert]

even now, nothing prevents an oracle to use the previous invitation it had for the old price feed and push a price to it. There should be no effect on the current price logic by merely pushing to a replaced feed.

That's correct. Nothing prevents them from re-using old invitations, except the code in the Oracles that replaces the old invitation when they get a new one.

There is no effect on the current price logic by merely pushing to a replaced feed. The performance problem didn't have anything to do with whether there would be impact on the price. If there were a simple way to revoke an invitation, that would be an easy way out. If we could kill the old price vats when we start new ones, that would also remove the problem.