6.) State Tracing & Balance Reconciliation

Part of Substrate Archive's goal is to aggregate data in a way which allows third-parties to come up with reasonable conclusions to the state of the chain at some point in time, without digging too deep into the internals of substrate themselves. The State Tracing initiative is part of this move in order to support more fine-grained detail into a substrate blockchains inner-workings, dependent on the information the end-user requires. State Tracing is a joint-initiative between (substrate-api-sidecar) and this project (substrate-archive). Substrate archive focuses more on historical data, while sidecar is the stateless counterpart.

What Is State Tracing?

State Tracing provides a way to look into which underlying runtime functions, both native & WASM, achieve the resulting state of the chain. A prominent use case for this can be to reconcile balances. A user may track a balance as it flows through the substrate runtime environment, coming to some conclusion about how some tokens were used, and where they ended up.

Current State of State Tracing and Functionality

Currently, the State-Tracing MvP is implemented in Archive with The State Tracing PR. A user can run Archive with state-tracing, collect traces from past runtimes, and query those traces with SQL. WASM-Tracing and Native are both supported.

How To

State Tracing can be enabled with these options in the TOML configuration files of Polkadot-Archive/Node-Template-Archive:

[wasm_tracing]
# Targets for tracing.
targets = '''wasm_tracing,pallet,frame,state'''

# Folder where tracing-enabled WASM binaries are kept.
folder = "/home/insipx/projects/parity/substrate-archive/master/wasm-tracing/"

More specifically, uncommenting 'folder' will enable tracing (traces will be crawled & committed to postgres), while commenting it will disable tracing (traces will not be crawled, no postgres inserts).

Options Overview

• targets
  • A filter for the traces. wasm_tracing indicates to accepts traces originating from inside WASM Runtimes, pallet and frame indicate to include traces from inside pallet/frame-runtimes. state indicates to include traces resulting from substrate state operations.
• folder
  • In order to enable tracing from inside historical WASM runtimes, or runtimes which came before the current on-chain runtime, we need to use specially-compiled WASM runtimes. Compiling these runtimes is described here. Runtimes with tracing-enabled that have already been compiled are available here. In order to use them, you should have the folder with the runtimes for the chain you wish to trace available on your local disk, and pass that folder path to the folder option.

Caveats

• Tracing for all historical WASM runtimes does not yet exist. So far, we have successfully compiled from polkadot/kusama/westend runtime version 0.8.25. If you want to include tracing for your own chain, and that chain has previous runtimes then those runtimes will have to have wasm-tracing backported into them. If your chain is new (I.E. does not have many runtime versions before tracing was added to substrate), this should not be a problem.

• Tracing values are not yet stored decoded in the tracing database, rather they are stored as strings of bytes. This means that a user will need their own type-decoding pipeline if they want human-readable versions of the data.

• Tracing includes runtime overhead, so it may be slower for substrate-archive to sync than if you were to sync without state-tracing.

Roadmap

• Integrate type-decoding for state-tracing
• Create a tangible flow for balance-reconciliation using substrate-archive state-tracing