feat(benchmarking): tooling for multi shard benchmarks

[Trisfald]

This PR adds the script I've been using to benchmark neard TPS on a single node (for the moment).

Benchmarks use the synth-bm tool under the hood.

The aim is to have an simple and reproducible way to run tests. In the future, the structure of the script may evolve drastically, because we want to support multi node setups like forknet.

• simple: just take a normal node and run a bunch of commands

export CASE=cases/local/1_node_5_shard
./bench.sh stop-nodes
./bench.sh reset
./bench.sh init
./bench.sh start-nodes
./bench.sh create-accounts
./bench.sh native-transfers

• reproducible: all configs, variables, test params are store in the scenarios inside case/local/

I believe this is the best place to add this tooling in nearcore , but let me know if you think otherwise.

Update

• I have improved the way config overrides work, to reduce the amount of repetition
• Added a way to configure log levels
• Support for localnet with N validators and 1 RPC node
• Better README

[mooori]

Have you considered adding multi shard support to synth-bm itself? Some potential benefits could be:

• Using Rust rather than shell scripting language. In particular that would leverage the compiler to avoid errors that are easy to make in shell scripts.
• Both the multi shard functionality and synth-bm are expected to change. Keeping them in sync is easier when everything is one tool (and again the compiler would help).
• You can probably add more detailed logging/tracing from inside the tool itself than from a shell script that invokes it.

Potential approach for multi shard support in synth-bm

Commands in synth-bm are just wrappers around async functions:

nearcore/benchmarks/synth-bm/src/main.rs

Lines 34 to 43 in e8bbcfb

	match &cli.command {
	Commands::CreateSubAccounts(args) => {
	create_sub_accounts(args).await?;
	}
	Commands::BenchmarkNativeTransfers(args) => {
	native_transfer::benchmark(args).await?;
	}
	Commands::BenchmarkMpcSign(args) => {
	contract::benchmark_mpc_sign(args).await?;
	}

Try adding new commands like Commands::CreateMultiShardSubAccounts and others which call these functions once for each shard. Roughly this might look like:

match &cli.command {
  Commands::CreateMultiShardSubAccounts(args) => {
    for n in 0..args.num_shards {
      // As this is just a sketch, it doesn't store task handles to check
      // if they terminated successfully.
      let shard_args = prepare_shard_args(args, n); // set account prefix, user_data_dir, etc
      tokio::spawn(async move {
        create_sub_accounts(shard_args).await?
      });
    }
  }
}

[Trisfald]

Have you considered adding multi shard support to synth-bm itself? Some potential benefits could be:

• Using Rust rather than shell scripting language. In particular that would leverage the compiler to avoid errors that are easy to make in shell scripts.
• Both the multi shard functionality and synth-bm are expected to change. Keeping them in sync is easier when everything is one tool (and again the compiler would help).
• You can probably add more detailed logging/tracing from inside the tool itself than from a shell script that invokes it.

Potential approach for multi shard support in synth-bm

Commands in synth-bm are just wrappers around async functions:

nearcore/benchmarks/synth-bm/src/main.rs

Lines 34 to 43 in e8bbcfb

	match &cli.command {
	Commands::CreateSubAccounts(args) => {
	create_sub_accounts(args).await?;
	}
	Commands::BenchmarkNativeTransfers(args) => {
	native_transfer::benchmark(args).await?;
	}
	Commands::BenchmarkMpcSign(args) => {
	contract::benchmark_mpc_sign(args).await?;
	}

Try adding new commands like Commands::CreateMultiShardSubAccounts and others which call these functions once for each shard. Roughly this might look like:

match &cli.command {
  Commands::CreateMultiShardSubAccounts(args) => {
    for n in 0..args.num_shards {
      // As this is just a sketch, it doesn't store task handles to check
      // if they terminated successfully.
      let shard_args = prepare_shard_args(args, n); // set account prefix, user_data_dir, etc
      tokio::spawn(async move {
        create_sub_accounts(shard_args).await?
      });
    }
  }
}

Thanks for the suggestion @mooori
I didn't look into this yet as we are in an exploratory phase, but it might be a valuable direction to take if we continue using synth-bm

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 70.52%. Comparing base (8b3fc09) to head (12143f0).

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #12918      +/-   ##
==========================================
- Coverage   70.54%   70.52%   -0.02%     
==========================================
  Files         851      851              
  Lines      174983   174983              
  Branches   174983   174983              
==========================================
- Hits       123433   123413      -20     
- Misses      46419    46438      +19     
- Partials     5131     5132       +1     

Flag	Coverage Δ
backward-compatibility	0.36% <ø> (ø)
db-migration	0.36% <ø> (ø)
genesis-check	1.42% <ø> (ø)
linux	70.36% <ø> (-0.01%)	⬇️
linux-nightly	70.15% <ø> (-0.02%)	⬇️
pytests	1.73% <ø> (ø)
sanity-checks	1.54% <ø> (ø)
unittests	70.36% <ø> (-0.02%)	⬇️
upgradability	0.36% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mooori]

Usage of synth-bm LGTM.

[mooori]

You might want near_transaction_processed_successfully_total or near_action_called_count.