Randomness (includes relay BABE randomness and local VRF) (#1376)

* try fix std node compilation error * try fix std error * bump serde dep * fix compile * update vrf pallet mock * init fix * mid config * save review comments * instant randomness generic over storage map * update not done * fix * clean pallet randomness * apply review suggestions * fix * fix * save * precompile in progress * update pallet deps * clean precompile compiles * clean fulfill body * propagate pallet errors to precompile add some functions * randomness precompile * remove Last storage items because unused can add back if remember use case * split randomness requested event * init reentrancy bool and clean precompile functions * fix vrf pallet to lookup VrfId instead of babe AuthorityId * continue configuring runtime and remove inner bool from RandomnessWrapper * remove EvmData trait bounds from precompile * moonbase compiles with randomness precompile added * fix toml * move set vrf inputs into inherent to be included after all other inherents in client * read babe randomness from relay chain state proof and fix pallet mock for unit tests * remove redundant slicing * fix line length * fix vrf unit test mock * unit tests for request randomness in randomness pallet * fix * fix moonbase config * precompile exists * precompile exists * read relay chain time in the vrf inherent as well need to consider renaming it to something more general * set babe randomness in vrf inherent * fix precompile naming * decouple pallets and unwrap or default for LastVrfInput in on initialize instead of panicking * fix naming conventions * simplify event handler to combine both vrf and randomness pallet updates in one handler impl * only one randomness pallet now and remove all babe storage items such that we use generic getter instead assigned in runtime to read relevant data from ParachainSystem * move vrf logic into own file * remove inherent as is not necessary * add back inherent to populate the Randomness babe values with most recent data of this block * add back inherent to populate babe randomness results at the beginning of the block but not during on initialize * use RandomnessResults from storage instead of getters to prepare and clean up in finish fulfill * also update RandomnessResults upon request expiration execution to cover all request removal scenarios * require inherent to be included in block and clean other code and add some todos * init client vrf digest * remove unused code * get nimbus digest provider commit * init runtime api * try configure closure but keystore cannot be borrowed in the expected way * struggling to access runtime API from inside node service file * fix rust compilation * move runtime api function helper into moonbeam vrf and use it from there * clean * fix pallet to use our VRF PreDigest instead of babes * add default genesis item for vrf input in first block after genesis * add migration and try fix randomness set output vrf verification which is panicking still in tests * try fix * add randomness inherent data provider for dev manual seal mode * add additional digests provider to manual seal params * fix * use cumulus patch to insert epoch index into relay sproof to prevent panic in dev mode * Revert "use cumulus patch to insert epoch index into relay sproof to prevent panic in dev mode" This reverts commit 45a358c. * add Randomness solidity interface to Randomness precompile * compute selectors and verify with unit test * toml sort * fix solidity interface and add test (#1643) * put utils solidity behind testing to access std * fix fix std * add precompile getters for relay block number and relay epoch index and temporarily unwrap or default for epoch index getter impl just until cumulus can be patched * fmt * fmt and try debug new on finalize error * make epoch requests take no epoch input and default ask for next epoch and fix relay block getter and fix inherent included logic to make it future proof for async backing when relay block will not change every parachain block * change per block request interface such that last param specifies number of blocks in the future instead of the exact block requested * add inherent in all runtimes * try update randomness precompile but solidity parsing is no longer in root * remove randomness from moonriver and moonbeam and redeclare solidity after merge conflict mistake * wow lots of duplication to only include randomness in moonbase * Revert "wow lots of duplication to only include randomness in moonbase" This reverts commit fa1669c. * Revert "remove randomness from moonriver and moonbeam and redeclare solidity after merge conflict mistake" This reverts commit 193d16a. * finish merge master * remove randomness from moonbeam and moonriver * Update pallets/randomness/src/lib.rs Co-authored-by: girazoki <[email protected]> * update nimbus and refactor randomness precompile ensure check and rm all instant randomness unsafe logic * Revert "update nimbus and refactor randomness precompile ensure check and rm all instant randomness unsafe logic" This reverts commit ac36956. * revert dep changes to save the Cargo lock * fix increase request fee to match substrate increase fee conventions to not pass in new value but pass in increase * camelcase solidity input var names * fix polkadot dependency * update nimbus * change epoch delay for epoch requests and add debug assert verification in pallet * wrap existing block executor with vrf verifying block executor * clean and disambiguate errors * replace reserve and unreserve with transfer to an escrow account which is assigned to precompile account * do not expect vrf digest nor set output in first block genesis or runtime upgrade * fix * update block executor as well with corner case first block logic * remove BlockExecutor and panic in pallet instead * remove migration and genesis from pallet randomness in favor of this dumb storage item * fix mocks * fix expected VrfInput to be set error in first block from runtime api * fix ts test error now 5 extrinsics * try fix ts * fix solidity test file and add randomness precompile tests (#1644) * fix solidity test file * Adds missing compiled precompiles * Adds simple test * handle memory arg cases, randomize test solidity file * cleanup * remove deleted file * add docs * add docs * add randomness precompile tests * reset Cargo.lock * fix tests * add more tests * add additional test for documented selectors * fmt * fix fmt and test * skip failing test * Fix merge * use custom defined structs * docs * fmt * Upgrading authoring version * Prevent runtime upgrade test to run when authoring version changed Co-authored-by: Crystalin <[email protected]> * add back panic for reading epoch index but now need to apply cumulus patch on current fork * patch cumulus fork to insert epoch index into RelaySproofBuilder * benchmark all public pallet functions used by precompile * TODO set output benchmark code requires inserting some dummy vrf pre digest into digests * remove tiny_keccak dep * fix tests * unskip tests * backward compatible vrf client (#1661) * return None when API is missing * fmt * get runtime api only once * do not use free balance checks * fix * replace ReservAccount config associated type with Pallet ID for randomness pallet * temp weight for set output and add todo to benchmark file * more unit tests * Add test to ensure fn remains Mandatory * Attempt to hook up pallet_randomness benchmarks to moonbase only * bring back moonbase-runtime-benchmarks * Comment out failing benchmarks * Hook up randomness weights * make expiration type depend on request type and ensure CannotMakeRandomnessRequestAfterExpirationDelay * fix and clean * clean * add num words before refactoring * Redesign Randomness Precompile Interface (#1669) * Providing new Randomness Precompile interfaces * Removes useless Request Status None * Fixes documentation rawFulfillRandomWords * init refactor * save need to add back ExpirationDelay and keep MaxDelay they are distinct * Tweak pallet randomness consumer interface * Adds randomness compiled contracts * get request first attempt impl * get request status * clean * Adds a Lottery Solidity Contract for demo/test (#1672) * Provides a randomness lottery demo * Adds the increaseRequestFees method * Fixes jackpot * Restore singular word "fee" * Adds doc to randomness getRequest * Updates compiled Lottery * Adds the getRequestStatus * Fix request id init + new events * disambiguate client errors to panic if vrf is not able to be provided * fix * insert default vrf output in the first block of the runtime upgrade * fix enum encoding * Makes requestId using uint256 * Adds the lottery test * Fix selector modifiers in precompiles randomness * return request id with request_randomness * First startlottery tests working * Adds more test for lottery * call fulfill in the precompile address * clean and add todo for fulfill needs function selector for callback * Adds test for fulfillment * clean code * add rawFulfillRandomWords function selector to the provide callback * Adds request id to fulfillment * add request id to provide randomness * Fixes randomness precompile selector callback * Adds support for deposit in randomness demo * Better setup for deposit * Adds test for lottery rewards * Fixes randomness event logs * fix and add number of random words constraint * fix order * Adds status to the lottery * Fix benchmark build * revert client panics * into master * Fix execute_request_expiration * Update weights.rs * address review * fix * address some comments * revert if RequestId input overflows u64 type set in pallet * Official benchmarks * add crude weight to gas need to replace with constants and add tests to verify equal to values in actual runtime * charge constant gas in precompile and convert weight to gas in tests need to move tests to runtime integration tests * move tests to runtime and remove randomness pallet WeightInfo config associated type * Fixes toml and unused * fix solidity test * Fixes demo solidity contract * Adds support for Babe test in demo * add 100 num words for benchmarking * add 100 words for all benchmarks * Update weights * fix integration test and update precompile gas values * thanks @girazoki make get last vrf output return option so digest is not included for moonriver and moonbeam * Adds babe request tests * address other review comments * rm unused dep * remove RandomnessResult when there are 0 requests left * Adds test for randomnessResults * Adds test for multiple ethereum transfers * Removes precompile tests in benefit of randomness tests * Adds vrf babe test (#1684) * add babe tests * Adds randomness to typescript api * Fix babe lottery tests * Fix vrf lottery * Adds test around the randomness result * prettier Co-authored-by: librelois <[email protected]> Co-authored-by: Crystalin <[email protected]> Co-authored-by: Nisheeth Barthwal <[email protected]> Co-authored-by: girazoki <[email protected]> Co-authored-by: nanocryk <[email protected]> Co-authored-by: notlesh <[email protected]>
moonbeam-foundation · Jul 18, 2022 · 6772792 · 6772792
1 parent 2e8425b
commit 6772792
Show file tree

Hide file tree

Showing 72 changed files with 33,580 additions and 868 deletions.
diff --git a/.cargo/config.toml b/.cargo/config.toml
@@ -1,5 +1,6 @@
 [alias]
 moonbase = " build --release -p moonbeam --no-default-features --features moonbase-native"
+moonbase-benchmarks = " build --release -p moonbeam --no-default-features --features moonbase-runtime-benchmarks"
 moonbase-rococo = " build --release -p moonbeam --no-default-features --features moonbase-native,rococo-native"
 moonriver = " build --release -p moonbeam --no-default-features --features moonriver-native"
 moonriver-rococo = " build --release -p moonbeam --no-default-features --features moonriver-native,rococo-native"

diff --git a/Cargo.lock b/Cargo.lock
diff --git a/Cargo.toml b/Cargo.toml
@@ -4,6 +4,7 @@ members = [
 	"bin/utils/moonkey",
 	"client/rpc/finality",
 	"client/rpc/manual-xcm",
+	"client/vrf",
 	"node",
 	"node/cli",
 	"node/service",
@@ -18,6 +19,7 @@ members = [
 	"precompiles/crowdloan-rewards",
 	"precompiles/pallet-democracy",
 	"precompiles/parachain-staking",
+	"precompiles/randomness",
 	"precompiles/relay-encoder",
 	"precompiles/utils/macro",
 	"precompiles/xcm-transactor",

diff --git a/client/vrf/Cargo.toml b/client/vrf/Cargo.toml
@@ -0,0 +1,27 @@
+[package]
+name = "moonbeam-vrf"
+authors = [ "PureStake" ]
+edition = "2018"
+homepage = "https://moonbeam.network"
+license = "GPL-3.0-only"
+repository = "https://github.com/PureStake/moonbeam/"
+version = "0.1.0"
+
+[dependencies]
+# Substrate
+codec = { package = "parity-scale-codec", version = "3.0.0", features = [ "derive" ] }
+sp-api = { git = "https://github.com/paritytech/substrate", branch = "polkadot-v0.9.23" }
+sp-application-crypto = { git = "https://github.com/paritytech/substrate", branch = "polkadot-v0.9.23", default-features = false }
+sp-consensus-vrf = { git = "https://github.com/paritytech/substrate", branch = "polkadot-v0.9.23", default-features = false }
+sp-core = { git = "https://github.com/paritytech/substrate", branch = "polkadot-v0.9.23" }
+sp-keystore = { git = "https://github.com/paritytech/substrate", branch = "polkadot-v0.9.23", default-features = false }
+sp-runtime = { git = "https://github.com/paritytech/substrate", branch = "polkadot-v0.9.23", default-features = false }
+
+# Moonbeam
+session-keys-primitives = { path = "../../primitives/session-keys" }
+
+# Nimbus
+nimbus-primitives = { git = "https://github.com/purestake/nimbus", branch = "moonbeam-polkadot-v0.9.23" }
+
+# Polkadot
+polkadot-primitives = { git = "https://github.com/purestake/polkadot", branch = "moonbeam-polkadot-v0.9.23" }
diff --git a/client/vrf/src/lib.rs b/client/vrf/src/lib.rs
@@ -0,0 +1,75 @@
+// Copyright 2019-2022 PureStake Inc.
+// This file is part of Moonbeam.
+
+// Moonbeam is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// Moonbeam is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+
+// You should have received a copy of the GNU General Public License
+// along with Moonbeam.  If not, see <http://www.gnu.org/licenses/>.
+
+//! VRF client primitives for client-side verification
+
+use nimbus_primitives::NimbusId;
+use session_keys_primitives::{make_transcript, make_transcript_data, PreDigest, VrfApi, VrfId};
+use sp_application_crypto::{AppKey, ByteArray};
+use sp_consensus_vrf::schnorrkel::{PublicKey, VRFOutput, VRFProof};
+use sp_core::H256;
+use sp_keystore::{SyncCryptoStore, SyncCryptoStorePtr};
+
+/// Uses the runtime API to get the VRF inputs and sign them with the VRF key that
+/// corresponds to the authoring NimbusId.
+pub fn vrf_pre_digest<B, C>(
+	client: &C,
+	keystore: &SyncCryptoStorePtr,
+	nimbus_id: NimbusId,
+	parent: H256,
+) -> Option<sp_runtime::generic::DigestItem>
+where
+	B: sp_runtime::traits::Block<Hash = sp_core::H256>,
+	C: sp_api::ProvideRuntimeApi<B>,
+	C::Api: VrfApi<B>,
+{
+	let at = sp_api::BlockId::Hash(parent);
+	let runtime_api = client.runtime_api();
+
+	// first ? for runtime API, second ? for if last vrf output is not available
+	let last_vrf_output = runtime_api.get_last_vrf_output(&at).ok()??;
+	// first ? for runtime API, second ? for not VRF key associated with NimbusId
+	let key: VrfId = runtime_api.vrf_key_lookup(&at, nimbus_id).ok()??;
+	let vrf_pre_digest = sign_vrf(last_vrf_output, key, &keystore)?;
+	Some(session_keys_primitives::digest::CompatibleDigestItem::vrf_pre_digest(vrf_pre_digest))
+}
+
+/// Signs the VrfInput using the private key corresponding to the input `key` public key
+/// to be found in the input keystore
+fn sign_vrf(last_vrf_output: H256, key: VrfId, keystore: &SyncCryptoStorePtr) -> Option<PreDigest> {
+	let transcript = make_transcript(last_vrf_output.clone());
+	let transcript_data = make_transcript_data(last_vrf_output);
+	let try_sign =
+		SyncCryptoStore::sr25519_vrf_sign(&**keystore, VrfId::ID, key.as_ref(), transcript_data);
+	if let Ok(Some(signature)) = try_sign {
+		let public = PublicKey::from_bytes(&key.to_raw_vec()).ok()?;
+		if signature
+			.output
+			.attach_input_hash(&public, transcript)
+			.is_err()
+		{
+			// VRF signature cannot be validated using key and transcript
+			return None;
+		}
+		Some(PreDigest {
+			vrf_output: VRFOutput(signature.output),
+			vrf_proof: VRFProof(signature.proof),
+		})
+	} else {
+		// VRF key not found in keystore or VRF signing failed
+		None
+	}
+}
diff --git a/node/Cargo.toml b/node/Cargo.toml
@@ -48,3 +48,9 @@ runtime-benchmarks = [
 	"pallet-xcm/runtime-benchmarks",
 	"xcm-builder/runtime-benchmarks",
 ]
+
+moonbase-runtime-benchmarks = [
+	"moonbeam-cli/moonbase-runtime-benchmarks",
+	"pallet-xcm/runtime-benchmarks",
+	"xcm-builder/runtime-benchmarks",
+]
diff --git a/node/cli/Cargo.toml b/node/cli/Cargo.toml
@@ -56,3 +56,5 @@ try-runtime = [
 	"try-runtime-cli",
 ]
 wasmtime = [ "sc-cli/wasmtime" ]
+
+moonbase-runtime-benchmarks = [ "service/moonbase-runtime-benchmarks" ]
diff --git a/node/cli/src/command.rs b/node/cli/src/command.rs
@@ -511,6 +511,12 @@ pub fn run() -> Result<()> {
 							#[cfg(not(feature = "moonbase-native"))]
 							_ => panic!("invalid chain spec"),
 						}
+					} else if cfg!(feature = "moonbase-runtime-benchmarks") {
+						return runner.sync_run(|config| {
+							cmd.run::<service::moonbase_runtime::Block, service::MoonbaseExecutor>(
+								config,
+							)
+						});
 					} else {
 						Err("Benchmarking wasn't enabled when building the node. \
 					You can enable it with `--features runtime-benchmarks`."

diff --git a/node/service/Cargo.toml b/node/service/Cargo.toml
@@ -40,7 +40,9 @@ moonbeam-rpc-primitives-debug = { path = "../../primitives/rpc/debug" }
 moonbeam-rpc-primitives-txpool = { path = "../../primitives/rpc/txpool" }
 moonbeam-rpc-trace = { path = "../../client/rpc/trace" }
 moonbeam-rpc-txpool = { path = "../../client/rpc/txpool" }
+moonbeam-vrf = { path = "../../client/vrf" }
 pallet-parachain-staking = { path = "../../pallets/parachain-staking" }
+session-keys-primitives = { path = "../../primitives/session-keys" }
 
 # Moonbeam runtimes
 moonbase-runtime = { path = "../../runtime/moonbase", optional = true }
@@ -180,3 +182,10 @@ try-runtime = [
 	"moonbase-runtime",
 	"moonbase-runtime/try-runtime",
 ]
+
+moonbase-runtime-benchmarks = [
+	"moonbase-native",
+	"moonbase-runtime/moonbase-runtime-benchmarks",
+	"moonbase-runtime/runtime-benchmarks",
+	"pallet-ethereum/runtime-benchmarks",
+]
diff --git a/node/service/src/client.rs b/node/service/src/client.rs
@@ -46,6 +46,7 @@ pub trait RuntimeApiCollection:
 	+ nimbus_primitives::NimbusApi<Block>
 	+ nimbus_primitives::AuthorFilterAPI<Block, nimbus_primitives::NimbusId>
 	+ cumulus_primitives_core::CollectCollationInfo<Block>
+	+ session_keys_primitives::VrfApi<Block>
 where
 	<Self as sp_api::ApiExt<Block>>::StateBackend: sp_api::StateBackend<BlakeTwo256>,
 {
@@ -67,7 +68,8 @@ where
 		+ moonbeam_rpc_primitives_txpool::TxPoolRuntimeApi<Block>
 		+ nimbus_primitives::NimbusApi<Block>
 		+ nimbus_primitives::AuthorFilterAPI<Block, nimbus_primitives::NimbusId>
-		+ cumulus_primitives_core::CollectCollationInfo<Block>,
+		+ cumulus_primitives_core::CollectCollationInfo<Block>
+		+ session_keys_primitives::VrfApi<Block>,
 	<Self as sp_api::ApiExt<Block>>::StateBackend: sp_api::StateBackend<BlakeTwo256>,
 {
 }