Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Only keep track of recently used stacks in memory. #2591

Open
wants to merge 33 commits into
base: staging
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
44fc174
Only keep track of recently used stacks in memory.
vicsn Jan 10, 2025
9486378
Revert MAX_TRANSMISSIONS_PER_BATCH location
vicsn Jan 10, 2025
9e56e07
Fix circleci resource
vicsn Jan 10, 2025
4a93690
Lower MAX_STACKS when testing
vicsn Jan 11, 2025
135e279
Resolve bugs
vicsn Jan 11, 2025
e76606d
Fix tests: don't superfluously cache credits.aleo
vicsn Jan 13, 2025
678fa94
Update circleci config
vicsn Jan 13, 2025
ce57e74
Update MAX_STACKS to fix test_long_import_chain
vicsn Jan 13, 2025
9e19924
Update ledger/narwhal/batch-header/src/lib.rs
vicsn Jan 14, 2025
8bf0d8d
Update synthesizer/process/src/lib.rs
vicsn Jan 14, 2025
b884134
Update synthesizer/process/src/lib.rs
vicsn Jan 14, 2025
ed2d269
Update synthesizer/process/src/lib.rs
vicsn Jan 14, 2025
f9eaab6
Update synthesizer/process/src/lib.rs
vicsn Jan 14, 2025
fc0ec5f
Get rid of stacks contention
vicsn Jan 15, 2025
3291ccc
Correct typedef name
vicsn Jan 20, 2025
12345aa
Fix slow test; might need DB fix to pass
vicsn Jan 20, 2025
b1e2a0a
Reduce size of feature gating diff
vicsn Jan 20, 2025
f1022e3
Only use num_stacks_in_memory in tests
vicsn Jan 20, 2025
a16db98
Cargo fmt
vicsn Jan 20, 2025
2dc6770
Remove useless logs
vicsn Jan 20, 2025
f3fb877
Remove clone
vicsn Jan 20, 2025
67ac36b
Howard's spirit lives on
vicsn Jan 20, 2025
f812a4b
Keep it DRY
vicsn Jan 20, 2025
9b61b78
contains_program_in_memory -> contains_program_in_cache
vicsn Jan 20, 2025
d3ab602
Improve logging
vicsn Jan 20, 2025
8b1a28a
Speed up credits.aleo comparison
vicsn Jan 20, 2025
c630a36
remove incorrect comment
vicsn Jan 20, 2025
c5fc425
Merge remote-tracking branch 'provable/staging' into stack_cache_relo…
vicsn Jan 20, 2025
0bddecd
Try to avoid OOM
vicsn Jan 20, 2025
f2bf9fe
Fix test_long_import_chain
vicsn Jan 20, 2025
aac3fd0
Fix circleci
vicsn Jan 20, 2025
2e8c41c
Fix real evict example
vicsn Jan 20, 2025
815eecd
Try to avoid computing credits.aleo program id
vicsn Jan 20, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -503,7 +503,7 @@ jobs:
ledger-with-rocksdb:
docker:
- image: cimg/rust:1.81.0 # Attention - Change the MSRV in Cargo.toml and rust-toolchain as well
resource_class: << pipeline.parameters.medium >>
resource_class: << pipeline.parameters.large >>
steps:
- run_serial:
flags: --features=rocks
Expand Down Expand Up @@ -692,7 +692,7 @@ jobs:
resource_class: << pipeline.parameters.twoxlarge >>
steps:
- run_serial:
flags: -- --ignored test_deployment_synthesis_overload test_deep_nested_execution_cost
flags: -- --ignored test_deployment_synthesis_overload test_deep_nested_execution_cost --test-threads=1
workspace_member: synthesizer
cache_key: v1.0.0-rust-1.81.0-snarkvm-synthesizer-mem-heavy-cache

Expand Down
2 changes: 2 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions console/network/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -203,6 +203,10 @@ pub trait Network:
// Increasing this value will require a migration to prevent forking during network upgrades.
const MAX_CERTIFICATES: u16;

/// The maximum number of stacks in the process.
/// Must be at least Self::MAX_PROGRAM_DEPTH * Self::MAX_IMPORTS to be safe.
const MAX_STACKS: usize = Self::MAX_PROGRAM_DEPTH * Self::MAX_IMPORTS * 10;

/// The maximum number of bytes in a transaction.
// Note: This value must **not** be decreased as it would invalidate existing transactions.
const MAX_TRANSACTION_SIZE: usize = 128_000; // 128 kB
Expand Down
2 changes: 1 addition & 1 deletion ledger/block/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -656,7 +656,7 @@ pub mod test_helpers {
let inputs = [address.to_string(), format!("{amount}_u64")];

// Initialize the process.
let process = Process::load().unwrap();
let process = Process::load_testing_only().unwrap();
// Authorize the function.
let authorization =
process.authorize::<CurrentAleo, _>(&private_key, locator.0, locator.1, inputs.iter(), rng).unwrap();
Expand Down
2 changes: 1 addition & 1 deletion ledger/block/src/transaction/deployment/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ function compute:
assert!(string.is_empty(), "Parser did not consume all of the string: '{string}'");

// Construct the process.
let process = Process::load().unwrap();
let process = Process::load_testing_only().unwrap();
// Compute the deployment.
let deployment = process.deploy::<CurrentAleo, _>(&program, rng).unwrap();
// Return the deployment.
Expand Down
4 changes: 2 additions & 2 deletions ledger/block/src/transaction/fee/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -241,7 +241,7 @@ pub mod test_helpers {
let priority_fee_in_microcredits = 1_000;

// Initialize the process.
let process = Process::load().unwrap();
let process = Process::load_testing_only().unwrap();
// Authorize the fee.
let authorization = process
.authorize_fee_private::<CurrentAleo, _>(
Expand Down Expand Up @@ -298,7 +298,7 @@ pub mod test_helpers {
let priority_fee = 1_000;

// Initialize the process.
let process = Process::load().unwrap();
let process = Process::load_testing_only().unwrap();
// Authorize the fee.
let authorization = process
.authorize_fee_public::<CurrentAleo, _>(
Expand Down
2 changes: 1 addition & 1 deletion ledger/narwhal/batch-header/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ impl<N: Network> BatchHeader<N> {
pub const MAX_CERTIFICATES_BEFORE_V3: u16 = 100;
/// The maximum number of rounds to store before garbage collecting.
pub const MAX_GC_ROUNDS: usize = 100;
/// The maximum number of transmissions in a batch.
/// The maximum number of transmissions per batch.
/// Note: This limit is set to 50 as part of safety measures to prevent DoS attacks.
/// This limit can be increased in the future as performance improves. Alternatively,
/// the rate of block production can be sped up to compensate for the limit set here.
Expand Down
2 changes: 1 addition & 1 deletion ledger/puzzle/epoch/src/synthesis/program/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ function synthesize:
let program = Program::from_str(&program_string)?;

// Initialize a new process.
let process = Process::<N>::load()?;
let process = Process::<N>::load_no_storage()?;
// Initialize the stack with the synthesis challenge program.
let stack = Stack::new(&process, &program)?;

Expand Down
4 changes: 2 additions & 2 deletions ledger/query/src/query.rs
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ impl<N: Network, B: BlockStorage<N>> QueryTrait<N> for Query<N, B> {
}
}

/// Returns a state path for the given `commitment`.
/// Returns the current block height.
fn current_block_height(&self) -> Result<u32> {
match self {
Self::VM(block_store) => Ok(block_store.max_height().unwrap_or_default()),
Expand All @@ -159,7 +159,7 @@ impl<N: Network, B: BlockStorage<N>> QueryTrait<N> for Query<N, B> {
}
}

/// Returns a state path for the given `commitment`.
/// Returns the current block height.
#[cfg(feature = "async")]
async fn current_block_height_async(&self) -> Result<u32> {
match self {
Expand Down
Loading