diff --git a/storage/design/diagrams/data_encoding_distribution.svg b/storage/design/diagrams/data_encoding_distribution.svg new file mode 100644 index 000000000..12c3e5597 --- /dev/null +++ b/storage/design/diagrams/data_encoding_distribution.svg @@ -0,0 +1,200 @@ + + + + + + + + + + + + + + + + + Canvas 1 + + + Layer 1 + + + + + Client-Block + + + + + + + subblock 0 + data + + + + + + + subblock 1 + data + + + + + + + subblock 2 + data + + + + + + + + + + + + + + Provider + + + + + + + Block + + + + + + + + + + + + + + + + + + Provider + + + + + + + Block + + + + + + + Block + + + + + + + + + + + + + + Provider + + + + + Person + + + + + + + Data + + + + + + + Block + + + + + + + Block + + + + + + + + + + + + + + Block + + + + + + + + + Erasure Coding + + + + + + + + + + + + + + + + + + + + + + + + A client-block is an erasure-coded shard of client data + structured as subblocks of fixed size. + + + + + Data Encoding and Distribution + + + + + diff --git a/storage/design/diagrams/original_data_commitment.svg b/storage/design/diagrams/original_data_commitment.svg new file mode 100644 index 000000000..5d74a8546 --- /dev/null +++ b/storage/design/diagrams/original_data_commitment.svg @@ -0,0 +1,210 @@ + + + + + + + + + + + + + + + + + Canvas 1 + + + Layer 1 + + + Poly + r + encodes all data in a given client block. + Subblocks are field-sized elements of the curve. + A commitment to r forms the “root identifier” in the rate certificate for the block. + Poly + r + is posted on-chain but *not* posted to blob space. + + + + + + + Provider + + + + + + + Client-Block + + + + + + + + + + | | | | + + + + + + | + + + + + + subblock index + + + + + 0 1 2 3 + + + + + + subblock + value + + + + + + + subblock + 0 + data + + + + + + + subblock + 1 + data + + + + + + + subblock + 2 + data + + + + + + + + + + + + + subblock + 0 + data + + + + + + subblock + 1 + data + + + + + + subblock + + data + + + + + + + + Rate Certificate + + + + + + + + + + + + + + + Polynomial + r + (“receipt”) + + r(i) = subblock[i] + + + + + + + + Root ID: xxx + + + + + + + + + + + + + + + Commitment (r) + “root id” + + + + + + + + RC Posted + on-chain + + + + + Original Data Commitment + + + + VPN Lock + + + + + diff --git a/storage/design/diagrams/periodic_p_commitment.svg b/storage/design/diagrams/periodic_p_commitment.svg new file mode 100644 index 000000000..8901b3317 --- /dev/null +++ b/storage/design/diagrams/periodic_p_commitment.svg @@ -0,0 +1,277 @@ + + + + + + + + + + + + + + + + + Canvas 1 + + + Layer 1 + + + + + + + + + block 0 + subblock r0 + data + + + + + + | | | | + + + + + + | + + + + + + + + + + + + + block index + + + + + 0 1 2 3 + + + + + + subblock + value + + + + + Track Changes + + + + + Beacon + r0, r1, r2, … + + + + + + + + Provider + + + + + + + Block 0 + + + + + + + subblock + 0 + + + + + + + subblock + 1 + + + + + + + + + + + + + + Block 1 + + + + + + + subblock + 0 + + + + + + + subblock + 1 + + + + + + + + + + + + + + Block n + + + + + + + + subblock + r0 + + + + + + + subblock + r1 + + + + + + + subblock + r2 + + + + + + + + + + + + + + + + + + + + + + + block 1 + subblock r1 + data + + + + + + + block 2 + subblock r2 + data + + + + + For each time period a provider posts a commitment that samples data + availability for each of its hosted client-blocks. + Poly + p + encodes a beacon-selected subblock from each client-block. + A commitment to + p + is posted on-chain but poly + p + is *not* posted to blob space. + + + + + + + + + + + + + + + Polynomial + p + (“periodic commitment”) + + beacon = r0, r1, r2, … + p(i) = block[i].subblock[beacon[i]] + + + + + + For each time period, the pseudo-random beacon designates + one subblock from each provider client-block. + + + + + + + + Commitment (p) + Posted on-chain + + + + + Periodic Data Commitment + + + + VPN Lock + + + + + diff --git a/storage/design/diagrams/periodic_q_commitment.svg b/storage/design/diagrams/periodic_q_commitment.svg new file mode 100644 index 000000000..d3228d85f --- /dev/null +++ b/storage/design/diagrams/periodic_q_commitment.svg @@ -0,0 +1,244 @@ + + + + + + + + + + + + + + + + + Canvas 1 + + + Layer 1 + + + + + + + + + client-block 0 + commitment + + + + + + | | | | + + + + + + | + + + + + + + + + + + + + client-block index + + + + + 0 1 2 3 + + + + + + Poly r + commitment + (Root id) + + + + + + + client-block 1 + commitment + + + + + + Provider + + + + + + + Block 0 + + + + + + + + + + + + + + + + + + + + Block 1 + + + + + + + + + + + + + + + + + + + + Block n + + + + + + + + Block 0 + Root id + commitment(r[I]) + + + + + + + Block 1 + Root id + commitment(r[I]) + + + + + + + Block n + Root id + commitment(r[I]) + + + + + + + + + + + + + + For each time period a provider posts a commitment encompassing all of its hosted client blocks. + Poly + q + encodes the + r + poly commitment (root id) of each block hosted by the provider. + A commitment to + q + is posted on-chain and poly + q + is posted to blob space. + Providers evaluate q to search for known block ids. + + + + + Periodic Client Commitment + + + + + + + + + + + + + + + Polynomial + q + (“qlient list”) + q(i)=commitment( r[i] ) + + + + + + + + Commitment (q) + Posted on-chain + + + + Filter Drama + + + + + + + + Poly q + Posted to blob space + + + + VPN Lock + + + + + + + client-block 2 + commitment + + + + + diff --git a/storage/design/settlement-contract-pseudocode.txt b/storage/design/settlement-contract-pseudocode.txt new file mode 100644 index 000000000..6cfeaf6c1 --- /dev/null +++ b/storage/design/settlement-contract-pseudocode.txt @@ -0,0 +1,138 @@ + +// +// Data Types +// + +Field // field-sized value +Point // point-sized value +BlockNumber // Ethereum block number +TokenValue // funds token value +Address // Ethereum address +Signature // signature components +Int // integer value + +// A polynomial opening / inclusion proof. (Used with a commitment: Point). +struct Opening { + proof: Point // Proof offered + z_eval: Field // z index of the evaluation + y_eval: Field // y output value of the evaluatiion +} + +// A rate certificate +struct RateCertificate { + block_root_id: Point // The block commitment to which the RC refers (commitment to poly r) + commitment_payment: TokenValue // Amount paid per commitment + client: Address // The client (block owner) address + signature: Signature // The client (block owner) signature +} + +// Periodic commitment to beacon-selected subblocks and hosted client block ids. +struct PeriodicCommitment { + p: Point // commitment to the beacon-selected subblock + q: Point // commitment to the client block ids +} + +// A provider-generated request to settle for a period +struct SettlementRequest { + period: BlockNumber // The period to settle + block_root_id: Point // The original r poly commitment to the block. + r_opening: Opening // Opening of r (original block commitment) + p_opening: Opening // Opening of p (periodic subblock commitment) + q_opening: Opening // Opening of q (periodic "client" commitment) +} + + +// +// The Settlement Contract +// + +function post_rate_certificate(rate_certificate: RateCertificate) -> void { + // Store the rate certificate in contract storage + // Note that there may be multiple valid rate certificates for the block. + store_rate_certificate(rate_certificate) +} + +function post_periodic_commitment(commitment: PeriodicCommitment) -> void { + // Store the commitments in contract storage associated with the current block number. + store_commitment(msg.sender, block.number, commitment) +} + +// Settle multiple time periods (allowing for aggregation optimizations) +function settle_multiple(requests: Array): void { + for request in requests { settle_request(request) } +} + +// Settle a single time period +function settle(request: SettlementRequest): void { + + // Get the on-chain commitments for the provider and time period [assuming contract storage here]. + // Note that since p and q committments are always stored with the current block number when posted, + // they serve as witness that the commitments were made during the requested settlement period. + provider = msg.sender // Address of the provider (caller) + commitment_p, commitment_q = get_stored_commitments(provider, request.period) + + // + // Verify the p, r, and q KZG openings. + // i.e. Confirm that the openings are individually consistent with their respective commitments. + // + s = trusted_setup() + commitment_r = request.block_root_id + assert verify_kzg_proof(commitment_r, request.r_opening.proof, request.r_opening.z_eval, request.r_opening.y_eval, s) + assert verify_kzg_proof(commitment_p, request.p_opening.proof, request.p_opening.z_eval, request.p_opening.y_eval, s) + assert verify_kzg_proof(commitment_q, request.q_opening.proof, request.q_opening.z_eval, request.q_opening.y_eval, s) + + // + // Verify the correspondence between p, r, and the beacon value for the period. + // i.e. the p (periodic) value matches the r (original data) value *at* the + // beacon-selected subblock index in r. + // + + // Determine the random beacon subblock index for the period [assuming accessible on-chain] + // Note that this assumes that the beacon selection is the same for all indexes (z_eval) in the p poly. + // If that is undesirable for some reason we can incorporate the p_opening.z_eval into the beacon. + beacon_selected_subblock_index = beacon_for_period(request.period) + + // Assert that the r opening is for the beacon-selected subblock + // i.e. The r opening is proving the correct subblock + assert request.r_opening.z_eval == beacon_selected_subblock_index + + // + // Assert that the p commitment value and r block commitment value match. + // i.e. the data comitted for the period matches the original subblock data. + // + assert request.p_opening.y_eval == request.r_opening.y_eval + + // + // Verify the correspondence between q and r: + // Assert that the q opening value matches the client block root id. + // i.e. the provider committed to the client block in question during the period. + // + block_root_id = request.r_opening.commitment + assert request.q_opening.y_eval == block_root_id + + // About q_poly: Note that the q_poly z_eval is used only in verifying the kzg opening and has no + // cohort-recognizable semantics here. In order for the cohort to do its job of scanning the periodic + // q poly commitments (utilizing the blob space data) it must search for known block root ids (r commits) in q. + + // Get all valid rate certificates for the block [assuming contract storage here] + rate_certificates = get_rate_certificates(block_root_id) + + // Do the remaining logic for each valid rate certificate + // ... + + // Issue payment + client = rate_certificate.client + payment = rate_certificate.commitment_payment + send_payment(client, provider, payment) + + // Mark the period as settled, preventing repeat claims. + // Note that the logic must apply to any potential provider making a rival claim, + // so simply removing the settled provider's commitments is not sufficient. + // e.g. Maybe we structure the storage to allow for removing all commitments for the period? + mark_payment(provider, request.period) + + // Clean up and reclaim or re-use any storage no longer needed for this period. + clean_up_storage() +} + + diff --git a/storage/design/storage-group-meetings/storage-group-meeting-summary-20250116.txt b/storage/design/storage-group-meetings/storage-group-meeting-summary-20250116.txt new file mode 100644 index 000000000..d1652d188 --- /dev/null +++ b/storage/design/storage-group-meetings/storage-group-meeting-summary-20250116.txt @@ -0,0 +1,70 @@ + +Meeting: Settlement Contract Verification & Completeness Constraints in Decentralized Storage + +Overview + +This meeting was a deep technical discussion refining the data requirements and verification steps for the settlement contract in the Orchid decentralized storage system. The team reviewed prior work, clarified outstanding issues, and identified necessary refinements to ensure both correctness (valid commitments) and completeness (ensuring a provider is actively storing data when required). The discussion focused on how commitments are verified on-chain, how to efficiently prove data commitments, and how to prevent equivocation. + +Key Technical Discussion Points + +1. Revisiting the Settlement Contract Data Requirements + • The discussion began with reviewing prior work on the settlement contract to ensure the necessary data is provided for verification. + • The group reviewed how the settlement verification involves checking multiple commitment periods, requiring proofs at different time intervals. + +2. Required Data for Settlement Verification + +The contract needs to validate four key components at the time of settlement: + 1. Commitment (P): The polynomial commitment proving a provider stored the expected data. + 2. Proof: The cryptographic proof to verify that the commitment corresponds to the claimed data. + 3. Evaluation Point (Z): The X-coordinate value where the polynomial is evaluated (linked to the commitment period). + 4. Evaluation Value (Y): The result of evaluating the polynomial at Z, representing the stored data. + +3. Ensuring Completeness: Matching Data to Rate Certificates + • After proving a commitment contains a valid subblock (Y), we must show it is part of the correct rate certificate (R). + • The rate certificate contains a root identifier, committing to the entirety of the stored data. + • The proof system needs to establish that: + • The subblock identifier in P matches an entry in R. + • R was included in Q, the client provider mapping polynomial, which the cohort uses for verification. + +4. Ensuring Non-Equivocation & Public Verification + • Cohort members can monitor Q in real-time and verify that a provider has committed to the correct client data. + • Without this, a provider could equivocate by committing different data privately (settlement contract) vs publicly (cohort monitoring). + • A KZG opening proof must link the values in P and Q to ensure the client list seen by the cohort is the same one verified on-chain. + +5. Storage of Random Beacon for Proof Verification + • The random beacon (used to select subblocks) must be available for verification at settlement time. + • Options discussed: + • Store beacons in contract storage (adds cost but ensures availability). + • Use block hashes, but they expire after 256 blocks (~64 hours), limiting historical verification. + • Other on-chain sources. + +6. Storage Costs & Feasibility + • Storing commitments & beacons in contract storage was suggested as the simplest default. + • Estimated storage cost for 32 bytes per commitment: 20,000 gas ($0.50 at current Ethereum gas prices). + • This was considered reasonable but requires further cost analysis. + * (rollups also possible) + + +Step-by-Step Verification Plan + 1. Validate that the commitment contains the correct data: + • Use KZG proof verification to confirm that  for a provided proof. + • Ensures that the provider actually committed to storing data. + 2. Ensure the data is part of the correct rate certificate: + • Verify that  appears in R (rate certificate commitment). + • Use a KZG proof on R to confirm that . + 3. Prove that the provider publicly advertised these commitments: + • Check that the subblock data in P corresponds to a client in Q. + • Ensure Q was published in the commitment period, preventing equivocation. + • Cohorts actively monitor Q to detect missing commitments. + 4. Verify the random beacon’s correctness: + • Ensure that the evaluation index (Z) corresponds to the correct time period. + • If using a stored beacon, directly reference it in contract storage. + • If using block hashes, verify they haven’t expired. + +Key Takeaways + • Settlement verification involves proving correctness (commitments to real data) and completeness (ensuring claims match public commitments). + • Cohorts play a crucial role in catching failures in real-time by monitoring Q. + • Commitments, proofs, and beacons must be carefully stored or verified to prevent equivocation. + • Storage costs are a concern but seem reasonable given the constraints. + • Next steps involve improving documentation, validating feasibility, and preparing for final implementation decisions. + diff --git a/storage/design/storage-group-meetings/storage-group-meeting-summary-20250122.txt b/storage/design/storage-group-meetings/storage-group-meeting-summary-20250122.txt new file mode 100644 index 000000000..db93dbdc9 --- /dev/null +++ b/storage/design/storage-group-meetings/storage-group-meeting-summary-20250122.txt @@ -0,0 +1,36 @@ + +Meeting Title: Refining Data Commitment, Verification, and Settlement in Decentralized Storage + +The meeting focused on refining and clarifying the settlement process in the Orchid Storage protocol, particularly the commitments, verification mechanisms, and corresponding notation used in the diagrams. The discussion aimed to ensure correctness, eliminate ambiguity, and align terminology with the litepaper. + +Key Technical Discussions and Outcomes + +1. Refinement of Diagrams and Notation + • Client Blocks vs. Provider Blocks: There was debate on whether to retain “client block” terminology or simplify to “blocks.” The consensus was to remove “client-” as a prefix and just refer to them as “blocks” while ensuring clarity through visual distinctions. + • Clarifying Diagram Elements. + +2. Data Commitment and Verification in Settlement + • KZG Verification of a Single Subblock: The fundamental verification operation in the settlement involves checking a single subblock’s correctness. + • Commitment to Q and Its Role: + • Q commitments serve as proof that a provider was participating in the protocol at the designated time. + • There was discussion about whether Q should explicitly link to P, concluding that P and Q share indexing constraints but don’t necessarily require an explicit additional linking mechanism. + +3. Establishing the Link Between P and R + • Ensuring Subblock Selection Integrity: + • The selected subblock in P must correspond to the same subblock in R to establish the correctness of the commitment. + • Providers need to submit an opening of R to verify that their selected subblock was included in the rate certificate’s corresponding data. + +4. Role of the Rate Certificate in Settlement + • Provider Identity and Payment Mapping: + • The rate certificate is designed to include a field identifying the provider, ensuring payments go to the intended entity. + • Providers must prove their data aligns with what the rate certificate earmarks for payment. + • Ensuring Rate Certificate Correspondence: + • The verification mechanism must ensure that the subblock committed to in P aligns with the erasure-coded data referenced in R. + +Next Steps + • Refine diagrams to make distinctions between different data sources explicit. + • Standardize notation for polynomial evaluations and commitment indexing. + • Review structural flow of the settlement contract to ensure clarity and correctness. + • Finalize how providers associate commitments with rate certificates in an unambiguous way. + +This meeting advanced the technical clarity of the settlement verification mechanism, refining both conceptual and practical aspects of provider commitments in the Orchid Storage protocol.