Create new REST API to support HttpKVStore

[sergiupopescu199]

Description of change

• created a new crate iota-rest-kv to accomodate the REST API implementation
• refactored the KvStoreWorker in iota-data-ingestion to reflect the changes made in these PRs:
  • refactor(storage,json-rpc)!: remove support for querying checkpoint contents by digest #5215
  • feat(storage): query events by transaction digest #5227
  • refactor(storage,json-rpc)!: remove obsolete kv queries #5259
• added Dockerfile and docker-compose file for the iota-rest-kv

Links to any relevant issues

fixes #5149

Type of change

• Enhancement (a non-breaking change which adds functionality)

How the change has been tested

The REST API server relies on several external components

• A running Kv Store worker which saves historical data from checkpoints in ingestion pipeline
• Cloud components (e.g AWS S3 & DynamoDB) simulated trough localstack

Start Localstack

# Start LocalStack
docker run -d \
--name localstack \
--network=iota-network \
-p 4566:4566 \
-e SERVICES=dynamodb,s3 \
-e DEBUG=1 \
-e AWS_ACCESS_KEY_ID=test \
-e AWS_SECRET_ACCESS_KEY=test \
-e AWS_DEFAULT_REGION=us-east-1 \
localstack/localstack

# Wait for services to be ready
sleep 5

# Create S3 bucket
aws --profile localstack \
s3 mb s3://iota-storage-bucket

# Create DynamoDB table
aws --profile localstack \
dynamodb create-table \
--table-name checkpoint-progress \
--attribute-definitions AttributeName=task_name,AttributeType=S \
--key-schema AttributeName=task_name,KeyType=HASH \
--provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5

aws --profile localstack \
dynamodb create-table \
--table-name iota-storage \
--attribute-definitions \
    AttributeName=digest,AttributeType=B \
    AttributeName=type,AttributeType=S \
--key-schema \
    AttributeName=digest,KeyType=HASH \
    AttributeName=type,KeyType=RANGE \
--provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5 \

start the IOTA Node

iota start --force-regenesis

KvStoreWorker in data ingestion pipeline

• Modify the KvStoreWorker process_checkpoint function by adding some debug prints to visualize the various transactions digests & checkpoint digests it saves into DynamoDB
• Go todev-tools/iota-data-ingestion directory
  • modify the docker-compose.yaml file to use the kv_store_config.yaml config file
  • build and run the docker-compose file in dev-tools/iota-data-ingestion more info in the README.md
• By inspecting the container logs we'll see the digests it prints after out change, it later will be used for testing

start the Kv Store REST API

• Go todev-tools/iota-rest-kv directory
• build and run the docker-compose file in dev-tools/iota-rest-kv more info in the README.md

Execute tests using HttpKvStore

#[tokio::test]
async fn test_rest_api_responses() {
   // get data from Rest Api
   let kv = HttpKVStore::new("http://0.0.0.0:3555").unwrap();

   let tx_digest =
       TransactionDigest::from_str("H2tetNL3CfroDF3iJNA7wFo6oRQiJedGTeykZi6HAGqP").unwrap();

   let (tx, effects) = kv.multi_get(&[tx_digest], &[tx_digest]).await.unwrap();

   assert!(tx.iter().all(|a| a.is_some()));
   assert!(effects.iter().all(|a| a.is_some()));

   let events = kv
       .multi_get_events_by_tx_digests(&[tx_digest])
       .await
       .unwrap();
   assert!(events.iter().all(|a| a.is_some()));

   let ob = kv
       .get_object(
           ObjectID::from_str(
               "0x0000000000000000000000000000000000000000000000000000000000000001",
           )
           .unwrap(),
           SequenceNumber::from_u64(1),
       )
       .await
       .unwrap();

   assert!(ob.is_some());

   let ch_digest =
       CheckpointDigest::from_str("4fTr7zjbb3KssdtrwdCb3RJ81Ha9JJjURP4seaAVsF5C").unwrap();

   let (ch_sum, ch_contents, ch_sum_dugest) = kv
       .multi_get_checkpoints(&[0], &[0], &[ch_digest])
       .await
       .unwrap();

   assert!(ch_sum.iter().all(|a| a.is_some()));
   assert!(ch_contents.iter().all(|a| a.is_some()));
   assert!(ch_sum_dugest.iter().all(|a| a.is_some()));

   let chk_seq_num = kv
       .multi_get_transactions_perpetual_checkpoints(&[
           tx_digest,
           TransactionDigest::from_str("5e9a72Ra4dhbaHGtcyXpWiheDK7wiZVbkxq8ABJT2xET").unwrap(),
           TransactionDigest::from_str("3eRDMTvd43VxyfZbjMRC3qNrR476FZd1xqz7paxxo8nF").unwrap(),
           TransactionDigest::from_str("7JC2CX3kEgURJKxdGWkNEkxre8RJ4cbeEXD2mtoV24HE").unwrap(),
       ])
       .await
       .unwrap();

   assert_eq!(chk_seq_num, vec![Some(0), Some(52), Some(6), Some(47)])
}

Change checklist

• I have followed the contribution guidelines for this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

4 Skipped Deployments

Name	Status	Preview	Comments	Updated (UTC)
apps-backend	⬜️ Ignored (Inspect)	Visit Preview		Feb 7, 2025 3:58pm
apps-ui-kit	⬜️ Ignored (Inspect)	Visit Preview		Feb 7, 2025 3:58pm
rebased-explorer	⬜️ Ignored (Inspect)	Visit Preview		Feb 7, 2025 3:58pm
wallet-dashboard	⬜️ Ignored (Inspect)	Visit Preview		Feb 7, 2025 3:58pm

[muXxer]

why are those those crates not part of the root workspace btw?
I can find them in several crates. Maybe @iotaledger/dev-tools knows?

[sergiupopescu199]

I was wondering the same thing, if needed I can make them part of the root workspace, let me know @muXxer @iotaledger/dev-tools

[kodemartin]

Hey @sergiupopescu199, I would like to ask you to split the PR into smaller patches distinguishing between scopes:

• One for the refactoring of iota-data-ingestion
• One for the REST service
• One for the docker configuration.

This will allow for more focused and detailed reviews on each scope, by the respective stakeholders each time.