initial commit

.gitignore

@@ -0,0 +1 @@
+.idea/

README.md

@@ -1,2 +1,109 @@
 # efficiencydemos
+
 A set of DynamoDB demo scripts and sample data that illustrate the read and write cost of various data access patterns.
+
+## Intro
+
+Amazon DynamoDB is a powerful serverless database that offers virtually unlimited scale when used efficiently.
+The table structure and access patterns you choose have a big impact on DynamoDB's efficiency, and ultimately the read and write consumption that you are billed for.
+Knowing the best access patterns takes practice and experimentation.  
+In these labs, you can run a series of DynamoDB data operations using the AWS Command Line Interface, 
+and get immediate visibility into the cost and effectiveness of your calls.
+
+## Pre-requisites
+
+ * An AWS Account with administrator access
+ * The [AWS CLI](https://aws.amazon.com/cli/) setup and configured
+ * [Node.JS](https://nodejs.org/en/download/) for loading the sample data
+ * A bash command-line environment such as Mac Terminal or Windows 10 bash shell
+
+ *If you do not have an AWS account, you can run [DynamoDB Local](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html) on your laptop instead.*
+
+## Scenario
+
+We are consultants who have been hired to build a shopping cart for an E-commerce website.
+Each cart that is created has a unique ID, such as Cart1 or Cart2.  
+Within each cart, one or many products can exist.  
+Each product is identified via IDs such as Product100, Product200, etc.
+A one-to-many pattern of cart to products is modeled in a DynamoDB table called **ShoppingCart**.
+
+![Cart1](https://dynamodb-images.s3.amazonaws.com/img/cart1.png)
+
+This table also contains other types of items, such as Customer details and Product details.
+The Description attribute for many of these items is a large string of 20,000 bytes, representing a typical JSON document payload.
+
+![Product100](https://dynamodb-images.s3.amazonaws.com/img/product100.png)
+
+![Customer](https://dynamodb-images.s3.amazonaws.com/img/customer.png)
+
+We will take a tour of the DynamoDB read and write operations using simple shell scripts that call the AWS CLI.
+
+## Consumed Capacity
+
+![Consumed Capacity](https://dynamodb-images.s3.amazonaws.com/img/consumed.png)
+
+A nice feature of calls to DynamoDB is that you can request a summary of the capacity consumed by your call.
+Even if your call is only sending or receiving a small amount of data, it may be consuming a much larger amount of Read Capacity or Write Capacity.
+
+Capacity is measured in Read Capacity Units and Write Capacity Units (RCUs and WCUs).
+Each RCU represents the size of data read, rounded up to the nearest 4 KB unit. 
+Each WCU represents the size of data written, rounded up to the nearest 1 KB unit.
+
+Be aware that these capacity units are measured according to how much data DynamoDB reads and writes internally, which may be more than is actually sent or received by the client.
+
+Data operations to DynamoDB have an optional "Return Consumed Capacity" parameter, where you can specify either TOTAL or INDEXES.   Specifying INDEXES will provide a total along with a breakdown of capacity consumed by indexes.
+
+For reads, you will also see two counts returned, the Scanned Count and the Returned Count.
+The Scanned Count is the total number of Items (rows) read by the DynamoDB engine, while the Returned Count is the number returned to the user.
+
+
+
+## Setup Steps
+
+1. Clone this repository to a folder on your laptop, or download to a working folder via the green button above.
+1. From your command prompt, navigate to the cart folder: ```cd cart```
+1. You may wish to run ```export PATH=$PATH:$(pwd)``` so that scripts in your current folder will be added to your path.
+1. Verify the AWS CLI is setup and running by running ```aws sts get-caller-identity``` 
+and ```aws dynamodb describe-limits```.  You should see no errors.
+1. Verify your AWS CLI is pointing to default region **us-east-1** (N. Virginia) by running ```aws configure``` and pressing enter four times.  You may enter ```us-east-1``` on the third prompt if necessary.
+
+#### Creating the ShoppingCart table
+1. Run ```recreate.sh``` to create your table.  Please ignore the ResourceNotFoundException that appears the first time, as it attempts to delete the table should it already exist.
+1. Run ```node load``` which will write the data to your new **ShoppingCart** table from the [CartData.csv](./cart/CartData.csv) file.
+
+## Running the Demos
+
+This sequence of commands helps illustrate the pros and cons of various access patterns.
+As you run each example, try and estimate the capacity you think will be consumed, before running each commmand. 
+
+
+| Demo | Command |
+| --- | --- |
+| Scan | ```scan.sh``` |
+| Scan with filter | ```scanfilter.sh Cart1``` |
+| GetItem Eventual Consistency | ```get.sh Cart1 Product100``` |
+| GetItem with projection expression | ```get.sh Cart1 Product100 Price``` |
+| GetItem with Strong Consistency | ```get.sh Cart1 Product100 Price STRONG``` |
+| BatchGetItem | ```batchget.sh``` |
+| Query | ```query.sh Cart1``` |
+| Query with Filter | ```queryfilter.sh Cart1 Orange``` |
+| Query with Sort Key Expression | ```queryfilter.sh Cart1 Product400``` |
+| Delete Item | ```delete.sh Cart1 Product500``` |
+| Put Item | ```put.sh``` <br/> *writes new item Cart7 Product700* |
+| UpdateItem | 1. ```update.sh Cart7 Product700 Price 22.33``` <br/> 2. ```update.sh Cart7 Product700 CustomerDescription MuchSmaller``` <br/> 3. ```update.sh Cart7 Product700 Price 44.55``` |
+| Add New GSI (ProductName + Price) | ```addgsi.sh``` <br/> Wait a couple minutes for **GSI-ProductPrice** to be created. |
+| UpdateItem GSI key attribute | ```update.sh Cart1 Product100 Price 12.34``` |
+| UpdateItem non-GSI key attribute | ```update.sh Cart1 Product100 Qty 5``` |
+| UpdateItem with cost and non-GSI attribute | ```update.sh Customer "John Stiles" Address "100 Main Street"``` |
+| Scan a GSI (works the same as base table) | ```scangsi.sh GSI-ProductPrice``` |
+| Query a GSI (works the same as base table) | ```querygsi.sh GSI-ProductPrice Turnip``` |
+| Write to a GSI | *N/A <br/> you can only write to base table!* |
+| Get Item from GSI | *N/A <br/> you can only get item from base table!* |
+
+
+## Next Steps
+
+Please contribute to this code sample by issuing a Pull Request or creating an Issue.
+
+Share your feedback at [@robmccauley](https://twitter.com/robmccauley)
+

cart/addgsi.sh

@@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+
+REGION=us-east-1
+TABLENAME=ShoppingCart
+
+# ENDPOINTURL=http://localhost:8000
+ENDPOINTURL=https://dynamodb.$REGION.amazonaws.com
+
+
+aws dynamodb update-table --table-name "ShoppingCart" \
+    --attribute-definitions AttributeName=ProductName,AttributeType=S AttributeName=Price,AttributeType=N \
+    --global-secondary-index-updates \
+        "[{\"Create\": \
+            {\"IndexName\": \"GSI-ProductPrice\", \
+             \"KeySchema\":[ \
+                 {\"AttributeName\":\"ProductName\",\"KeyType\":\"HASH\"}, \
+                 {\"AttributeName\":\"Price\",\"KeyType\":\"RANGE\"} \
+                 ], \
+        \"Projection\":{\"ProjectionType\":\"ALL\"}}}]" \
+    --region $REGION \
+    --endpoint-url $ENDPOINTURL \
+    --output json --query '{"New Table":TableDescription.TableName, "Status   ":TableDescription.TableStatus }'
+
+

cart/batchget.sh

@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+
+REGION=us-east-1
+TABLENAME=ShoppingCart
+ARG1="$1"
+
+
+# ENDPOINTURL=http://localhost:8000
+ENDPOINTURL=https://dynamodb.$REGION.amazonaws.com
+
+
+ITEMSFILE=$ARG1
+
+if [ -z "$ITEMSFILE" ]
+then
+      ITEMSFILE="cartproduct.json"
+      echo Querying $TABLENAME for Partition Key $PK
+fi
+
+
+aws dynamodb batch-get-item --region $REGION --endpoint-url $ENDPOINTURL \
+    --request-items file://$ITEMSFILE \
+    --return-consumed-capacity 'TOTAL' \
+    --output json \
+    --query '{"Item Keys": Responses.ShoppingCart[*].[*],  "Consumed RCUs ":ConsumedCapacity}'

cart/cartcustomer.json

@@ -0,0 +1,14 @@
+{
+  "ShoppingCart": {
+    "Keys": [
+      {
+        "PK": {"S": "Cart3"},
+        "SK": {"S": "Product300"}
+      },
+      {
+        "PK": {"S": "Customer"},
+        "SK": {"S": "John Stiles"}
+      }
+    ]
+  }
+}

cart/cartproduct.json

@@ -0,0 +1,14 @@
+{
+  "ShoppingCart": {
+    "Keys": [
+      {
+        "PK": {"S": "Cart3"},
+        "SK": {"S": "Product300"}
+      },
+      {
+        "PK": {"S": "Product300"},
+        "SK": {"S": "Product300"}
+      }
+    ]
+  }
+}

cart/delete.sh

@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+
+REGION=us-east-1
+TABLENAME=ShoppingCart
+ARG1="$1"
+ARG2="$2"
+ARG3="$3"
+
+
+# ENDPOINTURL=http://localhost:8000
+ENDPOINTURL=https://dynamodb.$REGION.amazonaws.com
+
+PK=$ARG1
+SK=$ARG2
+
+if [ -z "$SK" ]
+then
+    SK="Product500"
+
+    if [ -z "$PK" ]
+    then
+          PK="Customer5"
+          echo Deleting Item $PK:$SK
+    fi
+fi
+
+NEWITEM='{"PK":{"S":"'$PK'"},"SK":{"S":"'$SK'"},"Qty":{"N":"15"}}'
+
+# echo Connecting to $ENDPOINTURL
+# echo Scanning $TABLENAME with a filter on $SK
+
+aws dynamodb delete-item --region $REGION --endpoint-url $ENDPOINTURL \
+    --table-name $TABLENAME \
+    --key '{"PK":{"S":"'$PK'"},"SK":{"S":"'$SK'"}}' \
+    --return-consumed-capacity 'TOTAL' \
+    --output json \
+    --query '{"Consumed WCUs ":ConsumedCapacity}'
+
+#    --projection-expression '#a' \
+#    --expression-attribute-names '{ "#a": "'$RETURNATTR'" }'  \

cart/get.sh

@@ -0,0 +1,67 @@
+#!/usr/bin/env bash
+
+REGION=us-east-1
+TABLENAME=ShoppingCart
+ARG1="$1"
+ARG2="$2"
+ARG3="$3"
+ARG4="$4"
+
+# ENDPOINTURL=http://localhost:8000
+ENDPOINTURL=https://dynamodb.$REGION.amazonaws.com
+
+
+PK=$ARG1
+SK=$ARG2
+RETURNATTR=$ARG3
+CONSISTENCY=$ARG4
+
+PROJECTIONEXPRESSION=""
+CRFLAG="--no-consistent-read"
+
+if [ ! -z "$CONSISTENCY" ] && [ "STRONG" = $CONSISTENCY ]
+then
+    CRFLAG="--consistent-read"
+fi
+
+
+if [ -z "$RETURNATTR" ] || [ "ALL" = $RETURNATTR ] || [ "All" = $RETURNATTR ] || [ "all" = $RETURNATTR ]
+then
+
+    if [ -z "$SK" ]
+    then
+        SK="Product100"
+
+        if [ -z "$PK" ]
+        then
+              PK="Cart1"
+              echo Getting Item $PK:$SK
+        fi
+    fi
+else
+    PROJECTIONEXPRESSION="--projection-expression $RETURNATTR"
+fi
+
+
+
+echo $CRFLAG : $CONSISTENCY
+echo $PROJECTIONEXPRESSION
+
+# PROJECTIONEXPRESSION='--projection-expression "#a" --expression-attribute-names  "{ \"#a\": \"DateOrdered\" }"  '
+
+
+# echo Connecting to $ENDPOINTURL
+# echo Scanning $TABLENAME with a filter on $SK
+
+aws dynamodb get-item --region $REGION --endpoint-url $ENDPOINTURL \
+    --table-name $TABLENAME \
+    --key '{"PK":{"S":"'$PK'"},"SK":{"S":"'$SK'"}}' \
+    $PROJECTIONEXPRESSION \
+    $CRFLAG \
+    --return-consumed-capacity 'TOTAL' \
+    --output json \
+    --query '{"Item": Item,  "Consumed RCUs ":ConsumedCapacity.CapacityUnits}'
+
+
+#    --projection-expression '#a' \
+#    --expression-attribute-names '{ "#a": "'$RETURNATTR'" }'  \