Merge pull request #217 from awslabs/v1.1.1

v1.1.1 commit

.gitignore

@@ -37,10 +37,16 @@
 # build
 source/backend/functions/lambda-layers/aws_sdk/python
 source/backend/functions/lambda-layers/cr_helper/python
-!source/backend/functions/lambda-layers/decorators/python/decorators.py
-!source/backend/functions/lambda-layers/boto_utils/python/boto_utils.py
-# source/backend/functions/lambda-layers/decorators/python/*
-# source/backend/functions/lambda-layers/boto_utils/python/*
+source/backend/functions/lambda-layers/decorators/python/*
+source/backend/functions/lambda-layers/boto_utils/python/*
 source/backend/functions/cleanup-bucket/package
+source/backend/functions/cost-parser/src/setting-up-athena-integration.md
 
 local-deploy-perspective.sh
+
+# codebuild local testing
+codebuild_build.sh
+source/backend/functions/cost-parser/test/local-deploy.sh
+source/backend/functions/cost-parser/test/local-invoke-read-s3.sh
+source/backend/functions/cost-parser/test/local-invoke-service-cost.sh
+source/backend/functions/cost-parser/test/local-invoke.sh

CHANGELOG.md

@@ -5,7 +5,27 @@ All notable changes to this project are documented in this file.
 Based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.1.1] - 2021-09-28
+
+### Added
+
+- Missing icons for MariaDB, Aurora, SQL-Server RDS types.
+- OpensearchMultiAz parameter to CloudFormation template to set Amazon OpenSearch Service up with a single instance.
+
+### Changed
+
+- Migrated from Lambda@Edge to CloudFront Functions to handle secure headers for web requests to the frontend.
+- References to Amazon Elasticsearch Service to Amazon OpenSearch Service
+
+### Fixed
+
+- Fixed a bug causing a blank screen when expanding nodes whilst filters are enabled - https://github.com/awslabs/aws-perspective/issues/201
+- Fixed a bug that meant the time period for cost report queries was not persisted - https://github.com/awslabs/aws-perspective/issues/200
+- Fixed a bug that could result in python files being incorrectly excluded - https://github.com/awslabs/aws-perspective/issues/64
+- A bug causing some resource types to throw an exception when clicking "Show more details"
+
 ## [1.1.0] - 2021-08-26
+
 ### Added
 
 - Support for newer ECS task ARNs
@@ -49,4 +69,4 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [1.0.0] - 2020-09-21
 
-- Initial release
+- Initial release

README.md

@@ -1,8 +1,8 @@
-# AWS Perspective (v1.1.0)
+# AWS Perspective (v1.1.1)
 
 AWS Perspective is a tool that quickly visualizes AWS Cloud workloads as architecture diagrams. You can use the solution to build, customize, and share detailed workload visualizations based on live data from AWS. This solution works by maintaining an inventory of the AWS resources across your accounts and Regions, mapping relationships between them, and displaying them in a web user interface (web UI).
 
-v1.1.0 brings a new feature that uses AWS Cost & Usage Reports (AWS CUR) to help you identify AWS resources that have incurred a cost. You can build architecture diagrams displaying this cost information and generate Cost Reports which graph the overall cost of your workload over a configurable time period. These reports can be exported in CSV format.
+v1.1.1 brings a new feature that uses AWS Cost & Usage Reports (AWS CUR) to help you identify AWS resources that have incurred a cost. You can build architecture diagrams displaying this cost information and generate Cost Reports which graph the overall cost of your workload over a configurable time period. These reports can be exported in CSV format.
 
 The new release includes many UX improvements among them a Grouped Resources ** view which displays an inventory of your workloads. Resource type coverage has also been improved with Perspective now supporting your Amazon Redshift Clusters. 
 
@@ -217,9 +217,10 @@ Parameters required by the template:
 * **OptOutOfSendingAnonymousUsageMetrics** - Yes/No depending on whether you are happy to send anonymous usage metrics back to AWS.
 * **CreateNeptuneReplica** - Yes/No depending on whether you want a read-replica created for Amazon Neptune. Note, that this will increase the cost of running the solution.
 * **NeptuneInstanceClass** - Select from a range of instance types that will be provisioned for the Amazon Neptune database. Note, the selection could increase the cost associated with running the solution.
-* **ElasticsearchInstanceType** - Select the instance type that will be provisioned for the Amazon ElasticSearch Domain.
+* **OpensearchInstanceType** - Select the instance type that will be provisioned for the Amazon ElasticSearch Domain.
 * **CreateAPIGatewayCloudWatchLogsRole** - If set to Yes, the solution creates a role and overwrites the existing APIGatewayCloudWatchLogsLogsRole property. Set to No if you already have an existing role set.
 * **AthenaWorkgroup** - The Workgroup that will be used to issue the Athena query when the Cost feature is enabled.
+* **OpensearchMultiAz** - Choose whether to create an Opensearch cluster that spans multiple Availability Zone. Choosing Yes improves resilience; however, increases the cost of this solution.
 
 **Note** - You will need to deploy in the same account and region as the S3 bucket that the deployment artefacts are uploaded to.
 
@@ -301,6 +302,7 @@ curl -X POST "https://${DRAWIO_API_URL}.execute-api.${AWS_REGION}.amazonaws.com/
     --data-raw '{"elements":{"nodes":[], "edges": []}}'
 ```
 
+
 ##### Response
 
 You will receive a URL that when clicked will open up DrawIO in the browser and show your graph.
@@ -309,7 +311,17 @@ You will receive a URL that when clicked will open up DrawIO in the browser and
 
 ## Collecting Anonymous Operational Metrics
 
-This solution collects anonymous operational metrics to help AWS improve the quality of features of the solution. For more information, including how to disable this capability, please see the [Implementation Guide](https://docs.aws.amazon.com/solutions/latest/aws-perspective/appendix-g-collection-of-operational-metrics.html).
+This solution collects anonymous operational metrics to help AWS improve the quality of features of the solution. For more information, including how to disable this capability, please see the [Implementation Guide](https://docs.aws.amazon.com/solutions/latest/aws-perspective/collection-of-operational-metrics.html).
+
+## Acknowledgements
+
+AWS Perspective is able to generate its architecture diagrams thanks to these libraries developed and maintained by the [Info Visualization Research Lab](https://www.cs.bilkent.edu.tr/~ivis/) over at Bilkent University:
+
+* [cytoscape.js-fcose](https://github.com/iVis-at-Bilkent/cytoscape.js-fcose)
+* [cytoscape.js-grid-guide](https://github.com/iVis-at-Bilkent/cytoscape.js-grid-guide)
+* [cytoscape.js-context-menus](https://github.com/iVis-at-Bilkent/cytoscape.js-context-menus)
+* [cytoscape.js-expand-collapse](https://github.com/iVis-at-Bilkent/cytoscape.js-expand-collapse)
+
 
 Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
 

deployment/build-s3-dist.sh

@@ -124,19 +124,22 @@ echo "[Rebuild] Layers"
 echo "------------------------------------------------------------------------------"
 cd $source_dir/backend/functions/lambda-layers
 for i in `ls -d */ | sed 's#/##'` ; do
-  pip install -r $i/requirements.txt -t $i/python/
+  mkdir $i/python
+  [ -f "$i/$i.py" ] && cp $i/$i.py $i/python
+  [ -f "$i/requirements.txt" ] && pip install -r $i/requirements.txt -t $i/python/
   cd $i
   zip  -q  -r9 ../$i.zip ./python
   cd ..
+  rm -rf $i/python
 done
 cp ./*.zip $build_dist_dir/
 
 echo "------------------------------------------------------------------------------"
-echo "[Rebuild] Secured Edge Lambda"
+echo "[Rebuild] HSTS CloudFront Function"
 echo "------------------------------------------------------------------------------"
 cd $source_dir/backend/functions/secured-edge
-mkdir dist && zip -q -r9 dist/create_regional_edge_lambda.zip create_regional_edge_lambda.py
-cp ./dist/create_regional_edge_lambda.zip $build_dist_dir/create_regional_edge_lambda.zip
+rm -rf dist && mkdir dist && cp cff-hsts.js dist/cff-hsts.js
+cp ./dist/cff-hsts.js $build_dist_dir/cff-hsts.js
 
 echo "------------------------------------------------------------------------------"
 echo "[Rebuild] Cleanup Bucket Lambda"

deployment/perspective-setup.yaml

@@ -28,20 +28,20 @@ Parameters:
   AlreadyHaveConfigSetup:
     Type: String
     Default: 'No'
-    Description: 'Is AWS Config set-up within this Account or Region?'
+    Description: 'Is AWS Config set-up within this Region?'
     AllowedValues:
       - 'No'
       - 'Yes'
-    ConstraintDescription: 'Please specify if this account has config set-up (Yes / No)'
-  CreateElasticsearchServiceRole:
+    ConstraintDescription: 'Please specify if this Region has AWS Config set-up (Yes / No)'
+  CreateOpensearchServiceRole:
     Type: String
     Default: 'Yes'
-    Description: 'Do you need an ElasticSearch Service Role to be created?
+    Description: 'Do you need an OpenSearch Service Role to be created?
     You can check for a Role called AWSServiceRoleForAmazonElasticsearchService in your account. If it exists then you do NOT need one creating'
     AllowedValues:
       - 'No'
       - 'Yes'
-    ConstraintDescription: 'Please specify if this account has config set-up (Yes / No)'
+    ConstraintDescription: 'Please specify if this account has AWS Config set-up (Yes / No)'
   AdminUserEmailAddress:
     Type: String
     AllowedPattern: "^[\\w!#$%&’*+/=?`{|}~^-]+(?:\\.[\\w!#$%&’*+/=?`{|}~^-]+)*@(?:[a-zA-Z0-9-]+\\.)+[a-zA-Z]{2,6}$"
@@ -71,8 +71,8 @@ Parameters:
       - 'Yes'
     Default: 'No'
     Description: If you would like a read replica creating in a separate AZ. Please select 'Yes'. This will increase the cost of running the solution.
-  ElasticsearchInstanceType:
-    Description: The instance type for Elasticsearch data nodes
+  OpensearchInstanceType:
+    Description: The instance type for OpenSearch data nodes
     Type: String
     Default: m6g.large.elasticsearch
     AllowedValues:
@@ -136,7 +136,17 @@ Parameters:
       - i3.4xlarge.elasticsearch
       - i3.8xlarge.elasticsearch
       - i3.16xlarge.elasticsearch
-
+
+
+  OpensearchMultiAz:
+    Description: Deploys the OpenSearch cluster across two Availability Zones (AZs) in the same region to prevent
+      data loss and minimize downtime in the event of node or data center failure. This will increase the cost of running the solution
+    Type: String
+    Default: "No"
+    AllowedValues:
+      - 'Yes'
+      - 'No'
+
   CreateAPIGatewayCloudWatchLogsRole:
     Type: String
     Default: "Yes"
@@ -390,6 +400,18 @@ Resources:
                   - cloudfront:TagResource
                   - cloudfront:GetDistribution
                   - cloudfront:CreateInvalidation
+                  - cloudfront:CreateFunction
+                  - cloudfront:DeleteFunction
+                  - cloudfront:DescribeFunction
+                  - cloudfront:GetFunction
+                  - cloudfront:ListFunctions
+                  - cloudfront:UpdateFunction
+                  - cloudfront:TestFunction
+                  - cloudfront:PublishFunction
+                  - cloudfront:GetDistribution
+                  - cloudfront:GetDistributionConfig
+                  - cloudfront:ListTagsForResource
+                  - cloudfront:UpdateDistribution
                 Resource: '*'
               - Effect: Allow
                 Action:
@@ -509,7 +531,8 @@ Resources:
           ANONYMOUS_METRIC_OPT_OUT: !Ref OptOutOfSendingAnonymousUsageMetrics
           NEPTUNE_INSTANCE_CLASS: !Ref NeptuneInstanceClass
           CREATE_READ_REPLICA: !Ref CreateNeptuneReplica
-          ELASTICSEARCH_INSTANCE_TYPE: !Ref ElasticsearchInstanceType
+          OPENSEARCH_INSTANCE_TYPE: !Ref OpensearchInstanceType
+          OPENSEARCH_MULTI_AZ: !Ref OpensearchMultiAz
           ACCOUNT_ID: !Ref AWS::AccountId
           API_GATEWAY: !Sub https://${PerspectiveWebRestAPI}.execute-api.${AWS::Region}.amazonaws.com/Prod
           SERVER_API_GATEWAY: !Sub https://${ServerGremlinAPI}.execute-api.${AWS::Region}.amazonaws.com/Prod/
@@ -530,7 +553,7 @@ Resources:
           DISCOVERY_ARN: !GetAtt PerspectiveDiscoveryRole.Arn
           CONFIG_AGGREGATOR: !Sub aws-perspective-${AWS::Region}-${AWS::AccountId}-aggregator
           EXISTING_CONFIG: !Ref AlreadyHaveConfigSetup
-          CREATE_ES_SERVICE_ROLE: !Ref CreateElasticsearchServiceRole
+          CREATE_OPENSEARCH_SERVICE_ROLE: !Ref CreateOpensearchServiceRole
           ATHENA_WORKGROUP: !Ref AthenaWorkgroup
           APPSYNC_API_ARN: !GetAtt PerspectiveAppSyncApi.Arn
           APPSYNC_API_ID: !GetAtt PerspectiveAppSyncApi.ApiId
@@ -1113,7 +1136,9 @@ Resources:
           - Sid: HttpsOnly
             Action: '*'
             Effect: Deny
-            Resource: !Sub arn:aws:s3:::${WebUIBucket}/*
+            Resource: 
+              - !Sub arn:aws:s3:::${WebUIBucket}/*
+              - !Sub arn:aws:s3:::${WebUIBucket}
             Principal: '*'
             Condition:
               Bool:
@@ -1154,7 +1179,9 @@ Resources:
           - Sid: HttpsOnly
             Action: '*'
             Effect: Deny
-            Resource: !Sub arn:aws:s3:::${AmplifyStorageBucket}/*
+            Resource: 
+              - !Sub arn:aws:s3:::${AmplifyStorageBucket}/*
+              - !Sub arn:aws:s3:::${AmplifyStorageBucket}
             Principal: '*'
             Condition:
               Bool:
@@ -1197,7 +1224,9 @@ Resources:
           - Sid: HttpsOnly
             Action: '*'
             Effect: Deny
-            Resource: !Sub arn:aws:s3:::${AccessLogsBucket}/*
+            Resource: 
+              - !Sub arn:aws:s3:::${AccessLogsBucket}/*
+              - !Sub arn:aws:s3:::${AccessLogsBucket}
             Principal: '*'
             Condition:
               Bool:
@@ -1238,7 +1267,9 @@ Resources:
           - Sid: HttpsOnly
             Action: '*'
             Effect: Deny
-            Resource: !Sub arn:aws:s3:::${CostAndUsageAthenaResultsBucket}/*
+            Resource: 
+              - !Sub arn:aws:s3:::${CostAndUsageAthenaResultsBucket}/*
+              - !Sub arn:aws:s3:::${CostAndUsageAthenaResultsBucket}
             Principal: '*'
             Condition:
               Bool:
@@ -1273,7 +1304,9 @@ Resources:
           - Sid: HttpsOnly
             Action: '*'
             Effect: Deny
-            Resource: !Sub arn:aws:s3:::${CostAndUsageReportBucket}/*
+            Resource: 
+              - !Sub arn:aws:s3:::${CostAndUsageReportBucket}/*
+              - !Sub arn:aws:s3:::${CostAndUsageReportBucket}
             Principal: '*'
             Condition:
               Bool:
@@ -1308,7 +1341,9 @@ Resources:
           - Sid: HttpsOnly
             Action: '*'
             Effect: Deny
-            Resource: !Sub arn:aws:s3:::${DiscoveryBucket}/*
+            Resource: 
+              - !Sub arn:aws:s3:::${DiscoveryBucket}/*
+              - !Sub arn:aws:s3:::${DiscoveryBucket}
             Principal: '*'
             Condition:
               Bool: