Merge pull request #7 from develeap/feature/digger

Feature/digger

.github/workflows/digger_workflow.yml

@@ -1,3 +1,4 @@
+---
 name: Digger Pull Request Workflow
 on:
   workflow_dispatch:
@@ -25,16 +26,22 @@ jobs:
     runs-on: ubuntu-latest
     # Permissions for OIDC with AWS
     permissions:
-      contents: write # required to merge PRs
-      id-token: write # required for workload-identity-federation
-      pull-requests: write # required to post PR comments
-      statuses: write # required to validate combined PR status
+      contents: write  # required to merge PRs
+      id-token: write  # required for workload-identity-federation
+      pull-requests: write  # required to post PR comments
+      statuses: write  # required to validate combined PR status
     steps:
       - name: checkout
         uses: actions/checkout@v4
       - name: echo env
         run: env | sort
         shell: bash
+      - name: Setup Infracost
+        uses: infracost/actions/setup@v3
+        # See https://github.com/infracost/actions/tree/master/setup for other inputs
+        # If you can't use this action, use Docker image infracost/infracost:ci-0.10
+        with:
+          api-key: ${{ secrets.INFRACOST_API_KEY }}
       - name: digger run
         uses: diggerhq/[email protected]
         with:
@@ -71,7 +78,6 @@ jobs:
           # AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
           # DIGGER_ACCESS_KEY_ID: ${{ secrets.DIGGER_ACCESS_KEY_ID }}
           # DIGGER_SECRET_ACCESS_KEY: ${{ secrets.DIGGER_SECRET_ACCESS_KEY }}
-
 ##########
 # DEBUG  #
 ##########

.github/workflows/drift-detection.yml

@@ -0,0 +1,42 @@
+name: Digger Drift Detection
+on:
+  workflow_dispatch:
+  schedule: ## 12am daily.
+    - cron: 0 0 * * *
+env:
+  COLUMNS: 120
+jobs:
+  detect-drift:
+    runs-on: ubuntu-latest
+    steps:
+      - name: digger drift detection
+        uses: diggerhq/[email protected]
+        with:
+          mode: drift-detection
+          #########
+          # Setup  #
+          #########
+          setup-terragrunt: true
+          terragrunt-version: 0.54.12
+          # terraform-version: v1.5.5
+          # opentofu-version: v1.6.0-alpha3
+          setup-checkov: true
+          checkov-version: 2.3.360
+          ################
+          # AWS Settings  #
+          ################
+          setup-aws: true
+          aws-role-to-assume: arn:aws:iam::${{ secrets.ACCOUNT_ID }}:role/${{ secrets.ROLE_NAME }}
+          aws-region: ${{ secrets.AWS_REGION }}
+          ###################
+          # Digger Settings  #
+          ###################
+          disable-locking: true
+          digger-filename: digger.yaml
+          digger-hostname: https://cloud.digger.dev
+          digger-organisation: develeap
+          digger-token: ${{ secrets.DIGGER_TOKEN }}
+          drift-detection-slack-notification-url: ${{ secrets.DRIFT_DETECTION_SLACK_NOTIFICATION }}
+        env:
+          GITHUB_CONTEXT: ${{ toJson(github) }}
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

README.md

@@ -1,22 +1,33 @@
-# iac-terragrunt-template
-Repository to use as a template for future terragrunt (IaC tool) projects.
+<p align="center" width="100%">
+    <a href="https://www.develeap.com/">
+    <img src="https://github.com/develeap/iac-terragrunt-template/raw/feature/digger/docs/media/icon.png" alt="Develeap - we can take you there!" width="500" height="500">
+    </a>
+</p>
+
+[1]: https://www.develeap.com/
+
+## Introduction
+
+This repository is a template to be used for future terragrunt projects. It contains a basic structure for terragrunt projects and some basic modules to be used as examples.
 
 ## Table of Contents
+
 - [Introduction](#introduction)
 - [Prerequisites](#prerequisites)
 - [Getting Started](#getting-started)
+- [Cheat Sheet](docs/CheatSheet.md)
 - [Usage](#usage)
 - [Build the infrastructure folder structure](#build-the-infrastructure-folder-structure)
 - [Example](#example)
 - [Inheritance](#inheritance)
-- [Local vs Pipeline Architencture](#local-vs-pipeline-assume-role-diagram)
+- [Documentation](#documentation)
+  - [Local vs Pipeline Architencture](docs/local-vs-pipeline-assume-role-diagram.md)
+  - [Include Deep Dive](docs/include-deepdive.md)
 - [Contributing](#contributing)
 - [License](#license)
 
-## Introduction
-This repository is a template to be used for future terragrunt projects. It contains a basic structure for terragrunt projects and some basic modules to be used as examples.
-
 ## Prerequisites
+
 - [Terraform](https://www.terraform.io/downloads.html)
 - [Terragrunt](https://terragrunt.gruntwork.io/docs/getting-started/install/)
 - [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html)
@@ -25,143 +36,105 @@ This repository is a template to be used for future terragrunt projects. It cont
 - [terraform-docs](https://terraform-docs.io/)
 
 Can be install using:
+
 ```bash
 brew install terraform terragrunt awscli just graphviz terraform-docs
 ```
 
 ## Getting Started
+
 To get started, you can use this repository as a template and clone it to your local machine. After that, you can start creating your own modules and terragrunt configurations.
 
 ## Cheat Sheet
-Make sure to check the [Cheat Sheet](/CheatSheet.md) for some useful commands and tips.
+
+Make sure to check the [Cheat Sheet](docs/CheatSheet.md) for some useful commands and tips.
 
 ## Usage
+
 Use the Justfile provided in this repository to run various commands. You can see the available commands by running `just --list`.
 [![asciicast](https://asciinema.org/a/oy1cKWQRrgs5EUZDDvaOZBc2E.svg)](https://asciinema.org/a/oy1cKWQRrgs5EUZDDvaOZBc2E)
 
 ### Build the infrastructure folder structure
+
 To build the infrastructure folder structure, run the following command:
-*Make sure to set the desired account-ids, regions, environments in [build-folder-structure script](/scripts/build-folder-structure.sh)*
+_Make sure to set the desired account-ids, regions, environments in [build-folder-structure script](/scripts/build-folder-structure.sh)_
+
 ```bash
 just build-folder-structure
 ```
 
 #### Example:
+
 ```bash
 infrastructure-live
-├── 82281136
+├── 01234567890
 │   ├── account.hcl
 │   ├── dev
 │   │   ├── dev-1
-│   │   │   ├── us-east-1
-│   │   │   │   ├── compute
-│   │   │   │   │   └── compute
-│   │   │   │   ├── databases
-│   │   │   │   │   └── databases
-│   │   │   │   ├── network
-│   │   │   │   │   └── network
-│   │   │   │   ├── region.hcl
-│   │   │   │   ├── secrets
-│   │   │   │   │   └── secrets
-│   │   │   │   └── storage
-│   │   │   │       └── storage
-│   │   │   └── us-west-1
+│   │   │   └── il-central-1
 │   │   │       ├── compute
-│   │   │       │   └── compute
-│   │   │       ├── databases
-│   │   │       │   └── databases
+│   │   │       │   └── compute.hcl
+│   │   │       ├── database
+│   │   │       │   └── database.hcl
 │   │   │       ├── network
-│   │   │       │   └── network
+│   │   │       │   └── network.hcl
 │   │   │       ├── region.hcl
-│   │   │       ├── secrets
-│   │   │       │   └── secrets
+│   │   │       ├── secret
+│   │   │       │   └── secret.hcl
 │   │   │       └── storage
-│   │   │           └── storage
+│   │   │           └── storage.hcl
 │   │   └── env.hcl
 │   ├── prod
 │   │   ├── env.hcl
 │   │   └── prod-1
-│   │       ├── us-east-1
-│   │       │   ├── compute
-│   │       │   │   └── compute
-│   │       │   ├── databases
-│   │       │   │   └── databases
-│   │       │   ├── network
-│   │       │   │   └── network
-│   │       │   ├── region.hcl
-│   │       │   ├── secrets
-│   │       │   │   └── secrets
-│   │       │   └── storage
-│   │       │       └── storage
-│   │       └── us-west-1
+│   │       └── il-central-1
 │   │           ├── compute
-│   │           │   └── compute
-│   │           ├── databases
-│   │           │   └── databases
+│   │           │   ├── compute.hcl
+│   │           │   └── demo-ec2
+│   │           │       └── terragrunt.hcl
+│   │           ├── database
+│   │           │   └── database.hcl
 │   │           ├── network
-│   │           │   └── network
+│   │           │   └── network.hcl
 │   │           ├── region.hcl
-│   │           ├── secrets
-│   │           │   └── secrets
+│   │           ├── secret
+│   │           │   └── secret.hcl
 │   │           └── storage
-│   │               └── storage
+│   │               └── storage.hcl
+│   ├── provider.hcl
 │   └── stage
 │       ├── env.hcl
 │       └── stage-1
-│           ├── us-east-1
-│           │   ├── compute
-│           │   │   └── compute
-│           │   ├── databases
-│           │   │   └── databases
-│           │   ├── network
-│           │   │   └── network
-│           │   ├── region.hcl
-│           │   ├── secrets
-│           │   │   └── secrets
-│           │   └── storage
-│           │       └── storage
-│           └── us-west-1
+│           └── il-central-1
 │               ├── compute
-│               │   └── compute
-│               ├── databases
-│               │   └── databases
+│               │   └── compute.hcl
+│               ├── database
+│               │   └── database.hcl
 │               ├── network
-│               │   └── network
+│               │   └── network.hcl
 │               ├── region.hcl
-│               ├── secrets
-│               │   └── secrets
+│               ├── secret
+|               │   └── secret.hcl
 │               └── storage
-│                   └── storage
+│                   └── storage.hcl
 └── infrastructure.hcl
 
-44 directories, 41 files
+27 directories, 25 files
 ```
 
 ## Inheritance
-The folder structure is designed to inherit configurations from the parent folders. For example, the `compute_commons.hcl` file in the `dev-1` folder inherits configurations from the `compute_commons.hcl` file in the `dev` folder, which in turn inherits configurations from the `environment_commons.hcl` file in the `dev` folder, and so on.
 
-# local-vs-pipeline-assume-role-diagram
+The folder structure is designed to allow for inheritance of configurations. The `account.hcl` file is used to define the account specific configurations, while the `env.hcl` file is used to define the environment specific configurations. The `region.hcl` file is used to define the region specific configurations. And so forth.
+A single include block will be used to include all the configurations from the parent folders.
 
-When following best practices we generally use another IAM role to perform our actions. A genuine user can perform 
-
-```bash
-aws sts assume-role \
-    --role-arn <value> \
-    --role-session-name <value> \
-    --profile <value>
-```
-However for our pipelines, authentication is generally done using different auth methods, such as OIDC (Open ID Connect) where an organisation is authenticated against our cloud account. If our case, It's GitHub and AWS.
-
-Therefor, our pipeline with be assuming a role using a web identity, with is a little different. Also, there's no use of profile when running of a public or non-preconfigured private runner.
-
-The following diagram show what method I could to handle such case of working both locally (e.g. delicate actions such as `destroy` ) vs following GitOps using a pipeline (e.g. running `plan`  or `apply` ).
-
-![image info](./eraser-export-export-3-28-2024-11_03_36-AM/diagram-export-3-28-2024-11_03_36-AM.png)
+## Documentation
 
+Additional documentation can be found in the [docs](docs) folder.
 
 ## Contributing
+
 If you want to contribute to this repository, please create a pull request with your changes.
 
 ## License
-This repository is licensed under the MIT License. See the [LICENSE](LICENSE) file for more information.
 
+This repository is licensed under the MIT License. See the [LICENSE](LICENSE) file for more information.

digger.yaml

@@ -4,7 +4,6 @@
 # projects:
 # - name: production
 #   dir: prod
-
 # Or use the terragrunt_parsing workflow to generate the projects.
 generate_projects:
   terragrunt_parsing:
@@ -21,14 +20,17 @@ workflows:
         - init
         - run: checkov -d . --quiet --framework terraform --download-external-modules true
         - plan
+        - run: infracost breakdown --path=. | tee -a $DIGGER_OUT
+        - run: git checkout $DEFAULT_BRANCH && infracost breakdown --path . --format=json --out-file=$RUNNER_TEMP/infracost-base-${PROJECT_NAME}.json && git fetch origin $PR_BRANCH && git checkout $PR_BRANCH
+        - run: infracost diff --path=. --compare-to=$RUNNER_TEMP/infracost-base-${PROJECT_NAME}.json | tee -a $DIGGER_OUT
 
-  # To be run on only by invoking the /digger commands in a PR
-  # comments-only:
-  #   apply:
-  #     steps:
-  #       - hclfmt
-  #       - apply
-  #
-  #   destroy:
-  #     steps:
-  #       - destroy
+# To be run on only by invoking the /digger commands in a PR
+# comments-only:
+#   apply:
+#     steps:
+#       - hclfmt
+#       - apply
+#
+#   destroy:
+#     steps:
+#       - destroy