diff --git a/.gitmodules b/.gitmodules index a211319..e69de29 100644 --- a/.gitmodules +++ b/.gitmodules @@ -1,3 +0,0 @@ -[submodule "anthos-service-mesh-packages"] - path = anthos-service-mesh-packages - url = git@github.com:GoogleCloudPlatform/anthos-service-mesh-packages diff --git a/README.md b/README.md index e18b87f..c0f95cb 100644 --- a/README.md +++ b/README.md @@ -1,17 +1,25 @@ # Isidro (Chatbot) +> Isidro is an Anthos- and GKE-based microservices chatbot + Isidro includes: * Connectors to Slack and Mattermost for event subscription and response * Policy- and NLP-based workflow planning * Automated execution of workflows (e.g., provisioning, deployments, and test execution) * Automated presentation of data (e.g., deployment metrics, performance testing results, and spam trends) +* Cross-regional deployment + * Regional workload clusters in us-central1 (Council Bluffs, Iowa) and europe-west1 (Saint-Ghislain, Belgium) + * Regional MCI config cluster (GKE autopilot) in northamerica-northeast1 (Montreal, Canada) +* Security features like binary authorization, mTLS, workload identity, and network policies ## Prerequisites -1. Willingness and ability to run the system on Google Cloud Platform (other cloud providers are possible, but would require some hacking), including the following APIs and features: +1. APIs and features enabled on Google Cloud Platform: 1. API: Binary Authorization 1. API: Cloud KMS 1. API: Kubernetes Engine 1. API: GKE Hub + 1. API: Multi Cluster Ingress + 1. API: Multi-Cluster Service Discovery 1. API: Anthos 1. API: Anthos Service Mesh Certificate Authority 1. Anthos Feature: Service Mesh @@ -46,17 +54,28 @@ Navigate to the [provisioning/](provisioning/) directory, then set the `GOOGLE_A export GOOGLE_APPLICATION_CREDENTIALS=../isidro-provisioner.json ``` -Setup secondary IP ranges in the desired region and subnet (e.g., "gke-isidro-pods" and "gke-isidro-services"), then [run Terraform provisioning, with variable changes/overrides where required](provisioning/). Something like: +Setup secondary IP ranges in the desired regions and subnets, then [run Terraform provisioning, with variable changes/overrides where required](provisioning/). Something like: ```bash terraform init terraform apply \ - -var network=default \ - -var subnetwork=default \ - -var ip_range_pods="gke-isidro-pods" \ - -var ip_range_services="gke-isidro-services" + -var domain=isidro.example.com \ + -var ip_range_pods_primary="gke-isidro-pods" \ + -var ip_range_services_primary="gke-isidro-services" \ + -var ip_range_pods_secondary="gke-isidro-pods" \ + -var ip_range_services_secondary="gke-isidro-services" \ + -var ip_range_pods_config="gke-isidro-config-pods" \ + -var ip_range_services_config="gke-isidro-config-services" ``` -Configure kubectl to use the new cluster. Create a namespace, if a non-default namespace is desired. +Create kubecontext configurations for the three provisioned clusters: +```bash +gcloud container clusters get-credentials isidro-us --region us-central1 +gcloud container clusters get-credentials isidro-europe --region europe-west1 +gcloud container clusters get-credentials isidro-config --region northamerica-northeast1 +kubectl config rename-context gke_"$GOOGLE_PROJECT"_us-central1_isidro-us isidro-us +kubectl config rename-context gke_"$GOOGLE_PROJECT"_europe-west1_isidro-europe isidro-europe +kubectl config rename-context gke_"$GOOGLE_PROJECT"_northamerica-northeast1_isidro-config isidro-config +``` ### Certbot (for TLS) preparation @@ -76,42 +95,60 @@ gcloud iam service-accounts keys create isidro-certbot.json \ Add the service account key to Kubernetes as a secret: ```bash -kubectl create secret generic isidro-certbot-key --from-file isidro-certbot.json +for kubecontext in isidro-us isidro-europe +do + kubectl config use-context $kubecontext + kubectl create secret generic isidro-certbot-key --from-file isidro-certbot.json +done ``` -### Istio ingress gateway +### Enable GMP + +In the Google Cloud Console, enable Managed Prometheus for the US and Europe clusters +## Installation + +Setup a service account with the Cloud Build Service Account Role: ```bash -git submodule init -git submodule update +gcloud iam service-accounts create isidro-skaffold \ + --display-name="Isidro Skaffold" +gcloud projects add-iam-policy-binding $GOOGLE_PROJECT \ + --member="serviceAccount:isidro-skaffold@$GOOGLE_PROJECT.iam.gserviceaccount.com" \ + --role="roles/cloudbuild.builds.builder" +gcloud iam service-accounts keys create isidro-skaffold.json \ + --iam-account="isidro-skaffold@$GOOGLE_PROJECT.iam.gserviceaccount.com" ``` +Setup skaffold files and credentials: ```bash -kubectl create namespace istio-ingressgateway -kubectl label namespace istio-ingressgateway istio.io/rev=asm-managed --overwrite -kubectl apply -f anthos-service-mesh-packages/samples/gateways/istio-ingressgateway -n istio-ingressgateway +export GOOGLE_APPLICATION_CREDENTIALS=isidro-skaffold.json +cp skaffold.dev.yaml skaffold.yaml +sed -i "s/GOOGLE_PROJECT/$GOOGLE_PROJECT/g" skaffold.yaml ``` -### Enable GMP +### Non-persistent (i.e., development) environments -In the Google Cloud Console, enable Managed Prometheus for the provisioned cluster +Make any required `skaffold.yaml` configuration changes, then run skaffold: +```bash +skaffold dev +``` -### Helm installation +### Persistent environments -Change or override Helm values and then install. +Make any required `skaffold.yaml` configuration changes, then run skaffold: ```bash -cd chart -helm dependencies update -helm install isidro . +skaffold run ``` -### DNS setup - -Setup an A record DNS entry for the Istio Ingress Gateway Load Balancer IP +To teardown: ```bash -kubectl get svc -n istio-ingressgateway +skaffold delete ``` +### DNS setup + +Setup an A record DNS entry for the Istio Multi-Cluster Ingress IP + ## System configuration ### Slack configuration @@ -150,50 +187,9 @@ Create a personal access token, which includes `repo`, `workflow`, and `packages Mention @isidro in Slack messages, and get a response. Use separate message threads for separate chatbot conversations. -## Development - -### Skaffold - -Set the `GOOGLE_PROJECT` environment variable and configure kubeconfig to use the isidro cluster. - -Setup a service account with the Cloud Build Service Account Role: -```bash -gcloud iam service-accounts create isidro-skaffold \ - --display-name="Isidro Skaffold" -gcloud projects add-iam-policy-binding $GOOGLE_PROJECT \ - --member="serviceAccount:isidro-skaffold@$GOOGLE_PROJECT.iam.gserviceaccount.com" \ - --role="roles/cloudbuild.builds.builder" -gcloud iam service-accounts keys create isidro-skaffold.json \ - --iam-account="isidro-skaffold@$GOOGLE_PROJECT.iam.gserviceaccount.com" -``` - -Setup skaffold files and credentials: -```bash -export GOOGLE_APPLICATION_CREDENTIALS=isidro-skaffold.json -cp skaffold.dev.yaml skaffold.yaml -sed -i "s/GOOGLE_PROJECT/$GOOGLE_PROJECT/g" skaffold.yaml -``` - -Make any required `skaffold.yaml` configuration changes, then run skaffold: -```bash -skaffold dev -``` - -#### Skaffold-based persistent deployment - -To setup: -```bash -skaffold run -``` - -To teardown: -```bash -skaffold delete -``` - ### Test payload ```bash -curl -X POST https://example.com/api/v1/submit \ +curl -X POST https://isidro.example.com/api/v1/submit \ -H "Content-Type: application/json" \ -d '{"token": "1234567890", "event": {"channel": "quality", "ts": "1234567890", "user": "me", "text": "Hello"}}' ``` \ No newline at end of file diff --git a/anthos-service-mesh-packages b/anthos-service-mesh-packages deleted file mode 160000 index ff5c31c..0000000 --- a/anthos-service-mesh-packages +++ /dev/null @@ -1 +0,0 @@ -Subproject commit ff5c31cfa2cb4bae81d114999b8b22787d36172a diff --git a/chart/Chart.yaml b/chart/Chart.yaml index b46e22e..c14640f 100644 --- a/chart/Chart.yaml +++ b/chart/Chart.yaml @@ -5,9 +5,11 @@ dependencies: - name: rabbitmq version: 8.30.2 repository: https://charts.bitnami.com/bitnami + condition: rabbitmq.enabled - name: mongodb version: 11.1.1 repository: https://charts.bitnami.com/bitnami + condition: mongodb.enabled - name: mattermost-team-edition alias: "mattermost" version: 6.5.0 @@ -15,4 +17,5 @@ dependencies: condition: mattermost.enabled - name: redis version: 16.2.0 - repository: https://charts.bitnami.com/bitnami \ No newline at end of file + repository: https://charts.bitnami.com/bitnami + condition: redis.enabled \ No newline at end of file diff --git a/chart/templates/certbot.yaml b/chart/templates/certbot.yaml index 304048c..b09a602 100644 --- a/chart/templates/certbot.yaml +++ b/chart/templates/certbot.yaml @@ -33,10 +33,9 @@ spec: args: - | kubectl create secret tls ingress-tls \ - --namespace istio-ingressgateway \ --dry-run -o yaml \ --cert=/etc/ssl/fullchain.pem \ - --key=/etc/ssl/privkey.pem | kubectl apply -n istio-ingressgateway -f - + --key=/etc/ssl/privkey.pem | kubectl apply -f - securityContext: readOnlyRootFilesystem: true runAsNonRoot: true @@ -90,10 +89,9 @@ spec: args: - | kubectl create secret tls ingress-tls \ - --namespace istio-ingressgateway \ --dry-run -o yaml \ --cert=/etc/ssl/fullchain.pem \ - --key=/etc/ssl/privkey.pem | kubectl apply -n istio-ingressgateway -f - + --key=/etc/ssl/privkey.pem | kubectl apply -f - securityContext: readOnlyRootFilesystem: true runAsNonRoot: true @@ -118,7 +116,6 @@ kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: certbot - namespace: istio-ingressgateway subjects: - kind: ServiceAccount name: certbot @@ -132,7 +129,6 @@ kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: certbot - namespace: istio-ingressgateway rules: - apiGroups: - "" diff --git a/chart/templates/deployer-github.yaml b/chart/templates/deployer-github.yaml index 9f932b4..e6df845 100644 --- a/chart/templates/deployer-github.yaml +++ b/chart/templates/deployer-github.yaml @@ -1,3 +1,4 @@ +{{ if eq .Values.deployer.github.enabled true }} apiVersion: v1 kind: Service metadata: @@ -104,4 +105,5 @@ spec: ingress: - {} egress: - - {} \ No newline at end of file + - {} +{{ end }} \ No newline at end of file diff --git a/chart/templates/gatekeeper.yaml b/chart/templates/gatekeeper.yaml index db5749e..7344718 100644 --- a/chart/templates/gatekeeper.yaml +++ b/chart/templates/gatekeeper.yaml @@ -1,3 +1,4 @@ +{{ if eq .Values.gatekeeper.enabled true }} apiVersion: v1 kind: Service metadata: @@ -59,10 +60,7 @@ spec: - Egress - Ingress ingress: - - from: - - podSelector: - matchLabels: - istio: ingressgateway - - namespaceSelector: {} + - {} egress: - - {} \ No newline at end of file + - {} +{{ end }} \ No newline at end of file diff --git a/chart/templates/gateway.yaml b/chart/templates/gateway.yaml deleted file mode 100644 index 5b69c57..0000000 --- a/chart/templates/gateway.yaml +++ /dev/null @@ -1,51 +0,0 @@ -apiVersion: networking.istio.io/v1alpha3 -kind: Gateway -metadata: - name: gateway -spec: - selector: - istio: ingressgateway - servers: - - port: - number: 80 - name: http - protocol: HTTP - tls: - httpsRedirect: true - hosts: - - "{{ .Values.certbot.domain }}" - - port: - number: 443 - name: https - protocol: HTTPS - tls: - mode: SIMPLE - credentialName: ingress-tls - hosts: - - "{{ .Values.certbot.domain }}" ---- -apiVersion: networking.istio.io/v1alpha3 -kind: VirtualService -metadata: - name: gatekeeper -spec: - hosts: - - "{{ .Values.certbot.domain }}" - gateways: - - gateway - http: - - name: isidro - match: - - uri: - prefix: "/isidro/api/" - route: - - destination: - host: gatekeeper - port: - number: 80 - - name: mattermost - route: - - destination: - host: mattermost - port: - number: 8065 \ No newline at end of file diff --git a/chart/templates/keywords.yaml b/chart/templates/keywords.yaml index e66b0a5..e9d1f99 100644 --- a/chart/templates/keywords.yaml +++ b/chart/templates/keywords.yaml @@ -1,3 +1,4 @@ +{{ if eq .Values.keywords.enabled true }} apiVersion: v1 kind: Service metadata: @@ -55,4 +56,5 @@ spec: matchLabels: app: orchestration egress: - - {} \ No newline at end of file + - {} +{{ end }} \ No newline at end of file diff --git a/chart/templates/mattermost.yaml b/chart/templates/mattermost.yaml index 25c8de8..87af032 100644 --- a/chart/templates/mattermost.yaml +++ b/chart/templates/mattermost.yaml @@ -1,3 +1,4 @@ +{{ if eq .Values.mattermost.enabled true }} --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy @@ -11,10 +12,7 @@ spec: - Egress - Ingress ingress: - - from: - - podSelector: - matchLabels: - istio: ingressgateway - - namespaceSelector: {} + - {} egress: - - {} \ No newline at end of file + - {} +{{ end }} \ No newline at end of file diff --git a/chart/templates/mongodb.yaml b/chart/templates/mongodb.yaml index 4c6a723..d86f16e 100644 --- a/chart/templates/mongodb.yaml +++ b/chart/templates/mongodb.yaml @@ -1,3 +1,4 @@ +{{ if eq .Values.mongodb.enabled true }} --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy @@ -16,4 +17,5 @@ spec: matchLabels: app: orchestration egress: - - {} \ No newline at end of file + - {} +{{ end }} \ No newline at end of file diff --git a/chart/templates/monitoring.yaml b/chart/templates/monitoring.yaml deleted file mode 100644 index f89665c..0000000 --- a/chart/templates/monitoring.yaml +++ /dev/null @@ -1,12 +0,0 @@ -apiVersion: monitoring.googleapis.com/v1alpha1 -kind: PodMonitoring -metadata: - name: orchestration -spec: - selector: - matchLabels: - app: orchestration - endpoints: - - port: 80 - interval: 30s - timeout: 10s \ No newline at end of file diff --git a/chart/templates/mysql.yaml b/chart/templates/mysql.yaml index a1b5ed0..61c11e8 100644 --- a/chart/templates/mysql.yaml +++ b/chart/templates/mysql.yaml @@ -1,3 +1,4 @@ +{{ if eq .Values.mattermost.enabled true }} --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy @@ -14,4 +15,5 @@ spec: - from: - podSelector: matchLabels: - app.kubernetes.io/name: mattermost \ No newline at end of file + app.kubernetes.io/name: mattermost +{{ end }} \ No newline at end of file diff --git a/chart/templates/orchestration.yaml b/chart/templates/orchestration.yaml index e10677e..bb3482c 100644 --- a/chart/templates/orchestration.yaml +++ b/chart/templates/orchestration.yaml @@ -1,3 +1,4 @@ +{{ if eq .Values.orchestration.enabled true }} apiVersion: v1 kind: Service metadata: @@ -45,8 +46,23 @@ spec: value: "1234567890" - name: MONGODB_DATABASE value: "isidro" + - name: GREETING + value: {{ .Values.orchestration.greeting }} restartPolicy: Always --- +apiVersion: monitoring.googleapis.com/v1alpha1 +kind: PodMonitoring +metadata: + name: orchestration +spec: + selector: + matchLabels: + app: orchestration + endpoints: + - port: 80 + interval: 30s + timeout: 10s +--- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: @@ -69,4 +85,5 @@ spec: matchLabels: kubernetes.io/metadata.name: gmp-system egress: - - {} \ No newline at end of file + - {} +{{ end }} \ No newline at end of file diff --git a/chart/templates/policy-agent.yaml b/chart/templates/policy-agent.yaml index 14f819c..173a3f8 100644 --- a/chart/templates/policy-agent.yaml +++ b/chart/templates/policy-agent.yaml @@ -1,3 +1,4 @@ +{{ if eq .Values.policyAgent.enabled true }} --- apiVersion: networking.istio.io/v1alpha3 kind: VirtualService @@ -68,7 +69,7 @@ spec: spec: containers: - name: policy-agent - image: openpolicyagent/opa:0.33.1 + image: {{ .Values.policyAgent.image.repository }}:{{ .Values.policyAgent.image.tag }} ports: - name: http containerPort: 8181 @@ -110,7 +111,7 @@ data: }, { "action": { - "confirmation message": "Would my documentation be helpful to you?. Please respond by sending a threaded message reply, which includes an @isidro mention.", + "confirmation message": "Would my documentation be helpful to you? Please respond by sending a threaded message reply, which includes an @isidro mention.", "async": false, "category": "link", "href": "https://github.com/ndebuhr/isidro", @@ -301,4 +302,5 @@ spec: matchLabels: app: orchestration egress: - - {} \ No newline at end of file + - {} +{{ end }} \ No newline at end of file diff --git a/chart/templates/rabbitmq.yaml b/chart/templates/rabbitmq.yaml index 87df44c..98c1915 100644 --- a/chart/templates/rabbitmq.yaml +++ b/chart/templates/rabbitmq.yaml @@ -1,3 +1,4 @@ +{{ if eq .Values.rabbitmq.enabled true }} --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy @@ -13,4 +14,5 @@ spec: ingress: - {} egress: - - {} \ No newline at end of file + - {} +{{ end }} \ No newline at end of file diff --git a/chart/templates/redis.yaml b/chart/templates/redis.yaml index bc35ee6..21301c2 100644 --- a/chart/templates/redis.yaml +++ b/chart/templates/redis.yaml @@ -1,3 +1,4 @@ +{{ if eq .Values.redis.enabled true }} --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy @@ -13,4 +14,5 @@ spec: ingress: - {} egress: - - {} \ No newline at end of file + - {} +{{ end }} \ No newline at end of file diff --git a/chart/templates/repeater.yaml b/chart/templates/repeater.yaml index 75effee..e7ab147 100644 --- a/chart/templates/repeater.yaml +++ b/chart/templates/repeater.yaml @@ -1,3 +1,4 @@ +{{ if eq .Values.repeater.enabled true }} apiVersion: v1 kind: Service metadata: @@ -55,4 +56,5 @@ spec: matchLabels: app: orchestration egress: - - {} \ No newline at end of file + - {} +{{ end }} \ No newline at end of file diff --git a/chart/templates/responder.yaml b/chart/templates/responder.yaml index e7f1467..5efe415 100644 --- a/chart/templates/responder.yaml +++ b/chart/templates/responder.yaml @@ -1,3 +1,4 @@ +{{ if eq .Values.responder.enabled true }} apiVersion: v1 kind: Service metadata: @@ -71,4 +72,5 @@ spec: matchLabels: app: deployer-github-workers egress: - - {} \ No newline at end of file + - {} +{{ end }} \ No newline at end of file diff --git a/chart/templates/services.yaml b/chart/templates/services.yaml new file mode 100644 index 0000000..f183578 --- /dev/null +++ b/chart/templates/services.yaml @@ -0,0 +1,32 @@ +{{ if eq .Values.mcs.enabled true }} +--- +apiVersion: networking.gke.io/v1 +kind: MultiClusterService +metadata: + name: mattermost +spec: + template: + spec: + selector: + app.kubernetes.io/name: mattermost + ports: + - name: web + protocol: TCP + port: 8065 + targetPort: 8065 +--- +apiVersion: networking.gke.io/v1 +kind: MultiClusterService +metadata: + name: gatekeeper +spec: + template: + spec: + selector: + app: gatekeeper + ports: + - name: api + protocol: TCP + port: 80 + targetPort: 80 +{{ end }} \ No newline at end of file diff --git a/chart/templates/storage.yaml b/chart/templates/storage.yaml index 6eb911b..7e92402 100644 --- a/chart/templates/storage.yaml +++ b/chart/templates/storage.yaml @@ -11,6 +11,4 @@ allowedTopologies: - matchLabelExpressions: - key: topology.gke.io/zone values: - - us-central1-a - - us-central1-b - - us-central1-f \ No newline at end of file + {{ toYaml .Values.zones | nindent 4 }} \ No newline at end of file diff --git a/chart/templates/tasks.yaml b/chart/templates/tasks.yaml index 0f52490..921c59d 100644 --- a/chart/templates/tasks.yaml +++ b/chart/templates/tasks.yaml @@ -1,3 +1,4 @@ +{{ if eq .Values.tasks.enabled true }} apiVersion: v1 kind: Service metadata: @@ -55,4 +56,5 @@ spec: matchLabels: app: orchestration egress: - - {} \ No newline at end of file + - {} +{{ end }} \ No newline at end of file diff --git a/chart/values.yaml b/chart/values.yaml index ce1972c..fe10da0 100644 --- a/chart/values.yaml +++ b/chart/values.yaml @@ -1,4 +1,12 @@ project: PROJECT +region: us-central1 +zones: +- us-central1-a +- us-central1-b +- us-central1-f + +mcs: + enabled: false certbot: enabled: false @@ -11,43 +19,58 @@ certbot: tag: "1.20" gatekeeper: + enabled: true image: repository: us.gcr.io/PROJECT/isidro/gatekeeper tag: latest repeater: + enabled: true image: repository: us.gcr.io/PROJECT/isidro/repeater tag: latest responder: + enabled: true image: repository: us.gcr.io/PROJECT/isidro/responder tag: latest deployer: github: + enabled: true image: repository: us.gcr.io/PROJECT/isidro/deployer/github tag: latest token: "1234567890" orchestration: + enabled: true + greeting: "Hello!" image: repository: us.gcr.io/PROJECT/isidro/orchestration tag: latest +policyAgent: + enabled: true + image: + repository: openpolicyagent/opa + tag: 0.33.1 + keywords: + enabled: true image: repository: us.gcr.io/PROJECT/isidro/keywords tag: latest tasks: + enabled: true image: repository: us.gcr.io/PROJECT/isidro/tasks tag: latest rabbitmq: + enabled: true global: storageClass: isidro-storage auth: @@ -93,6 +116,7 @@ rabbitmq: loopback_users.isidro = false redis: + enabled: true architecture: standalone auth: enabled: false @@ -104,6 +128,7 @@ redis: resources: {} mongodb: + enabled: true global: storageClass: isidro-storage podLabels: diff --git a/gatekeeper/main.py b/gatekeeper/main.py index 6ae6b0c..7bd2879 100644 --- a/gatekeeper/main.py +++ b/gatekeeper/main.py @@ -46,6 +46,7 @@ FlaskInstrumentor().instrument_app(app) RequestsInstrumentor().instrument() + class Gatekeeper: def __init__(self, request): self.request = request @@ -170,3 +171,7 @@ def submission(): return gatekeeper.challenge_response() else: return gatekeeper.general_response() + +@app.route("/", methods=["GET"]) +def health(): + return "" \ No newline at end of file diff --git a/orchestration/main.py b/orchestration/main.py index 7069e0a..0801b69 100644 --- a/orchestration/main.py +++ b/orchestration/main.py @@ -24,11 +24,17 @@ "y", "correct", "confirm", - "sure", "ok", "okay", + "cool", + "sure" ] +GREETING = os.environ.get("GREETING") + +if not GREETING: + raise ValueError("No GREETING environment variable set") + set_global_textmap(CloudTraceFormatPropagator()) tracer_provider = TracerProvider() @@ -102,7 +108,10 @@ def send_confirmation(self): "channel": self.channel, "thread_ts": self.thread_ts, "user": self.user, - "text": self.action["confirmation message"], + "text": "{0} {1}".format( + GREETING, + self.action["confirmation message"] + ), }, ).raise_for_status() diff --git a/provisioning/binauthz.tf b/provisioning/binauthz.tf index ae8980d..73c2290 100644 --- a/provisioning/binauthz.tf +++ b/provisioning/binauthz.tf @@ -29,7 +29,14 @@ resource "google_binary_authorization_policy" "policy" { } cluster_admission_rules { - cluster = "${var.region}.${var.cluster_name}" + cluster = "${var.region_primary}.${var.cluster_name_primary}" + evaluation_mode = "REQUIRE_ATTESTATION" + enforcement_mode = "ENFORCED_BLOCK_AND_AUDIT_LOG" + require_attestations_by = [google_binary_authorization_attestor.isidro.name] + } + + cluster_admission_rules { + cluster = "${var.region_secondary}.${var.cluster_name_secondary}" evaluation_mode = "REQUIRE_ATTESTATION" enforcement_mode = "ENFORCED_BLOCK_AND_AUDIT_LOG" require_attestations_by = [google_binary_authorization_attestor.isidro.name] diff --git a/provisioning/config.tf b/provisioning/config.tf new file mode 100644 index 0000000..c0396bf --- /dev/null +++ b/provisioning/config.tf @@ -0,0 +1,100 @@ +data "google_client_config" "default" {} +data "google_project" "project" {} + +provider "kubernetes" { + alias = "config" + host = "https://${module.gke_config.endpoint}" + token = data.google_client_config.default.access_token + cluster_ca_certificate = base64decode(module.gke_config.ca_certificate) +} + +module "gke_config" { + source = "github.com/terraform-google-modules/terraform-google-kubernetes-engine//modules/beta-autopilot-public-cluster" + project_id = data.google_project.project.project_id + name = var.cluster_name_config + regional = true + region = var.region_config + release_channel = "REGULAR" + network = var.network + subnetwork = var.subnetwork_config + ip_range_pods = var.ip_range_pods_config + ip_range_services = var.ip_range_services_config + enable_vertical_pod_autoscaling = true + cluster_resource_labels = { "mesh_id" : "proj-${data.google_project.project.number}" } +} + +module "asm_config" { + source = "github.com/terraform-google-modules/terraform-google-kubernetes-engine//modules/asm" + cluster_name = module.gke_config.name + project_id = data.google_project.project.project_id + cluster_location = module.gke_config.location + enable_cni = true + enable_fleet_registration = true + fleet_id = "isidro" + providers = { + kubernetes = kubernetes.config + } +} + +resource "google_gke_hub_feature" "mci" { + name = "multiclusteringress" + location = "global" + spec { + multiclusteringress { + config_membership = "projects/${data.google_project.project.project_id}/locations/global/memberships/${var.cluster_name_config}-membership" + } + } + provider = google-beta +} + +resource "google_compute_global_address" "isidro" { + name = "isidro" +} + +resource "kubernetes_manifest" "mci" { + manifest = { + "apiVersion" = "networking.gke.io/v1" + "kind" = "MultiClusterIngress" + "metadata" = { + "name" = "isidro-ingress" + "namespace" = "default" + "annotations" = { + "networking.gke.io/static-ip" = google_compute_global_address.isidro.address + } + } + "spec" = { + "template" = { + "spec" = { + "tls" = [ + { + "hosts" = [ + var.domain + ] + "secretName" = "ingress-tls" + } + ] + "backend" = { + "serviceName" = "mattermost" + "servicePort" = "8065" + } + "rules" = [ + { + "http" = { + "paths" = [ + { + "path" = "/isidro/api/*" + "backend" = { + "serviceName" = "gatekeeper" + "servicePort" = "80" + } + } + ] + } + } + ] + } + } + } + } + provider = kubernetes.config +} \ No newline at end of file diff --git a/provisioning/outputs.tf b/provisioning/outputs.tf index 737ca2a..a23b6bd 100644 --- a/provisioning/outputs.tf +++ b/provisioning/outputs.tf @@ -1,6 +1,6 @@ output "kubernetes_endpoint" { sensitive = true - value = module.gke.endpoint + value = module.gke_primary.endpoint } output "client_token" { @@ -10,11 +10,15 @@ output "client_token" { output "ca_certificate" { sensitive = true - value = module.gke.ca_certificate + value = module.gke_primary.ca_certificate } output "service_account" { - value = module.gke.service_account + value = module.gke_primary.service_account +} + +output "isidro_ip" { + value = google_compute_global_address.isidro.address } output "binauthz_keyring" { diff --git a/provisioning/main.tf b/provisioning/primary.tf similarity index 69% rename from provisioning/main.tf rename to provisioning/primary.tf index f836fe5..4c6aad0 100644 --- a/provisioning/main.tf +++ b/provisioning/primary.tf @@ -1,24 +1,21 @@ -data "google_client_config" "default" {} - provider "kubernetes" { - host = "https://${module.gke.endpoint}" + alias = "primary" + host = "https://${module.gke_primary.endpoint}" token = data.google_client_config.default.access_token - cluster_ca_certificate = base64decode(module.gke.ca_certificate) + cluster_ca_certificate = base64decode(module.gke_primary.ca_certificate) } -data "google_project" "project" {} - -module "gke" { +module "gke_primary" { source = "github.com/terraform-google-modules/terraform-google-kubernetes-engine" project_id = data.google_project.project.project_id - name = var.cluster_name + name = var.cluster_name_primary regional = true - region = var.region + region = var.region_primary release_channel = "REGULAR" network = var.network - subnetwork = var.subnetwork - ip_range_pods = var.ip_range_pods - ip_range_services = var.ip_range_services + subnetwork = var.subnetwork_primary + ip_range_pods = var.ip_range_pods_primary + ip_range_services = var.ip_range_services_primary network_policy = true create_service_account = false service_account = google_service_account.isidro_nodes.email @@ -45,13 +42,15 @@ module "gke" { } } -module "asm" { +module "asm_primary" { source = "github.com/terraform-google-modules/terraform-google-kubernetes-engine//modules/asm" - cluster_name = module.gke.name + cluster_name = module.gke_primary.name project_id = data.google_project.project.project_id - cluster_location = module.gke.location + cluster_location = module.gke_primary.location enable_cni = true enable_fleet_registration = true - enable_mesh_feature = false fleet_id = "isidro" -} + providers = { + kubernetes = kubernetes.primary + } +} \ No newline at end of file diff --git a/provisioning/secondary.tf b/provisioning/secondary.tf new file mode 100644 index 0000000..0fdd197 --- /dev/null +++ b/provisioning/secondary.tf @@ -0,0 +1,57 @@ +provider "kubernetes" { + alias = "secondary" + host = "https://${module.gke_secondary.endpoint}" + token = data.google_client_config.default.access_token + cluster_ca_certificate = base64decode(module.gke_secondary.ca_certificate) +} + +module "gke_secondary" { + depends_on = [module.gke_primary] + source = "github.com/terraform-google-modules/terraform-google-kubernetes-engine" + project_id = data.google_project.project.project_id + name = var.cluster_name_secondary + regional = true + region = var.region_secondary + release_channel = "REGULAR" + network = var.network + subnetwork = var.subnetwork_secondary + ip_range_pods = var.ip_range_pods_secondary + ip_range_services = var.ip_range_services_secondary + network_policy = true + create_service_account = false + service_account = google_service_account.isidro_nodes.email + enable_binary_authorization = true + cluster_resource_labels = { "mesh_id" : "proj-${data.google_project.project.number}" } + node_pools = [ + { + name = "asm-node-pool" + autoscaling = false + auto_upgrade = true + # ASM requires minimum 4 nodes and e2-standard-4 + node_count = 1 + machine_type = "e2-standard-4" + }, + ] + node_pools_oauth_scopes = { + # TODO: This seems to have no effect, or is overwritten by cloud_platform + "asm-node-pool" : [ + "https://www.googleapis.com/auth/devstorage.read_only", + "https://www.googleapis.com/auth/logging.write", + "https://www.googleapis.com/auth/monitoring", + "https://www.googleapis.com/auth/trace.append" + ] + } +} + +module "asm_secondary" { + source = "github.com/terraform-google-modules/terraform-google-kubernetes-engine//modules/asm" + cluster_name = module.gke_secondary.name + project_id = data.google_project.project.project_id + cluster_location = module.gke_secondary.location + enable_cni = true + enable_fleet_registration = true + fleet_id = "isidro" + providers = { + kubernetes = kubernetes.secondary + } +} \ No newline at end of file diff --git a/provisioning/variables.tf b/provisioning/variables.tf index 0e21e06..a241b4b 100644 --- a/provisioning/variables.tf +++ b/provisioning/variables.tf @@ -1,25 +1,77 @@ -variable "cluster_name" { +variable "domain" { + description = "Domain from which to serve the Isidro system" +} + +variable "network" { + description = "The VPC network to host the cluster in" + default = "default" +} + +variable "cluster_name_primary" { description = "The name for the GKE cluster name" - default = "isidro" + default = "isidro-us" } -variable "region" { - description = "The region to host the cluster in" +variable "region_primary" { + description = "The first region to host the cluster in" default = "us-central1" } -variable "network" { - description = "The VPC network to host the cluster in" +variable "subnetwork_primary" { + description = "The region one subnetwork to host the cluster in" + default = "default" +} + +variable "ip_range_pods_primary" { + description = "The region one secondary ip range to use for pods" } -variable "subnetwork" { - description = "The subnetwork to host the cluster in" +variable "ip_range_services_primary" { + description = "The region one secondary ip range to use for services" } -variable "ip_range_pods" { - description = "The secondary ip range to use for pods" +variable "cluster_name_secondary" { + description = "The name for the GKE cluster name" + default = "isidro-europe" } -variable "ip_range_services" { - description = "The secondary ip range to use for services" +variable "region_secondary" { + description = "The second region to host the cluster in" + default = "europe-west1" } + +variable "subnetwork_secondary" { + description = "The region two subnetwork to host the cluster in" + default = "default" +} + +variable "ip_range_pods_secondary" { + description = "The region two secondary ip range to use for pods" +} + +variable "ip_range_services_secondary" { + description = "The region two secondary ip range to use for services" +} + +variable "cluster_name_config" { + description = "The name for the GKE cluster name" + default = "isidro-config" +} + +variable "region_config" { + description = "The region to host the config cluster in" + default = "northamerica-northeast1" +} + +variable "subnetwork_config" { + description = "The config cluster region subnetwork" + default = "default" +} + +variable "ip_range_pods_config" { + description = "The config cluster region secondary ip range to use for pods" +} + +variable "ip_range_services_config" { + description = "The config cluster region secondary ip range to use for services" +} \ No newline at end of file diff --git a/roles/provisioner.yaml b/roles/provisioner.yaml index 857e237..8678682 100644 --- a/roles/provisioner.yaml +++ b/roles/provisioner.yaml @@ -21,6 +21,9 @@ includedPermissions: - cloudkms.cryptoKeys.update - cloudkms.keyRings.create - cloudkms.keyRings.get +- compute.globalAddresses.create +- compute.globalAddresses.delete +- compute.globalAddresses.get - compute.instanceGroupManagers.get - compute.zones.list - container.clusters.create @@ -30,13 +33,21 @@ includedPermissions: - container.configMaps.create - container.configMaps.delete - container.configMaps.get +- container.customResourceDefinitions.list - container.namespaces.create - container.namespaces.delete - container.namespaces.get - container.operations.get +- container.thirdPartyObjects.create +- container.thirdPartyObjects.delete +- container.thirdPartyObjects.get +- container.thirdPartyObjects.update - containeranalysis.notes.create - containeranalysis.notes.delete - containeranalysis.notes.get +- gkehub.features.create +- gkehub.features.delete +- gkehub.features.get - gkehub.memberships.create - gkehub.memberships.delete - gkehub.memberships.get diff --git a/skaffold.dev.yaml b/skaffold.dev.yaml index 2c121da..f1686fe 100644 --- a/skaffold.dev.yaml +++ b/skaffold.dev.yaml @@ -1,9 +1,7 @@ apiVersion: skaffold/v2beta26 kind: Config metadata: - name: dev -requires: - - configs: ["nlp-dev"] + name: build build: artifacts: - image: us.gcr.io/GOOGLE_PROJECT/isidro/deployer/github @@ -50,7 +48,32 @@ build: - command: ["bash", "-c", "./skaffold-binauthz.sh $SKAFFOLD_IMAGE"] googleCloudBuild: timeout: 3600s +--- +apiVersion: skaffold/v2beta26 +kind: Config +metadata: + name: build-nlp +build: + artifacts: + - image: us.gcr.io/GOOGLE_PROJECT/isidro/keywords + context: keywords + kaniko: + cache: {} + hooks: + after: + - command: ["bash", "-c", "./skaffold-binauthz.sh $SKAFFOLD_IMAGE"] + googleCloudBuild: + timeout: 3600s + machineType: E2_HIGHCPU_32 +--- +apiVersion: skaffold/v2beta26 +kind: Config +metadata: + name: deploy-us +requires: + - configs: ["build-nlp", "build"] deploy: + kubeContext: isidro-us helm: releases: - name: isidro @@ -58,14 +81,15 @@ deploy: skipBuildDependencies: true namespace: default setValues: - project: example + project: GOOGLE_PROJECT + mci.enabled: false certbot.enabled: false certbot.domain: isidro.example.com certbot.email: admin@example.com + mattermost.enabled: true mattermost.accessToken: "1234567890" mattermost.verificationToken: "1234567890" - slack.verificationToken: "1234567890" - slack.oauthToken: "1234567890" + orchestration.greeting: "Hello!" deployer.github.token: "1234567890" artifactOverrides: deployer.github.image: us.gcr.io/GOOGLE_PROJECT/isidro/deployer/github @@ -81,16 +105,73 @@ deploy: apiVersion: skaffold/v2beta26 kind: Config metadata: - name: nlp-dev -build: - artifacts: - - image: us.gcr.io/GOOGLE_PROJECT/isidro/keywords - context: keywords - kaniko: - cache: {} - hooks: - after: - - command: ["bash", "-c", "./skaffold-binauthz.sh $SKAFFOLD_IMAGE"] - googleCloudBuild: - timeout: 3600s - machineType: E2_HIGHCPU_32 \ No newline at end of file + name: deploy-europe +requires: + - configs: ["build-nlp", "build"] +deploy: + kubeContext: isidro-europe + helm: + releases: + - name: isidro + chartPath: chart + skipBuildDependencies: true + namespace: default + setValues: + project: GOOGLE_PROJECT + region: europe-west1 + zones[0]: "europe-west1-b" + zones[1]: "europe-west1-c" + zones[2]: "europe-west1-d" + mci.enabled: false + certbot.enabled: false + certbot.domain: isidro.example.com + certbot.email: admin@example.com + mattermost.enabled: false + mattermost.accessToken: "1234567890" + mattermost.verificationToken: "1234567890" + orchestration.greeting: "Bonjour!" + deployer.github.token: "1234567890" + artifactOverrides: + deployer.github.image: us.gcr.io/GOOGLE_PROJECT/isidro/deployer/github + gatekeeper.image: us.gcr.io/GOOGLE_PROJECT/isidro/gatekeeper + keywords.image: us.gcr.io/GOOGLE_PROJECT/isidro/keywords + orchestration.image: us.gcr.io/GOOGLE_PROJECT/isidro/orchestration + repeater.image: us.gcr.io/GOOGLE_PROJECT/isidro/repeater + responder.image: us.gcr.io/GOOGLE_PROJECT/isidro/responder + tasks.image: us.gcr.io/GOOGLE_PROJECT/isidro/tasks + imageStrategy: + helm: {} +--- +apiVersion: skaffold/v2beta26 +kind: Config +metadata: + name: deploy-config +requires: + - configs: ["build-nlp", "build"] +deploy: + kubeContext: isidro-config + helm: + releases: + - name: isidro + chartPath: chart + skipBuildDependencies: true + namespace: default + setValues: + project: GOOGLE_PROJECT + mcs.enabled: true + certbot.enabled: true + certbot.domain: isidro.example.com + certbot.email: admin@example.com + gatekeeper.enabled: false + repeater.enabled: false + responder.enabled: false + deployer.github.enabled: false + orchestration.enabled: false + policyAgent.enabled: false + keywords.enabled: false + tasks.enabled: false + rabbitmq.enabled: false + redis.enabled: false + mongodb.enabled: false + mattermost.enabled: false + slack.enabled: false \ No newline at end of file