From 04c76f8b7dce1b83b4efd4d8b6ff67dcd3db3d4a Mon Sep 17 00:00:00 2001 From: Stephan Feurer Date: Fri, 15 Dec 2023 09:17:12 +0100 Subject: [PATCH 1/3] Add Exoscale architecture reference documentation --- .../images/ocp4-architecture-exoscale.svg | 4 + .../references/exoscale/architecture.adoc | 107 ++++++++++++++++++ .../architecture/glossary-general.adoc | 14 ++- .../ROOT/partials/architecture/storage.adoc | 13 ++- docs/modules/ROOT/partials/nav.adoc | 2 + 5 files changed, 138 insertions(+), 2 deletions(-) create mode 100644 docs/modules/ROOT/assets/images/ocp4-architecture-exoscale.svg create mode 100644 docs/modules/ROOT/pages/references/exoscale/architecture.adoc diff --git a/docs/modules/ROOT/assets/images/ocp4-architecture-exoscale.svg b/docs/modules/ROOT/assets/images/ocp4-architecture-exoscale.svg new file mode 100644 index 00000000..f780424c --- /dev/null +++ b/docs/modules/ROOT/assets/images/ocp4-architecture-exoscale.svg @@ -0,0 +1,4 @@ + + + +
Exoscale Zone
Exoscale Zone
Cluster Security Group
Cluster Security Group
OpenShift 4 Control plane nodes
3x 4vCPU/16GiB RAM
OpenShift 4 Control pla...
OpenShift 4 infrastructure nodes
4x 4vCPU/16GiB RAM
OpenShift 4 infrastruct...
OpenShift 4 worker nodes
3x 4vCPU/16GiB RAM
OpenShift 4 worker node...
Pod Network – Cilium VXLAN – 10.128.0.0/14
Pod Network – Cilium VXLAN – 10.128.0.0/14
Kubernetes API
Kubernetes API
Scheduler
Scheduler
Controllers
Controllers
OpenShift API
OpenShift API
etcd
etcd
Ingress Router
Ingress Router
Monitoring
Monitoring
Registry
Registry
Logging
Logging
...
...
Application A
Application A
Application B
Application B
Application C
Application C
Service Network – Cilium eBPF rules – 172.30.0.0/16
Service Network – Cilium eBPF rules – 172.30.0.0/16
Load Balancer Security Group
Load Balancer Security Group
DNS:
api.cluster.cust.vshnmanaged.net
DNS:...
DNS:
ingress.cluster.cust.vshnmanaged.net
*.apps.cluster.cust.vshnmanaged.net
DNS:...
Firewall rules:

ANY       -> 80/tcp   -> Ingress VIP*
ANY       -> 443/tcp  -> Ingress VIP*
ANY       -> 6443/tcp -> API VIP

*Should use PROXY protocol to
 preserve source IPs
Firewall rules:...
Other customer networks
Other customer networks
Other customer systems
Other customer sys...
Other customer systems
Other customer sys...
VSHN services

- LDAP / SSO
- GitLab
- Project Syn API
- Project Syn Vault
- acme-dns
- central metrics store
VSHN services...
Red Hat services

- OpenShift Update Service
- Container registries
- NTP servers
Red Hat services...
3rd party services

- OpsGenie
- Passbolt
- Let's Encrypt
- Container registries
3rd party services...
managed by VSHN
managed by VSHN
managed by Exoscale
managed by Exoscale
managed by Customer, optional
managed by Customer, optional
6443/tcp
6443/tcp
80/tcp 443/tcp
80/tcp 443/tcp
Load Balancer
Load Balancer
API IP
API IP
Ingress IP
Ingress IP Text is not SVG - cannot display \ No newline at end of file diff --git a/docs/modules/ROOT/pages/references/exoscale/architecture.adoc b/docs/modules/ROOT/pages/references/exoscale/architecture.adoc new file mode 100644 index 00000000..8e2f333f --- /dev/null +++ b/docs/modules/ROOT/pages/references/exoscale/architecture.adoc @@ -0,0 +1,107 @@ +:infra-type: Exoscale +:infra-svg: ocp4-architecture-exoscale.svg += APPUiO Managed OpenShift 4 on {infra-type} + +== Architecture overview + +include::partial$architecture/overview.adoc[] + +== {infra-type} requirements + +APPUiO Managed OpenShift 4 on {infra-type} needs a https://docs.openshift.com/container-platform/4.14/installing/installing_bare_metal/installing-bare-metal.html#installation-load-balancing-user-infra_installing-bare-metal[Load Balancer setup] that must meet the following requirements: + +1. API load balancer: Provides a common endpoint to interact with the OpenShift and Kubernetes. + +2. Ingress load balancer: Provides an ingress point for application traffic flowing in from outside the cluster. + +See the https://docs.openshift.com/container-platform/latest/installing/installing_bare_metal/installing-bare-metal.html#installation-requirements-user-infra_installing-bare-metal[upstream documentation] for details on {infra-type} requirements. + + +== Networking + +=== Security Groups + +On {infra-type} APPUiO Managed OpenShift 4 uses public ips for each node in the cluster. +See https://kb.vshn.ch/oc4/explanations/exoscale/limitations.html#_private_networks[Limitations] of the {infra-type} environment. + +The individual VMs are placed in https://community.exoscale.com/documentation/compute/security-groups[Security Groups] to restrict access and isolate the nodes from the public internet. + +=== Virtual IPs + +To expose applications and the Kubernetes API outside the cluster, APPUiO Managed OpenShift 4 manages two floating IPs: + +1. The "API VIP" for the Kubernetes and OpenShift API. +APPUiO Managed OpenShift 4 uses a public floating IP as the API VIP. +2. The "Ingress VIP" for the OpenShift Ingress Router +APPUiO Managed OpenShift 4 uses a public floating IP as the Ingress VIP. + +APPUiO Managed OpenShift 4 uses two Load Balancer instances to manage the API and ingress VIPs and distributes traffic to the master / infrastructure nodes. + +=== Pod and service networks + +include::partial$architecture/networking-pods.adoc[] + +=== Exposing the cluster + +On {infra-type} infrastructure two Load Balancer instances provide ingress to the cluster. +The Load Balancer setup exposes two public IPs: + +1. A public IP for the API. +Traffic to port `6443/tcp` on this IP must be forwarded to the control plane nodes in the machine network. +The forwarding of this traffic must happen transparently. +In particular, no TLS interception can be performed as the Kubernetes API depends on mutual TLS authentication. +VSHN will manage a DNS record pointing to this IP. +2. A public IP for HTTP(s) ingress. +Traffic to ports `80/tcp` and `443/tcp` on this IP must be forwarded to the infrastructure nodes in the machine network. +The PROXY protocol should be enabled to preserve source IPs. +Forwarding should happen transparently in TCP mode. +VSHN will manage a wildcard DNS record pointing to this IP. +Additional DNS records can be pointed to this IP by the customer. + +=== External services + +include::partial$architecture/networking-external.adoc[] + +== Storage + +include::partial$architecture/storage.adoc[] + +== Glossary + +=== Components {infra-type} + +[cols="1,3,1"] +|=== +|Name|Description|provided by + +|Security Group +a|Exoscale Security Groups provide a modular way to define and compose firewall rules. + +Security Groups hold two different types of information: +* A list of rules to apply to traffic +* A list of member instances in the security group which allows using groups as traffic sources or destinations in rules + +See https://community.exoscale.com/documentation/compute/security-groups[Upstream Documentation]. + +|{infra-type} + +|S3 compatible storage +a|Various OpenShift components require S3 compatible storage. +This storage is provided by {infra-type}. + +The main APPUiO Managed OpenShift 4 components that use object storage are + +* OpenShift integrated image registry +* OpenShift logging stack +* APPUiO Managed cluster backups +|{infra-type} + +|=== + +=== Components General + +include::partial$architecture/glossary-general.adoc[] + +=== Other terms + +include::partial$architecture/glossary-others.adoc[] diff --git a/docs/modules/ROOT/partials/architecture/glossary-general.adoc b/docs/modules/ROOT/partials/architecture/glossary-general.adoc index becd3b1e..5ee39d03 100644 --- a/docs/modules/ROOT/partials/architecture/glossary-general.adoc +++ b/docs/modules/ROOT/partials/architecture/glossary-general.adoc @@ -32,9 +32,21 @@ If the service network IP range conflicts with existing subnets, the service net | VSHN / Cilium |DNS -|The APPUiO Managed OpenShift 4 cluster's base DNS records are defined and managed by VSHN. +a|The APPUiO Managed OpenShift 4 cluster's base DNS records are defined and managed by VSHN. All records must be publicly resolvable. To expose applications under a customer domain, a CNAME target is provided. | VSHN + +ifeval::["{infra-type}" == "Exoscale"] +|Storage Cluster +a|The APPUiO Managed Storage Cluster offers advanced cloud-native storage capabilities for APPUiO Managed OpenShift 4. + +This product is based on https://rook.io/[Rook] and uses https://ceph.io/en/[Ceph] as it’s underlying storage technology. + +See https://products.vshn.ch/appuio/managed/storage_cluster.html[APPUiO Managed Storage Cluster] product page for more details. + +| VSHN / Rook +endif::[] + |=== diff --git a/docs/modules/ROOT/partials/architecture/storage.adoc b/docs/modules/ROOT/partials/architecture/storage.adoc index 5a35120e..afb935ee 100644 --- a/docs/modules/ROOT/partials/architecture/storage.adoc +++ b/docs/modules/ROOT/partials/architecture/storage.adoc @@ -19,16 +19,27 @@ They're allocated dynamically based on requests from workloads (applications or These block devices are automatically attached to the VM hosting the application container. They're deleted when the corresponding Kubernetes `PersistentVolume` resource is deleted. +ifeval::["{infra-type}" != "Exoscale"] The {infra-type} CSI driver is the in-cluster component which is responsible for allocating, attaching and deleting the persistent volume block devices. +endif::[] + +ifeval::["{infra-type}" == "Exoscale"] +NOTE: On the {infra-type} environment there is no Kubernetes integration available for such block devices. +https://products.vshn.ch/appuio/managed/storage_cluster.html[APPUiO Managed Storage Cluster] is available as and addon to the cluster offering if such functionality is required. +endif::[] These devices hold application data, but backups are usually done from within the cluster. === S3 compatible object storage Various OpenShift components, such as the integrated image registry, the logging stack and backups, require S3 compatible object storage. -The customer or vSphere infrastructure operator must provide S3 compatible object storage. +ifeval::["{infra-type}" != "Exoscale"] +ifeval::["{infra-type}" != "cloudscale.ch"] +The customer or {infra-type} infrastructure operator must provide S3 compatible object storage. Most modern storage solutions offer some object storage functionality. If https://products.vshn.ch/appcat/index.html[VSHN's Application Catalog (AppCat)] offering is required on the cluster, the object storage must support automatic bucket creation via an AppCat-supported provisioner. NOTE: If no object storage is available, we can use external object storage as a fallback. +endif::[] +endif::[] diff --git a/docs/modules/ROOT/partials/nav.adoc b/docs/modules/ROOT/partials/nav.adoc index 07fa5e30..c2db0333 100644 --- a/docs/modules/ROOT/partials/nav.adoc +++ b/docs/modules/ROOT/partials/nav.adoc @@ -15,6 +15,7 @@ ** xref:oc4:ROOT:references/architecture/single_sign_on.adoc[] ** Exoscale +*** xref:oc4:ROOT:references/exoscale/architecture.adoc[Architecture] *** xref:oc4:ROOT:explanations/exoscale/limitations.adoc[Limitations] ** Google Cloud Platform @@ -40,6 +41,7 @@ *** xref:oc4:ROOT:how-tos/cloudscale/decommission.adoc[Decommissioning] ** Exoscale +*** xref:oc4:ROOT:references/exoscale/architecture.adoc[Architecture] *** xref:oc4:ROOT:references/exoscale/config.adoc[Configuration] *** xref:oc4:ROOT:how-tos/exoscale/install.adoc[Install] // Node management From 717edbcdfb23347fb9551273c8ddb2b27716836f Mon Sep 17 00:00:00 2001 From: DebakelOrakel <78408921+DebakelOrakel@users.noreply.github.com> Date: Mon, 8 Jan 2024 08:15:08 +0100 Subject: [PATCH 2/3] Apply suggestions from code review Co-authored-by: Simon Gerber --- .../images/ocp4-architecture-exoscale.svg | 2 +- .../references/exoscale/architecture.adoc | 23 +++++-------------- 2 files changed, 7 insertions(+), 18 deletions(-) diff --git a/docs/modules/ROOT/assets/images/ocp4-architecture-exoscale.svg b/docs/modules/ROOT/assets/images/ocp4-architecture-exoscale.svg index f780424c..794d46db 100644 --- a/docs/modules/ROOT/assets/images/ocp4-architecture-exoscale.svg +++ b/docs/modules/ROOT/assets/images/ocp4-architecture-exoscale.svg @@ -1,4 +1,4 @@ -
Exoscale Zone
Exoscale Zone
Cluster Security Group
Cluster Security Group
OpenShift 4 Control plane nodes
3x 4vCPU/16GiB RAM
OpenShift 4 Control pla...
OpenShift 4 infrastructure nodes
4x 4vCPU/16GiB RAM
OpenShift 4 infrastruct...
OpenShift 4 worker nodes
3x 4vCPU/16GiB RAM
OpenShift 4 worker node...
Pod Network – Cilium VXLAN – 10.128.0.0/14
Pod Network – Cilium VXLAN – 10.128.0.0/14
Kubernetes API
Kubernetes API
Scheduler
Scheduler
Controllers
Controllers
OpenShift API
OpenShift API
etcd
etcd
Ingress Router
Ingress Router
Monitoring
Monitoring
Registry
Registry
Logging
Logging
...
...
Application A
Application A
Application B
Application B
Application C
Application C
Service Network – Cilium eBPF rules – 172.30.0.0/16
Service Network – Cilium eBPF rules – 172.30.0.0/16
Load Balancer Security Group
Load Balancer Security Group
DNS:
api.cluster.cust.vshnmanaged.net
DNS:...
DNS:
ingress.cluster.cust.vshnmanaged.net
*.apps.cluster.cust.vshnmanaged.net
DNS:...
Firewall rules:

ANY       -> 80/tcp   -> Ingress VIP*
ANY       -> 443/tcp  -> Ingress VIP*
ANY       -> 6443/tcp -> API VIP

*Should use PROXY protocol to
 preserve source IPs
Firewall rules:...
Other customer networks
Other customer networks
Other customer systems
Other customer sys...
Other customer systems
Other customer sys...
VSHN services

- LDAP / SSO
- GitLab
- Project Syn API
- Project Syn Vault
- acme-dns
- central metrics store
VSHN services...
Red Hat services

- OpenShift Update Service
- Container registries
- NTP servers
Red Hat services...
3rd party services

- OpsGenie
- Passbolt
- Let's Encrypt
- Container registries
3rd party services...
managed by VSHN
managed by VSHN
managed by Exoscale
managed by Exoscale
managed by Customer, optional
managed by Customer, optional
6443/tcp
6443/tcp
80/tcp 443/tcp
80/tcp 443/tcp
Load Balancer
Load Balancer
API IP
API IP
Ingress IP
Ingress IP Text is not SVG - cannot display \ No newline at end of file +
Exoscale Zone
Exoscale Zone
Cluster Security Group
Cluster Security Group
OpenShift 4 Storage
nodes
3x 8vCPU/16GiB RAM
OpenShift 4 Storage...
OpenShift 4 infrastructure nodes
4x 4vCPU/16GiB RAM
OpenShift 4 infrastruct...
OpenShift 4 worker nodes
3x 4vCPU/16GiB RAM
OpenShift 4 worker node...
Ceph MDS
Ceph MDS
Ceph OSD
Ceph OSD
Rook Operator
Rook Operator
Ingress Router
Ingress Router
Monitoring
Monitoring
Registry
Registry
Logging
Logging
...
...
Application A
Application A
Application B
Application B
Application C
Application C
Load Balancer Security Group
Load Balancer Security Group
DNS:
api.cluster.cust.vshnmanaged.net
DNS:...
DNS:
ingress.cluster.cust.vshnmanaged.net
*.apps.cluster.cust.vshnmanaged.net
DNS:...
Firewall rules:

ANY       -> 80/tcp   -> Ingress VIP*
ANY       -> 443/tcp  -> Ingress VIP*
ANY       -> 6443/tcp -> API VIP

*Should use PROXY protocol to
 preserve source IPs
Firewall rules:...
Other customer networks
Other customer networks
Other customer systems
Other customer sys...
Other customer systems
Other customer sys...
VSHN services

- LDAP / SSO
- GitLab
- Project Syn API
- Project Syn Vault
- acme-dns
- central metrics store
VSHN services...
Red Hat services

- OpenShift Update Service
- Container registries
- NTP servers
Red Hat services...
3rd party services

- OpsGenie
- Passbolt
- Let's Encrypt
- Container registries
3rd party services...
managed by VSHN
managed by VSHN
managed by Exoscale
managed by Exoscale
managed by Customer, optional
managed by Customer, optional
80/tcp 443/tcp
80/tcp 443/tcp
Load Balancer
Load Balancer
API IP
API IP
Ingress IP
Ingress IP
OpenShift 4 Control plane nodes
3x 4vCPU/16GiB RAM
OpenShift 4 Control pla...
Kubernetes API
Kubernetes API
Scheduler
Scheduler
Controllers
Controllers
OpenShift API
OpenShift API
etcd
etcd
Service Network – Cilium eBPF rules – 172.30.0.0/16
Service Network – Cilium eBPF rules – 172.30.0.0/16
Pod Network – Cilium VXLAN – 10.128.0.0/14
Pod Network – Cilium VXLAN – 10.128.0.0/14
6443/tcp
6443/tcp
Dynymic Public
Egress IPs
Dynymic Public...
Ceph Mon
Ceph Mon
Ceph Mgr
Ceph MgrText is not SVG - cannot display \ No newline at end of file diff --git a/docs/modules/ROOT/pages/references/exoscale/architecture.adoc b/docs/modules/ROOT/pages/references/exoscale/architecture.adoc index 8e2f333f..98e41667 100644 --- a/docs/modules/ROOT/pages/references/exoscale/architecture.adoc +++ b/docs/modules/ROOT/pages/references/exoscale/architecture.adoc @@ -10,9 +10,9 @@ include::partial$architecture/overview.adoc[] APPUiO Managed OpenShift 4 on {infra-type} needs a https://docs.openshift.com/container-platform/4.14/installing/installing_bare_metal/installing-bare-metal.html#installation-load-balancing-user-infra_installing-bare-metal[Load Balancer setup] that must meet the following requirements: -1. API load balancer: Provides a common endpoint to interact with the OpenShift and Kubernetes. +1. API load balancer: Provides a common endpoint to interact with OpenShift and Kubernetes. -2. Ingress load balancer: Provides an ingress point for application traffic flowing in from outside the cluster. +2. Ingress load balancer: Provides an endpoint for application traffic flowing in from outside the cluster. See the https://docs.openshift.com/container-platform/latest/installing/installing_bare_metal/installing-bare-metal.html#installation-requirements-user-infra_installing-bare-metal[upstream documentation] for details on {infra-type} requirements. @@ -21,11 +21,13 @@ See the https://docs.openshift.com/container-platform/latest/installing/installi === Security Groups -On {infra-type} APPUiO Managed OpenShift 4 uses public ips for each node in the cluster. +On {infra-type}, APPUiO Managed OpenShift 4 uses public IPs for each node in the cluster. See https://kb.vshn.ch/oc4/explanations/exoscale/limitations.html#_private_networks[Limitations] of the {infra-type} environment. The individual VMs are placed in https://community.exoscale.com/documentation/compute/security-groups[Security Groups] to restrict access and isolate the nodes from the public internet. +NOTE: On the {infra-type} environment there is no single stable egress IP. Every node uses a dynamic public IP for egress traffic, which it is not suited for any forms of whitelisting. + === Virtual IPs To expose applications and the Kubernetes API outside the cluster, APPUiO Managed OpenShift 4 manages two floating IPs: @@ -43,20 +45,7 @@ include::partial$architecture/networking-pods.adoc[] === Exposing the cluster -On {infra-type} infrastructure two Load Balancer instances provide ingress to the cluster. -The Load Balancer setup exposes two public IPs: - -1. A public IP for the API. -Traffic to port `6443/tcp` on this IP must be forwarded to the control plane nodes in the machine network. -The forwarding of this traffic must happen transparently. -In particular, no TLS interception can be performed as the Kubernetes API depends on mutual TLS authentication. -VSHN will manage a DNS record pointing to this IP. -2. A public IP for HTTP(s) ingress. -Traffic to ports `80/tcp` and `443/tcp` on this IP must be forwarded to the infrastructure nodes in the machine network. -The PROXY protocol should be enabled to preserve source IPs. -Forwarding should happen transparently in TCP mode. -VSHN will manage a wildcard DNS record pointing to this IP. -Additional DNS records can be pointed to this IP by the customer. +We provide a CNAME target record to point additional DNS records to. === External services From 93b9942e551ad3e384dc752292ad97bde030f9d7 Mon Sep 17 00:00:00 2001 From: DebakelOrakel <78408921+DebakelOrakel@users.noreply.github.com> Date: Wed, 17 Jan 2024 07:25:01 +0100 Subject: [PATCH 3/3] Apply suggestions from code review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Christian Häusler <794584+corvus-ch@users.noreply.github.com> --- docs/modules/ROOT/pages/references/exoscale/architecture.adoc | 2 +- docs/modules/ROOT/partials/architecture/storage.adoc | 4 ++-- docs/modules/ROOT/partials/nav.adoc | 3 +-- 3 files changed, 4 insertions(+), 5 deletions(-) diff --git a/docs/modules/ROOT/pages/references/exoscale/architecture.adoc b/docs/modules/ROOT/pages/references/exoscale/architecture.adoc index 98e41667..7d8e38c7 100644 --- a/docs/modules/ROOT/pages/references/exoscale/architecture.adoc +++ b/docs/modules/ROOT/pages/references/exoscale/architecture.adoc @@ -26,7 +26,7 @@ See https://kb.vshn.ch/oc4/explanations/exoscale/limitations.html#_private_netwo The individual VMs are placed in https://community.exoscale.com/documentation/compute/security-groups[Security Groups] to restrict access and isolate the nodes from the public internet. -NOTE: On the {infra-type} environment there is no single stable egress IP. Every node uses a dynamic public IP for egress traffic, which it is not suited for any forms of whitelisting. +NOTE: On the {infra-type} environment there is no single stable egress IP. Every node uses a dynamic public IP for egress traffic, which it's not suited for any forms of whitelisting. === Virtual IPs diff --git a/docs/modules/ROOT/partials/architecture/storage.adoc b/docs/modules/ROOT/partials/architecture/storage.adoc index afb935ee..2e864c1d 100644 --- a/docs/modules/ROOT/partials/architecture/storage.adoc +++ b/docs/modules/ROOT/partials/architecture/storage.adoc @@ -24,8 +24,8 @@ The {infra-type} CSI driver is the in-cluster component which is responsible for endif::[] ifeval::["{infra-type}" == "Exoscale"] -NOTE: On the {infra-type} environment there is no Kubernetes integration available for such block devices. -https://products.vshn.ch/appuio/managed/storage_cluster.html[APPUiO Managed Storage Cluster] is available as and addon to the cluster offering if such functionality is required. +IMPORTANT: {infra-type} does not provide storage usable by Kubernetes as persistent volumes. +To fill this gap, {product} in {intra-type} uses https://products.vshn.ch/appuio/managed/storage_cluster.html[APPUiO Managed Storage Cluster] to provide storage to be used as read write once and read write many persistent volumes. endif::[] These devices hold application data, but backups are usually done from within the cluster. diff --git a/docs/modules/ROOT/partials/nav.adoc b/docs/modules/ROOT/partials/nav.adoc index c2db0333..970ac108 100644 --- a/docs/modules/ROOT/partials/nav.adoc +++ b/docs/modules/ROOT/partials/nav.adoc @@ -14,8 +14,7 @@ ** xref:oc4:ROOT:references/architecture/metering-data-flow-appuio-managed.adoc[Resource Usage Reporting] ** xref:oc4:ROOT:references/architecture/single_sign_on.adoc[] -** Exoscale -*** xref:oc4:ROOT:references/exoscale/architecture.adoc[Architecture] +** xref:oc4:ROOT:references/exoscale/architecture.adoc[Exoscale] *** xref:oc4:ROOT:explanations/exoscale/limitations.adoc[Limitations] ** Google Cloud Platform