diff --git a/daprblog/content/posts/2023/dapr_flomesh/flomesh_dapr_blog_arch.png b/daprblog/content/posts/2023/dapr_flomesh/flomesh_dapr_blog_arch.png new file mode 100644 index 00000000..7484b033 Binary files /dev/null and b/daprblog/content/posts/2023/dapr_flomesh/flomesh_dapr_blog_arch.png differ diff --git a/daprblog/content/posts/2023/dapr_flomesh/flomesh_dapr_blog_demo_arch.png b/daprblog/content/posts/2023/dapr_flomesh/flomesh_dapr_blog_demo_arch.png new file mode 100644 index 00000000..a40b0363 Binary files /dev/null and b/daprblog/content/posts/2023/dapr_flomesh/flomesh_dapr_blog_demo_arch.png differ diff --git a/daprblog/content/posts/2023/dapr_flomesh/index.md b/daprblog/content/posts/2023/dapr_flomesh/index.md new file mode 100644 index 00000000..b2cdccf7 --- /dev/null +++ b/daprblog/content/posts/2023/dapr_flomesh/index.md @@ -0,0 +1,573 @@ +--- +date: "2023-02-26T07:00:00-07:00" +title: "Achieving Multi-cluster Dapr support with Flomesh Service Mesh" +linkTitle: "Achieving Multi-cluster Dapr support with Flomesh Service Mesh" +author: "Addo Zhang - Flomesh" +type: blog +--- + +How combining Dapr with [Flomesh](https://flomesh.io) Service Mesh brings Dapr functionality to multi-cluster services. + +## Background + +The continuous evolution of technology and architecture is trending towards multiple runtimes. The basic capabilities of modern applications are consistently separated from the application as independent runtimes. Among them are distributed application runtimes and service meshes. This article introduces the integration of [Dapr](https://dapr.io/) and [Flomesh](https://flomesh.io) Service Mesh to achieve "true" cross-cluster interconnectivity for service invocation. + +### Multi-Cluster + +Kubernetes adheres to a design philosophy of loose coupling and scalability, which encourages the flourishing development of the Kubernetes ecosystem. However, since most of these are limited to a single cluster, more and more clusters are being created within enterprises for various reasons and purposes, such as: +- Single-cluster failures +- Regulatory requirements +- Cross-data center disaster recovery for multiple regions +- Hybrid clouds for agility and cost reduction +- Multi-cloud deployment +- Limited capacity of a single cluster +- Coexistence of multiple versions of Kubernetes clusters, etc. + + +### Dapr + +[Dapr](https://dapr.io) is a distributed application toolkit that decouples applications and peripheral functional components by providing simple and stable APIs. This enables developers to focus on business functionality development. By decoupling from peripheral components, applications become more portable and cloud-native. Enterprises can easily migrate applications to different environments at a low cost. + +Dapr provides rich features such as service invocation, resilience policies, state storage, publish/subscribe, binding, distributed locking, name resolution, etc. However, it does not support advanced service governance functions, such as canary release and cross-cluster service invocation. + +### Flomesh Service Mesh + +With the rise of microservices architecture, as the scale grows larger, the difficulty and fragmentation of service governance has significantly increased. The emergence of service meshes solves these problems. A service mesh is a dedicated infrastructure layer for handling communication between services. Through it, functionality such as observability, traffic management, and security can be added transparently without adding it to your code. + +[Flomesh](https://flomesh.io) service mesh uses the programmable proxy [Pipy](https://github.com/flomesh-io/pipy) as the core to provide east-west and north-south traffic management. Through L7-based traffic management capabilities, it breaks through the network isolation between computing environments, proposes a virtual flat network, and enables communication between applications in different computing environments. Flomesh service mesh can be imagined as a "big mesh" that covers multiple clusters. Flomesh service mesh is a suite of components including [osm-edge](https://flomesh.io/osm-edge) a lightweight Service Mesh Interface (SMI) compatible service mesh for, [FSM](https://github.com/flomesh-io/fsm) components like Ingress controller, Egress gateway, Loadbalancer, Kubernetes MCS API implementation, Kubernetes Gateway API implementation etc. + +{{< imgproc flomesh_dapr_blog_arch.png Resize "800x" />}} + +## Demo + +The diagram below demonstrates how Dapr and Flomesh service mesh can be integrated to enable cross-cluster service communication. The example consists of two components: a server-side NodeApp and a client-side curl. + +The server-side NodeApp is a Dapr application, [modified from the NodeApp in the Dapr hello-kubernetes](https://github.com/addozhang/flomesh-dapr-demo/tree/main/nodeapp) tutorial example. When it returns a response, it displays the name of the current cluster. + +The client-side curl is used to send requests to the NodeApp, but is not declared as a Dapr application. This demonstrates how Flomesh service mesh can enable communication between a Dapr application and a non-Dapr application, facilitating cross-cluster communication. + + +{{< imgproc flomesh_dapr_blog_demo_arch.png Resize "1200x" />}} + + +The NodeApp in the example has three endpoints: + +- `GET /ports`: returns the ports that the application can be accessed on +- `POST /neworder`: creates a new order +- `GET /order`: queries an order + +Follow along with the demo to: +1. Create multiple clusters. +1. Configure the environment. +1. Install and configure various components. +1. Deploy the application. + +### Before you begin + + +#### One-click installation script + +For this demo, we provide a quick, one-click installation script to avoid the tedious configuration of the environment and components. [Get the script content from GitHub](https://github.com/addozhang/flomesh-dapr-demo). + +Before using the one-click script, install `Docker` and `kubectl`. The script will check during runtime and install tools such as `k3d`, `helm`, `jq`, `pv`, etc. Commands include: + +| Command | Description | +| --------------- | ------------- | +| `flomesh.sh` | Provides no parameters, the script will create 4 clusters, complete the environment installation and configuration, and run the demo | +| `flomesh.sh -h` | Prints help information | +| `flomesh.sh -i` | Creates 4 clusters, completes environment installation and configuration | +| `flomesh.sh -d` | Runs the demo | +| `flomesh.sh -r` | Cleans up demo-related resources | +| `flomesh.sh -u` | Deletes all clusters | + +After downloading the script, execute the following command to install and configure the environment and run the demo. + +```shell +curl -sL https://raw.githubusercontent.com/addozhang/flomesh-dapr-demo/main/flomesh.sh | bash - +``` + +#### Prerequisites + +To perform the demonstration, you need the following tools: + +- Docker +- Kubectl +- K3d +- Helm +- kubectx + + +### Create multiple clusters + +Obtain the IP address of the local machine as the communication address between clusters. + +```shell +export HOST_IP=10.0.0.13 +``` + +To create four clusters named `control-plane`, `cluster-1`, `cluster-2`, and `cluster-3`, execute the following command: + +```shell +API_PORT=6444 #6444 6445 6446 6447 +PORT=80 #81 82 83 +for CLUSTER_NAME in control-plane cluster-1 cluster-2 cluster-3 +do + k3d cluster create ${CLUSTER_NAME} \ + --image docker.io/rancher/k3s:v1.23.8-k3s2 \ + --api-port "${HOST_IP}:${API_PORT}" \ + --port "${PORT}:80@server:0" \ + --servers-memory 4g \ + --k3s-arg "--disable=traefik@server:0" \ + --network multi-clusters \ + --timeout 120s \ + --wait + ((API_PORT=API_PORT+1)) + ((PORT=PORT+1)) +done +``` + +The above command: +- Creates four `k3d` clusters +- Installs and configures the necessary components for the Flomesh service mesh on each cluster + +### Install Flomesh service mesh + +Install Flomesh with the following command: + +```shell +helm repo add fsm https://charts.flomesh.io +helm repo update +export FSM_NAMESPACE=flomesh +export FSM_VERSION=0.2.1-alpha.3 +for CLUSTER_NAME in control-plane cluster-1 cluster-2 cluster-3 +do + kubectx k3d-${CLUSTER_NAME} + sleep 1 + helm install --namespace ${FSM_NAMESPACE} --create-namespace --version=${FSM_VERSION} --set fsm.logLevel=5 fsm fsm/fsm + sleep 1 + kubectl wait --for=condition=ready pod --all -n $FSM_NAMESPACE --timeout=120s +done +``` + +To join the clusters `cluster-1`, `cluster-2`, and `cluster-3` to the `control-plane` cluster, run the following command: + + +```shell +kubectx k3d-control-plane +sleep 1 +PORT=81 +for CLUSTER_NAME in cluster-1 cluster-2 cluster-3 +do + kubectl apply -f - < 443/TCP 43h +``` + +To allow the pods to access the apiserver directly without going through the sidecar: + +```shell +for CLUSTER_NAME in cluster-1 cluster-2 cluster-3 +do + kubectx k3d-${CLUSTER_NAME} + kubectl patch meshconfig osm-mesh-config -n $OSM_NAMESPACE -p '{"spec":{"traffic":{"outboundIPRangeExclusionList":["10.43.0.1/32"]}}}' --type=merge +done +``` + +### Install Dapr + +Invoke below command to install Dapr on `cluster-1`, `cluster-2`, and `cluster-3` : + +```shell +for CLUSTER_NAME in cluster-1 cluster-2 cluster-3 +do + kubectx k3d-${CLUSTER_NAME} + dapr init --kubernetes \ + --enable-mtls=false \ + --wait +done +``` + +Check the status of the components: + +```shell +dapr status -k + NAME NAMESPACE HEALTHY STATUS REPLICAS VERSION AGE CREATED + dapr-placement-server dapr-system True Running 1 1.9.6 2m 2023-02-09 10:36.51 + dapr-operator dapr-system True Running 1 1.9.6 2m 2023-02-09 10:36.51 + dapr-dashboard dapr-system True Running 1 0.11.0 2m 2023-02-09 10:36.51 + dapr-sentry dapr-system True Running 1 1.9.6 2m 2023-02-09 10:36.51 + dapr-sidecar-injector dapr-system True Running 1 1.9.6 2m 2023-02-09 10:36.51 +``` + +Check all services and their respective ports: + +```shell +kubectl get svc -n dapr-system +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +dapr-placement-server ClusterIP None 50005/TCP,8201/TCP 5h50m +dapr-sidecar-injector ClusterIP 10.43.12.213 443/TCP 5h50m +dapr-webhook ClusterIP 10.43.103.31 443/TCP 5h50m +dapr-dashboard ClusterIP 10.43.172.156 8080/TCP 5h50m +dapr-api ClusterIP 10.43.126.14 80/TCP 5h50m +dapr-sentry ClusterIP 10.43.41.10 80/TCP 5h50m +``` + +To allow traffic to flow through the Dapr components and Redis ports, add the necessary traffic rules to the OSM data plane. + +```shell +for CLUSTER_NAME in cluster-1 cluster-2 cluster-3 +do + kubectx k3d-${CLUSTER_NAME} + kubectl patch meshconfig osm-mesh-config -n $OSM_NAMESPACE -p '{"spec":{"traffic":{"outboundPortExclusionList":[50005,8201,6379]}}}' --type=merge +done +``` + +### Install Redis +Install Redis with the following YAML: + +```yaml +version: '3' +services: + redis: + image: redis:latest + container_name: redis + ports: + - 6379:6379 + volumes: + - ./data:/data + command: redis-server --appendonly yes --requirepass changeme +``` + +### Create namespace + +Create and add a namespace to be monitored by osm-edge Service Mesh + +```shell +export NAMESPACE=dapr-test +for CLUSTER_NAME in cluster-1 cluster-2 cluster-3 +do + kubectx k3d-${CLUSTER_NAME} + kubectl create namespace $NAMESPACE + osm namespace add $NAMESPACE +done +``` + +### Register Redis state store component + +Run the following command to register the Redis state store component: + +```shell +export NAMESPACE=dapr-test +for CLUSTER_NAME in cluster-1 cluster-2 cluster-3 +do + kubectx k3d-${CLUSTER_NAME} + kubectl create secret generic redis -n $NAMESPACE --from-literal=redis-password=changeme + kubectl apply -n $NAMESPACE -f - <