[bitnami/spring-cloud-data-flow] Fixes and improvements (bitnami#2945)

* Ignore .DS_Store

* Spring Cloud Data Flow chart fixes and improvements:
1) added ability to configure Hibernate dialect
2) fixed issue with locating application.yaml in kubernetes cluster
3) added JDWP support
4) removed MariaDB from database comments, because external database can be PostgreSQL as well
5) fixed some wrong properties (externalDatabase.server -> externalDatabase.dataflow)

* Added changes to values-production.yaml.

* Bumped spring-cloud-dataflow to 0.1.3 version.

* Added hibernateDialect and jdwp variables to README.md.

* Fixed "missing starting space in comment"

* Fixed "java.sql.SQLInvalidAuthorizationSpecException: Access denied for user 'dataflow'@'10.244.0.5' (using password: YES)".

* Bump version to 0.2.0.

* Added support for external RabbitMQ.

* Added externalRabbitmq config to README.
Fixed typo: rabbitmq.rabbitmq -> rabbitmq.auth.

* Fixed config location, use /opt/bitnami/**/conf.

.gitignore

@@ -1,3 +1,4 @@
 *.tgz
 /.idea/*
 .vscode
+.DS_Store

bitnami/spring-cloud-dataflow/Chart.yaml

@@ -1,6 +1,6 @@
 apiVersion: v1
 name: spring-cloud-dataflow
-version: 0.1.4
+version: 0.2.0
 appVersion: 2.5.2
 description: Spring Cloud Data Flow is a microservices-based toolkit for building streaming and batch data processing pipelines in Cloud Foundry and Kubernetes.
 keywords:

bitnami/spring-cloud-dataflow/README.md

@@ -128,6 +128,8 @@ The following tables lists the configurable parameters of the Spring Cloud Data
 | `server.autoscaling.maxReplicas`               | Maximum number of Dataflow server replicas                          | `nil`                                                   |
 | `server.autoscaling.targetCPU`                 | Target CPU utilization percentage                                   | `nil`                                                   |
 | `server.autoscaling.targetMemory`              | Target Memory utilization percentage                                | `nil`                                                   |
+| `server.jdwp.enabled`                          | Enable Java Debug Wire Protocol (JDWP)                              | `false`                                                 |
+| `server.jdwp.port`                             | JDWP TCP port                                                       | `5005`                                                  |
 
 ### Dataflow Skipper parameters
 
@@ -178,6 +180,8 @@ The following tables lists the configurable parameters of the Spring Cloud Data
 | `skipper.autoscaling.maxReplicas`          | Maximum number of Skipper server replicas                           | `nil`                                                   |
 | `skipper.autoscaling.targetCPU`            | Target CPU utilization percentage                                   | `nil`                                                   |
 | `skipper.autoscaling.targetMemory`         | Target Memory utilization percentage                                | `nil`                                                   |
+| `skipper.jdwp.enabled`                     | Enable Java Debug Wire Protocol (JDWP)                              | `false`                                                 |
+| `skipper.jdwp.port`                        | JDWP TCP port                                                       | `5005`                                                  |
 | `externalSkipper.host`                     | Host of a external Skipper Server                                   | `localhost`                                             |
 | `externalSkipper.port`                     | External Skipper Server port number                                 | `7577`                                                  |
 
@@ -237,14 +241,20 @@ The following tables lists the configurable parameters of the Spring Cloud Data
 | `externalDatabase.dataflow.database`            | Name of the existing database to be used by Dataflow server             | `dataflow`                          |
 | `externalDatabase.skipper.user`                 | Existing username in the external db to be used by Skipper server       | `skipper`                           |
 | `externalDatabase.skipper.database`             | Name of the existing database to be used by Skipper server              | `skipper`                           |
+| `externalDatabase.hibernateDialect`             | Hibernate Dialect used by Dataflow/Skipper servers                      | `""`                                |
 
 ### RabbitMQ chart parameters
 
 | Parameter                                       | Description                                                             | Default                                                 |
 |-------------------------------------------------|-------------------------------------------------------------------------|---------------------------------------------------------|
 | `rabbitmq.enabled`                              | Enable/disable RabbitMQ chart installation                              | `true`                                                  |
-| `rabbitmq.auth.user`                            | RabbitMQ username                                                       | `user`                                                  |
+| `rabbitmq.auth.username`                        | RabbitMQ username                                                       | `user`                                                  |
 | `rabbitmq.auth.password`                        | RabbitMQ password                                                       | _random 40 character alphanumeric string_               |
+| `externalRabbitmq.enabled`                      | Enable/disable external RabbitMQ                                        | `false`                                                 |
+| `externalRabbitmq.host`                         | Host of the external RabbitMQ                                           | `localhost`                                             |
+| `externalRabbitmq.port`                         | External RabbitMQ port number                                           | `5672`                                                  |
+| `externalRabbitmq.username`                     | External RabbitMQ username                                              | `guest`                                                 |
+| `externalRabbitmq.password`                     | External RabbitMQ password                                              | `guest`                                                 |
 
 ### Kafka chart parameters
 

bitnami/spring-cloud-dataflow/templates/_helpers.tpl

@@ -150,7 +150,7 @@ Return true if a configmap object should be created for Spring Cloud Skipper
 {{- end -}}
 
 {{/*
-Return the MariaDB database Hostname
+Return the database Hostname
 */}}
 {{- define "scdf.database.host" -}}
 {{- if .Values.mariadb.enabled }}
@@ -161,7 +161,7 @@ Return the MariaDB database Hostname
 {{- end -}}
 
 {{/*
-Return the MariaDB database Port
+Return the database Port
 */}}
 {{- define "scdf.database.port" -}}
 {{- if .Values.mariadb.enabled }}
@@ -172,24 +172,35 @@ Return the MariaDB database Port
 {{- end -}}
 
 {{/*
-Return the MariaDB database driver
+Return the database driver
 */}}
 {{- define "scdf.database.driver" -}}
   {{- if .Values.mariadb.enabled -}}
     {{- printf "org.mariadb.jdbc.Driver" -}}
   {{- else -}}
-    {{- .Values.database.driver -}}
+    {{- .Values.externalDatabase.driver -}}
   {{- end -}}
 {{- end -}}
 
 {{/*
-Return the MariaDB database scheme
+Return the database scheme
 */}}
 {{- define "scdf.database.scheme" -}}
   {{- if .Values.mariadb.enabled -}}
     {{- printf "mariadb" -}}
   {{- else -}}
-    {{- .Values.database.scheme -}}
+    {{- .Values.externalDatabase.scheme -}}
+  {{- end -}}
+{{- end -}}
+
+{{/*
+Return the JDBC URL parameters
+*/}}
+{{- define "scdf.database.jdbc.parameters" -}}
+  {{- if .Values.mariadb.enabled -}}
+    {{- printf "?useMysqlMetadata=true" -}}
+  {{- else -}}
+    {{- printf "" -}}
   {{- end -}}
 {{- end -}}
 
@@ -200,7 +211,7 @@ Return the Data Flow Database Name
 {{- if .Values.mariadb.enabled }}
     {{- printf "dataflow" -}}
 {{- else -}}
-    {{- printf "%s" .Values.externalDatabase.server.database -}}
+    {{- printf "%s" .Values.externalDatabase.dataflow.database -}}
 {{- end -}}
 {{- end -}}
 
@@ -211,7 +222,7 @@ Return the Data Flow Database User
 {{- if .Values.mariadb.enabled }}
     {{- printf "dataflow" -}}
 {{- else -}}
-    {{- printf "%s" .Values.externalDatabase.server.user -}}
+    {{- printf "%s" .Values.externalDatabase.dataflow.user -}}
 {{- end -}}
 {{- end -}}
 
@@ -238,7 +249,7 @@ Return the Skipper Database User
 {{- end -}}
 
 {{/*
-Return the MariaDB secret name
+Return the Database secret name
 */}}
 {{- define "scdf.database.secretName" -}}
 {{- if .Values.mariadb.enabled }}
@@ -248,6 +259,61 @@ Return the MariaDB secret name
 {{- end -}}
 {{- end -}}
 
+{{/*
+Return the RabbitMQ host
+*/}}
+{{- define "scdf.rabbitmq.host" -}}
+{{- if .Values.rabbitmq.enabled }}
+    {{- printf "%s" (include "scdf.rabbitmq.fullname" .) -}}
+{{- else -}}
+    {{- printf "%s" .Values.externalRabbitmq.host -}}
+{{- end -}}
+{{- end -}}
+
+{{/*
+Return the RabbitMQ Port
+*/}}
+{{- define "scdf.rabbitmq.port" -}}
+{{- if .Values.rabbitmq.enabled }}
+    {{- printf "%d" (.Values.rabbitmq.service.port | int ) -}}
+{{- else -}}
+    {{- printf "%d" (.Values.externalRabbitmq.port | int ) -}}
+{{- end -}}
+{{- end -}}
+
+{{/*
+Return the RabbitMQ username
+*/}}
+{{- define "scdf.rabbitmq.user" -}}
+{{- if .Values.rabbitmq.enabled }}
+    {{- printf "%s" .Values.rabbitmq.auth.username -}}
+{{- else -}}
+    {{- printf "%s" .Values.externalRabbitmq.username -}}
+{{- end -}}
+{{- end -}}
+
+{{/*
+Return the RabbitMQ secret name
+*/}}
+{{- define "scdf.rabbitmq.secretName" -}}
+{{- if .Values.rabbitmq.enabled }}
+    {{- printf "%s" (include "scdf.rabbitmq.fullname" .) -}}
+{{- else -}}
+    {{- printf "%s-%s" (include "scdf.fullname" .) "externalrabbitmq" -}}
+{{- end -}}
+{{- end -}}
+
+{{/*
+Return the Hibernate dialect
+*/}}
+{{- define "scdf.database.hibernate.dialect" -}}
+  {{- if .Values.mariadb.enabled -}}
+    {{- printf "org.hibernate.dialect.MariaDB102Dialect" -}}
+  {{- else -}}
+    {{- .Values.externalDatabase.hibernateDialect -}}
+  {{- end -}}
+{{- end -}}
+
 {{/*
 Compile all warnings into a single message, and call fail.
 */}}

bitnami/spring-cloud-dataflow/templates/externaldb-secrets.yaml

@@ -2,7 +2,7 @@
 apiVersion: v1
 kind: Secret
 metadata:
-  name: {{ printf "%s-%s" (include "scdf.fullname" .) "externaldb" -}}
+  name: {{ printf "%s-%s" (include "scdf.fullname" .) "externaldb" }}
   labels: {{- include "common.labels.standard" . | nindent 4 }}
 type: Opaque
 data:

bitnami/spring-cloud-dataflow/templates/externalrabbitmq-secrets.yaml

@@ -0,0 +1,10 @@
+{{- if and (.Values.externalRabbitmq.enabled) (not .Values.rabbitmq.enabled) }}
+apiVersion: v1
+kind: Secret
+metadata:
+  name: {{ printf "%s-%s" (include "scdf.fullname" .) "externalrabbitmq" }}
+  labels: {{- include "common.labels.standard" . | nindent 4 }}
+type: Opaque
+data:
+  rabbitmq-password: {{ .Values.externalRabbitmq.password | b64enc | quote }}
+{{- end }}

bitnami/spring-cloud-dataflow/templates/scripts-configmap.yaml

@@ -23,7 +23,9 @@ data:
         echo "Rollout exit code: '${return_code}'"
         return $return_code
     }
+    {{- if .Values.mariadb.enabled }}
     k8s_wait_for_statefulset {{ $releaseNamespace }} {{ $mariadbFullname }}
+    {{- end }}
     {{- if or .Values.skipper.enabled .Values.server.configuration.streamingEnabled }}
     {{- if .Values.rabbitmq.enabled }}
     k8s_wait_for_statefulset {{ $releaseNamespace }} {{ $rabbitmqFullname }} 

bitnami/spring-cloud-dataflow/templates/server/configmap.yaml

@@ -56,16 +56,20 @@ data:
                         port: {{ $rsocketPort }}
             {{- end }}
           {{- end }}
+      {{- $hibernateDialect := include "scdf.database.hibernate.dialect" .  }}
+      {{- if $hibernateDialect }}
       jpa:
         properties:
           hibernate:
-            dialect: org.hibernate.dialect.MariaDB102Dialect
+            dialect: {{ $hibernateDialect }}
+      {{- end }}
       datasource:
         {{- $databaseScheme := include "scdf.database.scheme" .  }}
         {{- $databaseHost := include "scdf.database.host" .  }}
         {{- $databasePort := include "scdf.database.port" .  }}
         {{- $databaseName := include "scdf.database.server.name" .  }}
-        url: 'jdbc:{{ $databaseScheme }}://{{ $databaseHost }}:{{ $databasePort }}/{{ $databaseName }}?useMysqlMetadata=true'
+        {{- $jdbcParameters := include "scdf.database.jdbc.parameters" .  }}
+        url: 'jdbc:{{ $databaseScheme }}://{{ $databaseHost }}:{{ $databasePort }}/{{ $databaseName }}{{ $jdbcParameters }}'
         driverClassName: {{ include "scdf.database.driver" . }}
         username: {{ include "scdf.database.server.user" . }}
         password: ${mariadb-password}

bitnami/spring-cloud-dataflow/templates/server/deployment.yaml

@@ -84,12 +84,12 @@ spec:
               value: "8080"
             - name: SPRING_CLOUD_CONFIG_ENABLED
               value: "false"
+            - name: SPRING_CLOUD_KUBERNETES_CONFIG_ENABLE_API
+              value: "false"
             - name: SPRING_CLOUD_KUBERNETES_SECRETS_ENABLE_API
-              value: "true"
+              value: "false"
             - name: SPRING_CLOUD_KUBERNETES_SECRETS_PATHS
               value: "/etc/secrets"
-            - name: SPRING_CLOUD_KUBERNETES_CONFIG_NAME
-              value: {{ (include "scdf.server.configmapName" .) | quote }}
             - name: SPRING_CLOUD_DATAFLOW_SERVER_URI
               {{- if ne $serverServicePort 80 }}
               value: {{ printf "http://%s-server.%s.svc.%s:%d" $fullname $releaseNamespace $clusterDomain $serverServicePort | quote }}
@@ -122,6 +122,10 @@ spec:
                   fieldPath: metadata.namespace
             - name: KUBERNETES_TRUST_CERTIFICATES
               value: {{ ternary "true" "false" .Values.server.configuration.trustK8sCerts | quote }}
+            {{- if .Values.server.jdwp.enabled }}
+            - name: JAVA_TOOL_OPTIONS
+              value: "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address={{ .Values.server.jdwp.port }}"
+            {{- end }}
             {{- range $key, $value := .Values.server.extraEnvVars }}
             - name: {{ $key }}
               value: "{{ $value }}"
@@ -141,6 +145,11 @@ spec:
             - name: http
               containerPort: 8080
               protocol: TCP
+            {{- if .Values.server.jdwp.enabled }}
+            - name: jdwp
+              containerPort: {{ .Values.server.jdwp.port }}
+              protocol: TCP
+            {{- end }}
           {{- if .Values.server.livenessProbe.enabled }}
           livenessProbe:
             httpGet:
@@ -174,17 +183,25 @@ spec:
             - name: database
               mountPath: /etc/secrets/database
               readOnly: true
+            - name: config
+              mountPath: /opt/bitnami/spring-cloud-dataflow/conf
+              readOnly: true
         {{- if .Values.server.sidecars }}
         {{- include "common.tplvalues.render" ( dict "value" .Values.server.sidecars "context" $) | nindent 8 }}
         {{- end }}
       volumes:
         - name: database
           secret:
             secretName: {{ include "scdf.database.secretName" . }}
+        - name: config
+          configMap:
+            name: {{ include "scdf.server.configmapName" . }}
+            items:
+              - key: application.yaml
+                path: application.yml
         {{- if .Values.waitForBackends.enabled }}
         - name: scripts
           configMap:
             name: {{ include "scdf.fullname" . }}-scripts
             defaultMode: 0755
         {{- end }}
-

bitnami/spring-cloud-dataflow/templates/skipper/configmap.yaml

@@ -15,10 +15,10 @@ data:
               kubernetes:
                 accounts:
                   {{ .Values.skipper.configuration.accountName }}:
-                    {{- if .Values.rabbitmq.enabled }}
-                    {{- $rabbitmqHost := include "scdf.rabbitmq.fullname" . }}
-                    {{- $rabbitmqPort := int .Values.rabbitmq.service.port }}
-                    {{- $rabbitmqUser := .Values.rabbitmq.auth.username }}
+                    {{- if or (.Values.rabbitmq.enabled) (.Values.externalRabbitmq.enabled) }}
+                    {{- $rabbitmqHost := include "scdf.rabbitmq.host" . }}
+                    {{- $rabbitmqPort := include "scdf.rabbitmq.port" . }}
+                    {{- $rabbitmqUser := include "scdf.rabbitmq.user" . }}
                     environmentVariables: 'SPRING_RABBITMQ_HOST={{ $rabbitmqHost }},SPRING_RABBITMQ_PORT={{ $rabbitmqPort }},SPRING_RABBITMQ_USERNAME={{ $rabbitmqUser }},SPRING_RABBITMQ_PASSWORD=${rabbitmq-password}'
                     {{- else if .Values.kafka.enabled }}
                     environmentVariables: 'SPRING_CLOUD_STREAM_KAFKA_BINDER_BROKERS=${{ printf "{" }}{{ template "scdf.envrelease" . }}_KAFKA_SERVICE_HOST}:${{ printf "{" }}{{ template "scdf.envrelease" . }}_KAFKA_SERVICE_PORT},SPRING_CLOUD_STREAM_KAFKA_BINDER_ZK_NODES=${{ printf "{" }}{{ template "scdf.envrelease" . }}_ZOOKEEPER_SERVICE_HOST}:${{ printf "{" }}{{ template "scdf.envrelease" . }}_ZOOKEEPER_SERVICE_PORT}'
@@ -31,16 +31,20 @@ data:
                     {{- end }}
                     readinessProbeDelay: {{ .Values.deployer.readinessProbe.initialDelaySeconds }}
                     livenessProbeDelay: {{ .Values.deployer.livenessProbe.initialDelaySeconds }}
+      {{- $hibernateDialect := include "scdf.database.hibernate.dialect" .  }}
+      {{- if $hibernateDialect }}
       jpa:
         properties:
           hibernate:
-            dialect: org.hibernate.dialect.MariaDB102Dialect
+            dialect: {{ $hibernateDialect }}
+      {{- end }}
       datasource:
         {{- $databaseScheme := include "scdf.database.scheme" .  }}
         {{- $databaseHost := include "scdf.database.host" .  }}
         {{- $databasePort := include "scdf.database.port" .  }}
         {{- $databaseName := include "scdf.database.skipper.name" .  }}
-        url: 'jdbc:{{ $databaseScheme }}://{{ $databaseHost }}:{{ $databasePort }}/{{ $databaseName }}?useMysqlMetadata=true'
+        {{- $jdbcParameters := include "scdf.database.jdbc.parameters" .  }}
+        url: 'jdbc:{{ $databaseScheme }}://{{ $databaseHost }}:{{ $databasePort }}/{{ $databaseName }}{{ $jdbcParameters }}'
         driverClassName: {{ include "scdf.database.driver" . }}
         username: {{ include "scdf.database.skipper.user" . }}
         password: ${mariadb-password}