glint-cloud-config-server

This is a Golang implementation of the Spring Cloud Config Server, an HTTP resource-based API for external configuration. We have used this in Docker form via https://github.com/hyness/spring-cloud-config-server for over a year with no problems.

The aims for our project are:

• Complete compatibility with the existing Server API
• We have had to reimplement client-side functionality - that Spring Boot clients get for free - in library form for Golang-only clients. By shifting this to the server, a wider range of clients can get full value.
• Quicker server / container startup
• New capabilities of our choice and design

---

Server Usage:

Docker image:

docker pull glintpay/glint-cloud-config-server

Docker Hub

Docker run:

docker run -p 8888:80 \
    -v /here/conf/:/conf \
    -v /here/ssh/:/ssh \
    -e APP_CONFIG_FILE_YML_PATH=/conf/application.yml \
    glintpay/glint-cloud-config-server

Example configuration file:

server:
  port: 8888  # defaults to 80

defaults:
  flattenHierarchicalConfig: true
  flattenedIndexedLists: true
  resolvePropertySources: false

prometheus:
  path: /metrics  # ignored if not set

tracing:
  enabled: true
  endpoint: opentelemetry-traces:4318
  samplerFraction: 0.5  # Sample only 50% of traces

file:
  order: 0
  path: /config-dir
   
git:
  order: 1
  uri: [email protected]:Org/repo.git
  knownHostsFile: /ssh/github_known_hosts
  privateKey: |
  -----BEGIN RSA PRIVATE KEY-----
  ...
  -----END RSA PRIVATE KEY-----

  basedir: /tmp/cloud-config
  timeout: 30000
  clone-on-start: true
  force-pull: false
  show-progress: false  # log clones and pulls
  refreshRate: 0        # default = 0 secs => Fetch updated configuration from the Git repo every time it is requested

Testing:

One application, multiple ordered profiles, main Git branch:

http "localhost:8888/myapp/production-usa,production-base?resolve=true&flatten=true"

One application, one profile, custom label (labelled Git branch):

http "localhost:8888/myapp/testing-uk/refactor?resolve=true&flatten=true"

Inject, override custom properties:

http "localhost:8888/myapp/production-uk?resolve=true&flatten=true" baseUrl=http://uk-test bypass=true

---

Client Usage:

tbc

---

Concepts:

Application:

The primary discriminator for the application or client you are looking to configure, e.g. accounts

Multiple names can be supplied to represent groupings of applications, as a comma-separated list, e.g. accounts,finance-shared,security-aware

Ordering occurs from right to left, i.e. the more specific first. Thus security-aware is overridden by finance-shared, and in turn by accounts.

Profile:

Use profiles to group cross-application configuration by environment, data centre etc.

Multiple names can be supplied to represent groupings, as a comma-separated list, e.g. production-uk,production,uk-datacentre

Ordering occurs from right to left, i.e. the more specific first. Thus uk-datacentre is overridden by production, and in turn by production-uk.

Label:

Use this to switch competing configuration sets by version, by time (e.g. commit hash), or to pick up a refactoring branch.

This is a simple text value. If left blank, will default to the main Git branch, or equivalent "latest" for the backend in question.

Backend:

A repository for configuration files. Currently supported:

• Git
• File
• Kubernetes ConfigMap (planned)

Configurations are aggregated across all non-disabled repositories, ordered (if necessary) by the backend's configured order value.

Load:

Acquisition of the configurations for the applications / profiles / labels specified, across the available and enabled backends.

By default, a list of Spring Cloud Config Server-compatible PropertySources are available:

▶ http "localhost:8888/accounts/prod-us,prod/"
HTTP/1.1 200 OK
Content-Type: application/json
Date: Tue, 16 Aug 2022 16:22:38 GMT
Transfer-Encoding: chunked

{
    "label": "",
    "name": "accounts",
    "profiles": [
        "prod-us",
        "prod"
    ],
    "propertySources": [
        {
            "name": "[email protected]:Org/config.git/accounts-prod-us.yml",
            "source": {
                "foo": "bar",
            }
        },
        {
            "name": "[email protected]:Org/config.git/application-prod-us.yml",
            "source": {
                "mysql": {
                  "host": "prod-us-mysql.xxx"
                },
                "postgres": {
                  "host": "prod-us-pg.xxx"
                }
            }
        },
        {
            "name": "[email protected]:Org/config.git/application.yml",
            "source": {
                "mysql": {
                  "host": "test-mysql.xxx"
                },
                "postgres": {
                  "host": "test-pg.xxx"
                }
                [...]

The ordering of the PropertySources is non-deterministic.

Flattening:

Flattening of hierarchical data structures is enabled either at request time, via flatten=true, or by setting the application configuration:

defaults:
  flattenHierarchicalConfig: true

Which produces:

▶ http "localhost:8888/accounts/prod-us,prod/?flatten=true"
HTTP/1.1 200 OK
Content-Type: application/json
Date: Tue, 16 Aug 2022 16:22:38 GMT
Transfer-Encoding: chunked

{
    "label": "",
    "name": "accounts",
    "profiles": [
        "prod-us",
        "prod"
    ],
    "propertySources": [
        {
            "name": "[email protected]:Org/config.git/accounts-prod-us.yml",
            "source": {
                "foo": "bar",
            }
        },
        {
            "name": "[email protected]:Org/config.git/application-prod-us.yml",
            "source": {
                "mysql.host": "prod-us-mysql.xxx",
                "postgres.host": "prod-us-pg.xxx",
            }
        },
        {
            "name": "[email protected]:Org/config.git/application.yml",
            "source": {
                "mysql.host": "test-mysql.xxx",
                "postgres.host": "test-pg.xxx",
                [...]

Lists can be further flattened via:

defaults:
  flattenHierarchicalConfig: true
  flattenedIndexedLists: true

which turns:

"currencies": [
    "GBP",
    "EUR",
    "USD"
],

into:

"currencies[0]": "GBP",
"currencies[1]": "EUR",
"currencies[2]": "USD",

Resolution:

The resolution phase squashes these multiple property sources into one single source, according to the specified application and profile precedence ordering.

It is enabled either at request time, via resolve=true, or by setting the application configuration:

defaults:
  resolvePropertySources: false

Thus http://localhost:8888/service/env/?resolve=true applies the following:

service-env.yaml > service.yaml > application-env.yaml > application.yaml

The precise ordering applied can be seen via HTTP headers when resolution is enabled:

HTTP/1.1 200 OK
Content-Type: application/json
Date: Tue, 16 Aug 2022 12:00:12 GMT
Transfer-Encoding: chunked
X-Resolution-Label: 
X-Resolution-Name: service
X-Resolution-Precedencedisplaymessage: service-env.yaml > service.yaml > application-env.yaml > application.yaml
X-Resolution-Profiles: env
X-Resolution-Version: 281cd74427f6c5b87d7f8ff109a1e4e678b1b7cf

{
  [... json ...]

This can be combined with flattening. If flattening is disabled, a single hierarchical structure is returned.

---

Options

tbc

---

Observability

An OpenTelemetry exporter is available using the OTLP/HTTP protocol, if configured and enabled, e.g. via:

tracing:
  enabled: true
  endpoint: http://opentelemetry-traces:4318
  samplerFraction: 0.5  # Sample only 50% of traces

This allows instrumentation of HTTP requests with some additional observability of backend events.

---

Features

• Property resolution via ${propertyName} syntax, e.g.

  serviceName: myservice
  [...]
  mysql:
    dbName: ${propertyName}_db
  

• Support for Go templates, including Sprig functions, e.g.

  mysql:
    dbName: "{{{ dashToUnderscore (first .Applications) }}}_db"
  

• Property injection - client-side:

  Send a JSON structure of configuration property name / values to the REST endpoint via the PATCH verb.

  Property names prefixed with ^ will be applied before any externally loaded properties (i.e. lowest precendence). Else, properties will be applied after (i.e. highest precendence)

• Property injection - server-side:

  tbc

• Support for SOPS decryption

---

What is SOPS?

SOPS (Secrets OPerationS) is an editor for encrypted files, designed to manage sensitive data in configuration files.

It supports various formats like YAML, JSON, ENV, INI, and BINARY, but in the context of glint-cloud-config-server, YAML is the only supported format.

SOPS encrypts data using different key management systems, including:

• AWS KMS
• GCP KMS
• Azure Key Vault
• age
• PGP

For more detailed information, refer to the official SOPS documentation.

How to Use SOPS

1. Download SOPS: Obtain the latest stable release from the SOPS releases page, or use a package manager like Homebrew or DevBox.

2. Set up Encryption Keys: Choose your preferred encryption method and configure the necessary environment variables.

3. Encrypt Files: Use the sops encrypt command with the file path as an argument.

   SOPS will encrypt the file using the configured encryption method.

   SOPS-encrypted files contain all the necessary information for decryption, requiring only valid credentials and KMS key permissions.

4. Edit Encrypted Files: Use the sops edit command with the file path as an argument.

   SOPS will decrypt the file, open it in your default editor (defined by the $EDITOR environment variable, or vim if not set), and re-encrypt it upon saving.

5. Decrypt Files: To decrypt a file for viewing, use the -d flag with the sops decrypt command.

glint-cloud-config-server decrypts SOPS-encrypted configuration values before returning them to the client.

---

Missing features

• Configurable Helm chart
• Spring Cloud Bus-style event notifications