Skip to content

Commit

Permalink
Add Control Loop (#225)
Browse files Browse the repository at this point in the history 
* Migrate to kube_codegen.sh

* Migrate to reconciliation loop using controller-runtime (#208)

* Setup optional controller-runtime manager in main

Removes the kubeconfig flag and instead uses ctrl.RegisterFlags(fs)
 and ctrl.GetConfig(). The controller-runtime currently registers the kubeconfig flag, which lead to a redefined flag error when registering it again.

* Add update permissions for resource finalizers

* Add envtest to Makefile

This is based on the Makefile of an operator-sdk based project.

* Update test to include envtest and run the internal/controller test suite

* Add account, consumer and stream controller stubs to be implemented

Controllers and tests are based on files generated by operator-sdk.
Adds a minimal test suite for the controllers with a etcd test env and a test nats jetStream server to test against.

* Add logs to Reconcile functions

* Add internal/controller to jetstreamSrc

* Register account, consumer and stream reconcilers

* Add jsClient to test suit variables

* Remove format from log string

* feat(controller-runtime): Implement stream controller (#211)

* Add account, consumer and stream controller stubs to be implemented

Controllers and tests are based on files generated by operator-sdk.
Adds a minimal test suite for the controllers with a etcd test env and a test nats jetStream server to test against.

* Add logs to Reconcile functions

* Add jsClient to test suit variables

* Remove format from log string

* Make upsertCondition public to be used in new controllers

* Implement basic cases for stream reconciliation

See TODOs on what still needs to be implemented.

* refactor to use shared base controller

* Support jetstream connection options in stream spec

* implement stream deletion

* update observedGeneration of status

* check Spec.PreventDelete before stream deletion

* remove base js client

Use a single use client on every connection.
This should be replaced by a client pool in the future.

* move asJsonString to jetstream_controller

* check namespace read only and prevent update mode

* Update comments and log

* Fix test docs and check precondition

* Add preventUpdate test cases

* Add tests for read-only or namespace restricted mode

* fix empty ca when no ca set

Setting  CAs: []string{*ca} resulted in  []string{""} when no CA was set, leading to an error when creating clients.

* simplify error message

* fix error loop when the underlying stream was deleted

* refactor each phase into separate method

* Fix errors during parallel reconciliation & Refactor tests

- Trigger only on generation changes
- Split initialization and create into separate calls to Reconcile

* make test description strings more uniform

* Update docs and log messages

* extract configuration to buildNatsConfig method

* fix checking for preventDelete in the update step

Instead check for preventUpdate. Introduced during refactor.

* fix k8s binaries not downloaded for tests

* add /bin to gitignore

* rename stream helper functions

Prefix with stream to prevent conflict with other resources.

* update naming as suggested

* fix assumed reason in log message

* Update todo comments marked with review

 - Add note on opts.Account
 - Add comment on possible feature to expose TLSFirst in the spec.

* separate CA config from client cert and key

* set streamName and consumerName fields once on logger

Reword log messages.

* feat(controller-runtime): Add consumer controller (#212)

* implement consumerSpecToConfig

* implement consumer resource initialization

* implement consumer update/creation

* implement preventUpdate, readonly and namespace restrictions

Checks for the PreventUpdate or readonly mode during creation/update.
Skips reconciliation when resource is in namespace not matching restriction.

* test consumer creation on alternative server

* implement consumer deletion

* handle deletion when the underlying stream was deleted

* add missing GenerationChanged event filter to consumerReconciler

* update logging

Set streamName and consumerName fields once.
Reword log messages.

* feat(controller-runtime): Add keyvalue store spec and controller (#215)

* Bump helm/kind-action from 1.10.0 to 1.11.0 (#213)

Bumps [helm/kind-action](https://github.com/helm/kind-action) from 1.10.0 to 1.11.0.
- [Release notes](https://github.com/helm/kind-action/releases)
- [Commits](helm/kind-action@v1.10.0...v1.11.0)

---
updated-dependencies:
- dependency-name: helm/kind-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump helm/kind-action from 1.11.0 to 1.12.0 (#214)

Bumps [helm/kind-action](https://github.com/helm/kind-action) from 1.11.0 to 1.12.0.
- [Release notes](https://github.com/helm/kind-action/releases)
- [Commits](helm/kind-action@v1.11.0...v1.12.0)

---
updated-dependencies:
- dependency-name: helm/kind-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* * Formatting

* Add initial definitions for KeyValue store

* Deps

* Fix test

* Add KeyValue controller

* Add KeyValue tests

* Update PreventUpdate behavior

* Minor error handling change

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add initial object store types. Update Stream config and reorg to keep in-line with nats client.

* Add ObjectStore tests and remaining options

* Add test for sealed stream option

* Add Account Controller (#224)

* Add explicit Scheme field to Reconciler structs to better match convention

* Update spec to assume consumer field 'Name' over deprecated 'DurableName'

* Add account controller and support for account CRD auth config

* Update CRDs. Slight change to connection options

* Namespace and error handling improvements

* Improve deleted log message

* Namespace fix

* Add check for removed Push Consumer options

* Revert consumer name change

* Add InactiveThreshold parsing

* Fix test

* Add Watch for Account resource changes to trigger reconcile of dependent resources. Improve connection opts handling

* Add actual/desired state comparison for stream/consumer to avoid unnecessary update calls. Corrected ready state

* Fix duplicate resource tracking

* Improve config comparison logic

* Add flags for sync interval and cache directory

* Add back generation changed filter. Move finalizer add to after deletion check.

* Rework Reconcile scheduled sync

* Deps

* Create configured cache dir if DNE

* Move stream controller to jsm.go for pedantic mode

* Move consumer controller to jsm.go for pedantic mode

* Remove debug log entry

* deps

* Improve connection config priority. Add missing option from consumer CRD.

* Deps. Fix placement config enforcement

* Bump jsm.go. Fix typo

* Log diff on resource update

* Improve README. Modernize examples.

* Avoid excess disk writes to cache directory. README tweaks.

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Adrian <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
  • Loading branch information
@samuelattwood @adriandieter @dependabot
3 people authored Feb 18, 2025
1 parent 6c5157e commit e3bf14d
Show file tree
Hide file tree
Showing 112 changed files with 13,623 additions and 2,442 deletions.
1 change: 1 addition & 0 deletions .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,4 +18,5 @@
/nats-boot-config
/nats-boot-config.docker
/tools
/bin
/.idea
67 changes: 50 additions & 17 deletions Makefile
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
export GO111MODULE := on

SHELL=/usr/bin/env bash

ENVTEST_K8S_VERSION = 1.31.0

now := $(shell date -u +%Y-%m-%dT%H:%M:%S%z)
gitBranch := $(shell git rev-parse --abbrev-ref HEAD)
gitCommit := $(shell git rev-parse --short HEAD)
Expand All @@ -9,8 +13,7 @@ VERSION ?= version-not-set
linkerVars := -X main.BuildTime=$(now) -X main.GitInfo=$(gitBranch)-$(gitCommit)$(repoDirty) -X main.Version=$(VERSION)
drepo ?= natsio

jetstreamGenIn:= $(shell grep -l -R -F "// +k8s:" pkg/jetstream/apis)
jetstreamSrc := $(shell find cmd/jetstream-controller pkg/jetstream controllers/jetstream -name "*.go") pkg/jetstream/apis/jetstream/v1beta2/zz_generated.deepcopy.go
jetstreamSrc := $(shell find cmd/jetstream-controller pkg/jetstream internal/controller controllers/jetstream -name "*.go") pkg/jetstream/apis/jetstream/v1beta2/zz_generated.deepcopy.go

configReloaderSrc := $(shell find cmd/nats-server-config-reloader/ pkg/natsreloader/ -name "*.go")

Expand All @@ -27,20 +30,21 @@ default:
# make nats-server-config-reloader
# make nats-boot-config

pkg/jetstream/generated pkg/jetstream/apis/jetstream/v1beta2/zz_generated.deepcopy.go: fetch-modules $(jetstreamGenIn) pkg/k8scodegen/file-header.txt
generate: fetch-modules pkg/k8scodegen/file-header.txt
rm -rf pkg/jetstream/generated
# Temporary chmod fix until we migrate to kube_codegen.sh
D="$(codeGeneratorDir)"; : "$${D:=`go list -m -f '{{.Dir}}' k8s.io/code-generator`}"; \
chmod u+x "$$D/generate-internal-groups.sh"; \
GOFLAGS='' bash "$$D/generate-groups.sh" all \
github.com/nats-io/nack/pkg/jetstream/generated \
github.com/nats-io/nack/pkg/jetstream/apis \
"jetstream:v1beta2" \
--output-base . \
--go-header-file pkg/k8scodegen/file-header.txt
mv github.com/nats-io/nack/pkg/jetstream/generated pkg/jetstream/generated
mv github.com/nats-io/nack/pkg/jetstream/apis/jetstream/v1beta2/zz_generated.deepcopy.go pkg/jetstream/apis/jetstream/v1beta2/zz_generated.deepcopy.go
rm -rf github.com
source "$$D/kube_codegen.sh" ; \
kube::codegen::gen_helpers \
--boilerplate pkg/k8scodegen/file-header.txt \
pkg/jetstream/apis; \
kube::codegen::gen_client \
--with-watch \
--with-applyconfig \
--boilerplate pkg/k8scodegen/file-header.txt \
--output-dir pkg/jetstream/generated \
--output-pkg github.com/nats-io/nack/pkg/jetstream/generated \
--one-input-api jetstream/v1beta2 \
pkg/jetstream/apis

jetstream-controller: $(jetstreamSrc)
go build -race -o $@ \
Expand Down Expand Up @@ -173,10 +177,39 @@ fetch-modules:
.PHONY: build
build: jetstream-controller nats-server-config-reloader nats-boot-config

# Setup envtest tools based on a operator-sdk project makefile
LOCALBIN ?= $(shell pwd)/bin
$(LOCALBIN):
mkdir -p $(LOCALBIN)

# go-install-tool will 'go install' any package with custom target and name of binary, if it doesn't exist
# $1 - target path with name of binary (ideally with version)
# $2 - package url which can be installed
# $3 - specific version of package
define go-install-tool
@[ -f $(1) ] || { \
set -e; \
package=$(2)@$(3) ;\
echo "Downloading $${package}" ;\
GOBIN=$(LOCALBIN) go install $${package} ;\
mv "$$(echo "$(1)" | sed "s/-$(3)$$//")" $(1) ;\
}
endef

ENVTEST ?= $(LOCALBIN)/setup-envtest-$(ENVTEST_VERSION)
ENVTEST_VERSION ?= release-0.19

.PHONY: envtest
envtest: $(ENVTEST) ## Download setup-envtest locally if necessary.
$(ENVTEST): $(LOCALBIN)
$(call go-install-tool,$(ENVTEST),sigs.k8s.io/controller-runtime/tools/setup-envtest,$(ENVTEST_VERSION))


.PHONY: test
test:
go vet ./controllers/... ./pkg/natsreloader/...
go test -race -cover -count=1 -timeout 10s ./controllers/... ./pkg/natsreloader/...
test: envtest
go vet ./controllers/... ./pkg/natsreloader/... ./internal/controller/...
$(ENVTEST) use $(ENVTEST_K8S_VERSION) --bin-dir $(LOCALBIN) -p path ## Get k8s binaries
go test -race -cover -count=1 -timeout 10s ./controllers/... ./pkg/natsreloader/... ./internal/controller/...

.PHONY: clean
clean:
Expand Down
93 changes: 63 additions & 30 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,22 +9,30 @@

## JetStream Controller

The JetStream controllers allows you to manage [NATS JetStream](https://github.com/nats-io/jetstream) [Streams](https://github.com/nats-io/jetstream#streams-1) and [Consumers](https://github.com/nats-io/jetstream#consumers-1) via K8S CRDs.
The JetStream controllers allows you to manage [NATS JetStream](https://docs.nats.io/nats-concepts/jetstream) [Streams](https://docs.nats.io/nats-concepts/jetstream/streams), [Consumers](https://docs.nats.io/nats-concepts/jetstream/consumers), [Key/Value Stores](https://docs.nats.io/nats-concepts/jetstream/key-value-store), and [Object Stores](https://docs.nats.io/nats-concepts/jetstream/obj_store) via Kubernetes CRDs.

Resources managed by NACK controllers are expected to _exclusively_ be managed by NACK, and configuration state will be enforced if mutated by an external client.

## [API Reference](docs/api.md)

### Getting started

First install the JetStream CRDs:
Install with Helm:

```sh
$ kubectl apply -f https://github.com/nats-io/nack/releases/latest/download/crds.yml
```
helm repo add nats https://nats-io.github.io/k8s/helm/charts/
helm upgrade --install nats nats/nats --set config.jetstream.enabled=true --set config.cluster.enabled=true
helm upgrade --install nack nats/nack --set jetstream.nats.url=nats://nats.default.svc.cluster.local:4222
```

#### (Optional) Enable Experimental `controller-runtime` Controllers

Now install with Helm:
> **Note**: The updated controllers will more reliably enforce resource state. If migrating from an older version of NACK, as long as all NATS resources are in-sync with NACK resources no modifications are expected.
>
> The `jetstream-controller` logs will contain a diff of any changes the controller has made.
```
helm repo add nats https://nats-io.github.io/k8s/helm/charts/
helm install nats nats/nats --set=config.jetstream.enabled=true
helm install nack nats/nack --set jetstream.nats.url=nats://nats:4222
helm upgrade nack nats/nack --set jetstream.additionalArgs={--control-loop=true}
```

#### Creating Streams and Consumers
Expand Down Expand Up @@ -66,6 +74,28 @@ spec:
filterSubject: orders.received
maxDeliver: 20
ackPolicy: explicit
---
apiVersion: jetstream.nats.io/v1beta2
kind: KeyValue
metadata:
name: my-key-value
spec:
bucket: my-key-value
history: 20
storage: file
maxBytes: 2048
compression: true
---
apiVersion: jetstream.nats.io/v1beta2
kind: ObjectStore
metadata:
name: my-object-store
spec:
bucket: my-object-store
storage: file
replicas: 1
maxBytes: 536870912 # 512 MB
compression: true
```
```sh
Expand All @@ -75,7 +105,7 @@ $ kubectl apply -f https://raw.githubusercontent.com/nats-io/nack/main/deploy/ex
# Check if it was successfully created.
$ kubectl get streams
NAME STATE STREAM NAME SUBJECTS
mystream Created mystream [orders.*]
mystream Ready mystream [orders.*]

# Create a push-based consumer
$ kubectl apply -f https://raw.githubusercontent.com/nats-io/nack/main/deploy/examples/consumer_push.yml
Expand All @@ -86,8 +116,8 @@ $ kubectl apply -f https://raw.githubusercontent.com/nats-io/nack/main/deploy/ex
# Check if they were successfully created.
$ kubectl get consumers
NAME STATE STREAM CONSUMER ACK POLICY
my-pull-consumer Created mystream my-pull-consumer explicit
my-push-consumer Created mystream my-push-consumer none
my-pull-consumer Ready mystream my-pull-consumer explicit
my-push-consumer Ready mystream my-push-consumer none

# If you end up in an Errored state, run kubectl describe for more info.
# kubectl describe streams mystream
Expand Down Expand Up @@ -178,7 +208,7 @@ metadata:
spec:
name: a
servers:
- nats://nats:4222
- nats://nats:4222
tls:
secret:
name: nack-a-tls
Expand Down Expand Up @@ -209,31 +239,32 @@ Server URL and TLS certificates.
```sh
# Install cert-manager
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.6.0/cert-manager.yaml
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.17.0/cert-manager.yaml
# Install TLS certs
cd examples/secure
# Install certificate issuer
kubectl apply -f issuer.yaml
# Install account A cert
kubectl apply -f nack-a-client-tls.yaml
# Install server cert
kubectl apply -f server-tls.yaml
# Install nats-box cert
kubectl apply -f client-tls.yaml
# Install NATS cluster
helm install -f nats-helm.yaml nats nats/nats
helm upgrade --install -f nats-helm.yaml nats nats/nats
# Verify pods are healthy
kubectl get pods
# Install nats-box to run nats cli later
kubectl apply -f nats-client-box.yaml
# Install JetStream Controller from nack
helm install --set jetstream.enabled=true jetstream-controller nats/nack
# Install CRDs
kubectl apply -f ../../deploy/crds.yml
helm upgrade --install nack nats/nack --set jetstream.enabled=true
# Verify pods are healthy
kubectl get pods
Expand All @@ -242,6 +273,7 @@ kubectl apply -f nack/nats-account-a.yaml
# Create stream using account A
kubectl apply -f nack/nats-stream-foo-a.yaml
# Create consumer using account A
kubectl apply -f nack/nats-consumer-bar-a.yaml
```
Expand All @@ -251,30 +283,28 @@ container to run the management CLI.
```sh
# Get container shell
kubectl exec -it nats-client-box-abc-123 -- sh
# Change to TLS directory
cd /etc/nats-certs/clients/nack-a-tls
kubectl exec -it deployment/nats-box -- /bin/sh
```
There should now be some Streams available, verify with `nats` command.
```sh
# List streams
nats --tlscert tls.crt --tlskey tls.key --tlsca ca.crt -s tls://nats.default.svc.cluster.local stream ls
nats stream ls
```
You can now publish messages on a Stream.
```sh
# Push message
nats --tlscert tls.crt --tlskey tls.key --tlsca ca.crt -s tls://nats.default.svc.cluster.local pub foo hi
nats pub foo hi
```
And pull messages from a Consumer.
```sh
# Pull message
nats --tlscert tls.crt --tlskey tls.key --tlsca ca.crt -s tls://nats.default.svc.cluster.local consumer next foo bar
nats consumer next foo bar
```
### Local Development
Expand All @@ -298,6 +328,7 @@ nats-server -DV -js
```
Build Docker image
```sh
make jetstream-controller-docker ver=1.2.3
```
Expand All @@ -314,15 +345,15 @@ For more information see the
```
helm repo add nats https://nats-io.github.io/k8s/helm/charts/
helm install my-nats nats/nats
helm upgrade --install nats nats/nats
```
### Configuring
```yaml
reloader:
enabled: true
image: natsio/nats-server-config-reloader:0.6.0
image: natsio/nats-server-config-reloader:0.16.1
pullPolicy: IfNotPresent
```
Expand All @@ -337,6 +368,7 @@ make nats-server-config-reloader
```
Build Docker image
```sh
make nats-server-config-reloader-docker ver=1.2.3
```
Expand All @@ -350,14 +382,14 @@ For more information see the
```
helm repo add nats https://nats-io.github.io/k8s/helm/charts/
helm install my-nats nats/nats
helm upgrade --install nats nats/nats
```
### Configuring
```yaml
bootconfig:
image: natsio/nats-boot-config:0.5.2
image: natsio/nats-boot-config:0.16.1
pullPolicy: IfNotPresent
```
Expand All @@ -372,6 +404,7 @@ make nats-boot-config
```
Build Docker image
```sh
make nats-boot-config-docker ver=1.2.3
```
6 changes: 3 additions & 3 deletions cicd/Dockerfile
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#syntax=docker/dockerfile-upstream:1.5
#syntax=docker/dockerfile:1.13
ARG GO_APP

FROM alpine:3.21.2 as deps
FROM alpine:3.21.3 as deps

ARG GO_APP
ARG GORELEASER_DIST_DIR=/go/src/dist
Expand All @@ -28,7 +28,7 @@ RUN <<EOT
cp ${BIN_PATH} /go/bin
EOT

FROM alpine:3.21.2
FROM alpine:3.21.3

ARG GO_APP
ENV GO_APP ${GO_APP}
Expand Down
3 changes: 2 additions & 1 deletion cicd/Dockerfile_goreleaser
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
#syntax=docker/dockerfile-upstream:1.5
#syntax=docker/dockerfile:1.13
FROM --platform=$BUILDPLATFORM golang:1.24.0-bullseye as build


RUN <<EOT
set -e

Expand Down
Loading

0 comments on commit e3bf14d

Please sign in to comment.