Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tenant stuck at provisioning initial users #2285

Closed
aqeelat opened this issue Aug 22, 2024 · 1 comment
Closed

Tenant stuck at provisioning initial users #2285

aqeelat opened this issue Aug 22, 2024 · 1 comment
Labels
community triage

Comments

@aqeelat
Copy link

aqeelat commented Aug 22, 2024

I'm deploying the tenant using this values.yaml:

secrets:
tenant:
  name: minio
  image:
    repository: quay.io/minio/minio
    pullPolicy: IfNotPresent
  configuration:
    name: minio-env-configuration
  configSecret:
    name: minio-env-configuration
    existingSecret: true
  pools:
    - servers: 4
      storageClassName: xfs
  metrics:
    enabled: false
  buckets:
    - name: bucket1
    - name: bucket2
  users:
    - name: "minio-user-dev"

However, the cluster does not get fully initialized and throws this event Users creation failed: The Access Key Id you provided does not exist in our records.

Operator logs:

I0822 07:11:33.687999       1 minio.go:185] Generating private key
I0822 07:11:33.688056       1 minio.go:198] Generating CSR with CN=*.minio-hl.minio.svc.cluster.local
I0822 07:11:33.702789       1 csr.go:183] Start polling for certificate of csr/minio-minio-csr, every 5s, timeout after 20m0s
I0822 07:11:33.702882       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144761320", FieldPath:""}): type: 'Normal' reason: 'CSRCreated' MinIO CSR Created
I0822 07:11:38.707749       1 csr.go:209] Certificate successfully fetched, creating secret with Private key and Certificate
I0822 07:11:43.959644       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144761492", FieldPath:""}): type: 'Normal' reason: 'SvcCreated' MinIO Service Created
I0822 07:11:43.967932       1 status.go:55] Hit conflict issue, getting latest version of tenant
I0822 07:11:44.009715       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144761499", FieldPath:""}): type: 'Normal' reason: 'SvcCreated' Console Service Created
I0822 07:11:44.020353       1 status.go:55] Hit conflict issue, getting latest version of tenant
I0822 07:11:44.117219       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144761505", FieldPath:""}): type: 'Normal' reason: 'SvcCreated' Headless Service created
I0822 07:11:44.136501       1 sync.go:151] created key minio/minio-sa kind /v1, Kind=ServiceAccount diff [ObjectMeta.UID:  != ab0f04c2-995a-4b66-a988-46ef7f225229 ObjectMeta.ResourceVersion:  != 144761512 ObjectMeta.CreationTimestamp.Time: 0001-01-01 00:00:00 +0000 UTC != 2024-08-22 07:11:44 +0000 UTC ObjectMeta.OwnerReferences: <nil slice> != [{minio.min.io/v2 Tenant minio efcb0a18-41cb-4af6-a632-af2c1000c060 0xc001a3fef5 0xc001a3fef6}] ObjectMeta.ManagedFields: <nil slice> != [{minio-operator Update v1 2024-08-22 07:11:44 +0000 UTC FieldsV1 {"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"efcb0a18-41cb-4af6-a632-af2c1000c060\"}":{}}}} }]]
I0822 07:11:44.176652       1 sync.go:151] created key minio/minio-role kind rbac.authorization.k8s.io/v1, Kind=Role diff [ObjectMeta.UID:  != 1defb74a-7507-4657-b919-9bf3530a9710 ObjectMeta.ResourceVersion:  != 144761513 ObjectMeta.CreationTimestamp.Time: 0001-01-01 00:00:00 +0000 UTC != 2024-08-22 07:11:44 +0000 UTC ObjectMeta.OwnerReferences: <nil slice> != [{minio.min.io/v2 Tenant minio efcb0a18-41cb-4af6-a632-af2c1000c060 0xc000f14be5 0xc000f14be6}] ObjectMeta.ManagedFields: <nil slice> != [{minio-operator Update rbac.authorization.k8s.io/v1 2024-08-22 07:11:44 +0000 UTC FieldsV1 {"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"efcb0a18-41cb-4af6-a632-af2c1000c060\"}":{}}},"f:rules":{}} }]]
I0822 07:11:44.214944       1 sync.go:151] created key minio/minio-binding kind rbac.authorization.k8s.io/v1, Kind=RoleBinding diff [ObjectMeta.UID:  != 334ff367-5939-4d1f-8871-2c9a3809b4de ObjectMeta.ResourceVersion:  != 144761515 ObjectMeta.CreationTimestamp.Time: 0001-01-01 00:00:00 +0000 UTC != 2024-08-22 07:11:44 +0000 UTC ObjectMeta.OwnerReferences: <nil slice> != [{minio.min.io/v2 Tenant minio efcb0a18-41cb-4af6-a632-af2c1000c060 0xc001233565 0xc001233566}] ObjectMeta.ManagedFields: <nil slice> != [{minio-operator Update rbac.authorization.k8s.io/v1 2024-08-22 07:11:44 +0000 UTC FieldsV1 {"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"efcb0a18-41cb-4af6-a632-af2c1000c060\"}":{}}},"f:roleRef":{},"f:subjects":{}} }]]
I0822 07:11:44.230981       1 status.go:89] Hit conflict issue, getting latest version of tenant
I0822 07:11:44.270291       1 main-controller.go:985] Detected we are updating a legacy tenant deployment
I0822 07:11:44.473768       1 main-controller.go:1030] 'minio/minio': Deploying pool pool-0
I0822 07:11:44.687930       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144761529", FieldPath:""}): type: 'Normal' reason: 'PoolCreated' Tenant pool pool-0 created
I0822 07:11:45.672032       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144761591", FieldPath:""}): type: 'Warning' reason: 'WaitingMinIOIsHealthy' Waiting for MinIO to be ready: %!s(<nil>)
I0822 07:11:48.714573       1 minio-services.go:156] Headless Services don't match: service ports don't match
I0822 07:11:48.724080       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144761644", FieldPath:""}): type: 'Normal' reason: 'Updated' Headless Service Updated
I0822 07:11:48.767165       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144761644", FieldPath:""}): type: 'Warning' reason: 'WaitingMinIOIsHealthy' Waiting for MinIO to be ready: %!s(<nil>)
I0822 07:11:53.797707       1 minio-services.go:156] Headless Services don't match: service ports don't match
I0822 07:11:53.803247       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144761732", FieldPath:""}): type: 'Normal' reason: 'Updated' Headless Service Updated
I0822 07:11:53.846845       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144761732", FieldPath:""}): type: 'Warning' reason: 'WaitingMinIOIsHealthy' Waiting for MinIO to be ready: %!s(<nil>)
I0822 07:11:58.863896       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144761830", FieldPath:""}): type: 'Warning' reason: 'WaitingMinIOIsHealthy' Waiting for MinIO to be ready: %!s(<nil>)
I0822 07:12:01.068455       1 monitoring.go:123] 'minio/minio' Failed to get cluster health: Get "https://minio.minio.svc.cluster.local/minio/health/cluster": dial tcp 10.57.235.22:443: i/o timeout
I0822 07:12:03.896762       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144761940", FieldPath:""}): type: 'Warning' reason: 'WaitingMinIOIsHealthy' Waiting for MinIO to be ready: %!s(<nil>)
I0822 07:12:08.924281       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144762041", FieldPath:""}): type: 'Warning' reason: 'WaitingMinIOIsHealthy' Waiting for MinIO to be ready: %!s(<nil>)
I0822 07:12:08.972778       1 status.go:89] Hit conflict issue, getting latest version of tenant
I0822 07:12:09.007298       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:09.050063       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:10.148741       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:11.193489       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:12.235314       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:13.278542       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:13.989062       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144762115", FieldPath:""}): type: 'Warning' reason: 'WaitingMinIOIsHealthy' Waiting for MinIO to be ready: %!s(<nil>)
I0822 07:12:14.327557       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:15.372854       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:16.416350       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:17.463190       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:18.513950       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:19.061076       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144762211", FieldPath:""}): type: 'Warning' reason: 'WaitingMinIOIsHealthy' Waiting for MinIO to be ready: %!s(<nil>)
I0822 07:12:19.559816       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:20.604286       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:21.650320       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:22.694493       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:23.737728       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:24.134795       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144762297", FieldPath:""}): type: 'Warning' reason: 'WaitingMinIOIsHealthy' Waiting for MinIO to be ready: %!s(<nil>)
I0822 07:12:24.801100       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:25.853544       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:26.933422       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:27.983578       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:29.030070       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:29.211035       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144762399", FieldPath:""}): type: 'Warning' reason: 'WaitingMinIOIsHealthy' Waiting for MinIO to be ready: %!s(<nil>)
I0822 07:12:30.073877       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:31.119628       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:32.166078       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:33.208655       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:34.242599       1 status.go:89] Hit conflict issue, getting latest version of tenant
I0822 07:12:34.271857       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:34.283145       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144762489", FieldPath:""}): type: 'Warning' reason: 'WaitingMinIOIsHealthy' Waiting for MinIO to be ready: %!s(<nil>)
I0822 07:12:35.318628       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:36.367337       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:39.432785       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144762577", FieldPath:""}): type: 'Warning' reason: 'UsersCreationFailed' Users creation failed: The Access Key Id you provided does not exist in our records.
I0822 07:12:41.785686       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:44.569890       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144762671", FieldPath:""}): type: 'Warning' reason: 'UsersCreationFailed' Users creation failed: The Access Key Id you provided does not exist in our records.
I0822 07:12:49.709243       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144762748", FieldPath:""}): type: 'Warning' reason: 'UsersCreationFailed' Users creation failed: The Access Key Id you provided does not exist in our records.
I0822 07:12:50.340084       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:54.701937       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144762827", FieldPath:""}): type: 'Warning' reason: 'UsersCreationFailed' Users creation failed: The Access Key Id you provided does not exist in our records.
I0822 07:12:59.804930       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144762909", FieldPath:""}): type: 'Warning' reason: 'UsersCreationFailed' Users creation failed: The Access Key Id you provided does not exist in our records.
I0822 07:13:04.833746       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144762986", FieldPath:""}): type: 'Warning' reason: 'UsersCreationFailed' Users creation failed: The Access Key Id you provided does not exist in our records.

Your Environment

  • Version used (minio-operator): 6.0.2
  • Environment name and version (e.g. kubernetes v1.17.2): v1.29.6-gke.1254000
@ramondeklein
Copy link
Contributor

ramondeklein commented Aug 22, 2024 •
edited
Loading

Please provide a valid values.yaml file, because the one that you submitted isn't valid:

  • Cannot set secrets: like this, probably meant secrets: ~ instead.
  • Pools should have a name.

I first applied the following namespace/secret that:

  • Creates the namespace my-minio
  • Created a MinIO configuration using minio / minio123
  • Provides access key/secret testtest / testtest for the user.
apiVersion: v1
kind: Namespace
metadata:
  name: my-minio
---
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: minio-env-configuration
  namespace: my-minio
data:
  config.env: ZXhwb3J0IE1JTklPX1NUT1JBR0VfQ0xBU1NfU1RBTkRBUkQ9IkVDOjIiCmV4cG9ydCBNSU5JT19CUk9XU0VSPSJvbiIKZXhwb3J0IE1JTklPX1JPT1RfVVNFUj0ibWluaW8iCmV4cG9ydCBNSU5JT19ST09UX1BBU1NXT1JEPSJtaW5pbzEyMyIK
---
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: minio-user-dev
  namespace: my-minio
data:
  CONSOLE_ACCESS_KEY: dGVzdHRlc3QK
  CONSOLE_SECRET_KEY: dGVzdHRlc3QK

Then ran helm install --namespace my-minio --create-namespace my-minio -f tenant.yaml minio-operator/tenant with the following tenant.yaml:

secrets: ~
tenant:
  name: minio-test
  image:
    repository: quay.io/minio/minio
    pullPolicy: IfNotPresent
  configuration:
    name: minio-env-configuration
  configSecret:
    name: minio-env-configuration
    existingSecret: true
  pools:
    - servers: 4
      name: pool-0
  metrics:
    enabled: false
  buckets:
    - name: bucket1
    - name: bucket2
  users:
    - name: "minio-user-dev"

It creates the tenant, buckets and user.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
community triage
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@aqeelat @ramondeklein