Mongo certficate DNS name patch

[qpdpQ]

What this PR does / why we need it:
enhancement for DNS name patch, previous PR only update the DNSnames in certificate icp-mongodb-client-cert is not enough, this certificaet isn't used directly by the MongoDB runtime. Mongodb init container will use a leaf certificate and that leaf certificate is signed by icp-mongodb-client-cert, we need to update that leaf certificate instead.

While preload_data.sh is executed, the script would:

1. Scale down ibm-mongodb-operator
2. Update the script in icp-mongodb-init CM's .data["on-start.sh"] field and add the additional DNS names
3. Trigger a rolling upgrade on the icp-mongodb StatefulSet, adding migrating: "true" to .spec.template.metadata.labels in mongodb StatefulSet
4. Scale up ibm-mongodb-operator

Which issue(s) this PR fixes:
Fixes # https://github.ibm.com/IBMPrivateCloud/roadmap/issues/65642

[qpdpQ]

The new generated mongo.pem in icp-mongodb-0 pod contains full DNSname

sh-4.4$ openssl x509 -in mongo.pem -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            7a:86:5e:d0:23:26:bb:4d:69:c9:7a:10:db:0c:49:5e:d9:65:7e:c4
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = mongodb
        Validity
            Not Before: Jan 27 19:49:26 2025 GMT
            Not After : Jan 25 19:49:26 2035 GMT
        Subject: CN = icp-mongodb-0
...
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            X509v3 Key Usage: 
                Digital Signature, Non Repudiation, Key Encipherment
            X509v3 Subject Alternative Name: 
                DNS:icp-mongodb, DNS:icp-mongodb-0, DNS:icp-mongodb-0.icp-mongodb.ltsr-data.svc.cluster.local, DNS:localhost, DNS:127.0.0.1, DNS:mongodb.ibm-common-services.svc.cluster.local <----- this one
...

[qpdpQ]

Full test log

allen@Allens-MBP ibm-common-service-operator % ./preload_data.sh --rerun --original-cs-ns ibm-common-services --services-ns ltsr-data
All arguments passed into the script: --rerun --original-cs-ns ibm-common-services --services-ns ltsr-data
[✔] oc command available
[✔] yq command available
[✔] oc command logged in as kube:admin
[INFO] Checking cert manager readiness.
issuer.cert-manager.io/test-issuer created
certificate.cert-manager.io/test-certificate created
certificate.cert-manager.io "test-certificate" deleted
issuer.cert-manager.io "test-issuer" deleted
[✔] Cert manager is ready, preload can proceed.
[INFO] Rerun specified...
# Deleting the stand up mongodb statefulset in ltsr-data
-----------------------------------------------------------------------
statefulset.apps "icp-mongodb" deleted
service "icp-mongodb" deleted
issuer.cert-manager.io "god-issuer" deleted
configmap "ibm-cpp-config" deleted
certificate.cert-manager.io "icp-mongodb-client-cert" deleted
configmap "icp-mongodb" deleted
configmap "icp-mongodb-init" deleted
configmap "icp-mongodb-install" deleted
secret "icp-mongodb-keyfile" deleted
secret "icp-mongodb-metrics" deleted
serviceaccount "ibm-mongodb-operand" deleted
service "mongodb" deleted
certificate.cert-manager.io "mongodb-root-ca-cert" deleted
issuer.cert-manager.io "mongodb-root-ca-issuer" deleted
configmap "namespace-scope" deleted
[✗] Volume for pvc cs-mongodump not found in ltsr-data. It may have already been deleted.
[✔] MongoDB removed from services namespace ltsr-data
[INFO] Copying mongodb from namespace ibm-common-services to namespace ltsr-data
[INFO] Adding full DNS name into icp-mongodb-client-cert certificate in ibm-common-services
[INFO] Adding full DNS name into leaf certificate icp-mongodb runtime in ibm-common-services
deployment.apps/ibm-mongodb-operator scaled
statefulset.apps/icp-mongodb scaled
Warning: resource configmaps/icp-mongodb-init is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically.
configmap/icp-mongodb-init configured
statefulset.apps/icp-mongodb scaled
[INFO] Waiting for MongoDB to initialize
icp-mongodb-0   0/2   Init:1/2   0     12s
[INFO] Waiting for MongoDB to initialize
icp-mongodb-0   0/2   Init:1/2   0     26s
[INFO] Waiting for MongoDB to initialize
icp-mongodb-0   0/2   Init:1/2   0     40s
[INFO] Waiting for MongoDB to initialize
icp-mongodb-0   0/2   Init:1/2   0     54s
[INFO] Waiting for MongoDB to initialize
icp-mongodb-0   0/2   Init:1/2   0     69s
[INFO] Waiting for MongoDB to initialize
icp-mongodb-0   0/2   Running   0     83s
deployment.apps/ibm-mongodb-operator scaled
[✔] DNS name in namespace: ibm-common-services updated
# Cleaning up any previous copy operations...
-----------------------------------------------------------------------
job.batch "mongodb-backup" deleted
persistentvolume/pvc-896b7aa5-d05b-45b9-ac50-dd9ffbe1beee patched (no change)
persistentvolumeclaim "cs-mongodump" deleted
[✔] Previous run cleaned up.
# Deploying a temporary mongodb in ltsr-data
-----------------------------------------------------------------------
configmap/icp-mongodb-install created
configmap/icp-mongodb-init created
issuer.cert-manager.io/god-issuer created
configmap/ibm-cpp-config created
secret/icp-mongodb-admin unchanged
certificate.cert-manager.io/icp-mongodb-client-cert created
configmap/icp-mongodb created
secret/icp-mongodb-keyfile created
secret/icp-mongodb-metrics created
serviceaccount/ibm-mongodb-operand created
service/mongodb created
service/icp-mongodb created
certificate.cert-manager.io/mongodb-root-ca-cert created
issuer.cert-manager.io/mongodb-root-ca-issuer created
configmap/namespace-scope created
statefulset.apps/icp-mongodb created
[INFO] Waiting for MongoDB copy to initialize
icp-mongodb-0   0/1   Init:1/2   0     12s
[INFO] Waiting for MongoDB copy to initialize
icp-mongodb-0   0/1   Init:1/2   0     27s
[INFO] Waiting for MongoDB copy to initialize
icp-mongodb-0   0/1   Init:1/2   0     41s
[INFO] Waiting for MongoDB copy to initialize
icp-mongodb-0   0/1   Init:1/2   0     56s
[INFO] Waiting for MongoDB copy to initialize
icp-mongodb-0   0/1   Init:1/2   0     70s
[INFO] Waiting for MongoDB copy to initialize
icp-mongodb-0   1/1   Running   0     84s
[✔] Temporary Mongo copy deployed to namespace ltsr-data
# Creating a PVC for the MongoDB dump
-----------------------------------------------------------------------
Now using project "ibm-common-services" on server "https://api.allen.cp.fyre.ibm.com:6443".
persistentvolumeclaim/cs-mongodump created
[✔] MongoDB PVC ready
# Backing up MongoDB in namespace ibm-common-services
-----------------------------------------------------------------------
[INFO] Running Backup
job.batch/mongodb-backup created
mongodb-backup-xwn2m                                   0/1     ContainerCreating            0             2s
[INFO] Waiting for job pod mongodb-backup to complete
[✔] Job mongodb-backup completed in namespace ibm-common-services

[INFO] Saving mongodb-backup logs in _mongodb-backup.log
[INFO] Deleting job mongodb-backup
job.batch "mongodb-backup" deleted
[✔] Backup Complete
# Moving restored mongodb volume to ltsr-data
-----------------------------------------------------------------------
persistentvolume/pvc-51fb18a4-527f-4b51-bd30-4da36f5e234c patched
persistentvolumeclaim "cs-mongodump" deleted
persistentvolume/pvc-51fb18a4-527f-4b51-bd30-4da36f5e234c patched
persistentvolumeclaim/cs-mongodump created
[✔] Restored MongoDB volume moved to namespace ltsr-data
# Restoring MongoDB to copy in namespace ltsr-data
-----------------------------------------------------------------------
[INFO] Running Restore
job.batch/mongodb-restore created
[INFO] Waiting for job pod mongodb-restore to complete
[✔] Job mongodb-restore completed in namespace ltsr-data

[INFO] Saving mongodb-restore logs in _mongodb-restore.log
[INFO] Deleting job mongodb-restore
job.batch "mongodb-restore" deleted
[✔] Restore Complete
# Deleting the stand up mongodb statefulset in ltsr-data
-----------------------------------------------------------------------
statefulset.apps "icp-mongodb" deleted
service "icp-mongodb" deleted
issuer.cert-manager.io "god-issuer" deleted
configmap "ibm-cpp-config" deleted
certificate.cert-manager.io "icp-mongodb-client-cert" deleted
configmap "icp-mongodb" deleted
configmap "icp-mongodb-init" deleted
configmap "icp-mongodb-install" deleted
secret "icp-mongodb-keyfile" deleted
secret "icp-mongodb-metrics" deleted
serviceaccount "ibm-mongodb-operand" deleted
service "mongodb" deleted
certificate.cert-manager.io "mongodb-root-ca-cert" deleted
issuer.cert-manager.io "mongodb-root-ca-issuer" deleted
configmap "namespace-scope" deleted
persistentvolume/pvc-51fb18a4-527f-4b51-bd30-4da36f5e234c patched
persistentvolumeclaim "cs-mongodump" deleted
[✔] MongoDB removed from services namespace ltsr-data
configmap/mongodb-preload-endpoint unchanged
#  Copying secret platform-auth-idp-credentials from ibm-common-services to ltsr-data 
secret/platform-auth-idp-credentials unchanged
[✔] secret platform-auth-idp-credentials copied over to ltsr-data.
#  Copying secret platform-auth-ldaps-ca-cert from ibm-common-services to ltsr-data 
secret/platform-auth-ldaps-ca-cert unchanged
[✔] secret platform-auth-ldaps-ca-cert copied over to ltsr-data.
#  Copying secret platform-oidc-credentials from ibm-common-services to ltsr-data 
secret/platform-oidc-credentials unchanged
[✔] secret platform-oidc-credentials copied over to ltsr-data.
#  Copying secret oauth-client-secret from ibm-common-services to ltsr-data 
secret/oauth-client-secret unchanged
[✔] secret oauth-client-secret copied over to ltsr-data.
#  Copying configmap ibm-cpp-config from ibm-common-services to ltsr-data 
configmap/ibm-cpp-config created
[✔] configmap ibm-cpp-config copied over to ltsr-data.
#  Copying configmap common-web-ui-config from ibm-common-services to ltsr-data 
configmap/common-web-ui-config unchanged
[✔] configmap common-web-ui-config copied over to ltsr-data.
#  Copying configmap platform-auth-idp from ibm-common-services to ltsr-data 
configmap/platform-auth-idp unchanged
[✔] configmap platform-auth-idp copied over to ltsr-data.
#  Copying commonservice common-service from ibm-common-services to ltsr-data 
Deleting storageClass field from commonservice CR
commonservice.operator.ibm.com/preload-common-service-from-ibm-common-services configured
[✔] commonservice preload-common-service-from-ibm-common-services copied over to ltsr-data.
#  Copying secret icp-mongodb-client-cert from ibm-common-services to ltsr-data 
secret/icp-mongodb-client-cert unchanged
[✔] secret icp-mongodb-client-cert copied over to ltsr-data.
#  Copying secret mongodb-root-ca-cert from ibm-common-services to ltsr-data 
secret/mongodb-root-ca-cert unchanged
[✔] secret mongodb-root-ca-cert copied over to ltsr-data.
#  Copying secret icp-mongodb-admin from ibm-common-services to ltsr-data 
secret/icp-mongodb-admin configured
[✔] secret icp-mongodb-admin copied over to ltsr-data.

[bitscuit]

/lgtm

[ibm-ci-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bitscuit, qpdpQ

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [bitscuit,qpdpQ]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment