upgrade 6.0.28 to 6.0.40 results in jquery errors

[rdxmb]

Image and OTRS versions

Upgrading a "fresh" container running

image: "juanluisbaptiste/otrs:6.0.28" to image: "juanluisbaptiste/otrs:6.0.40"

Kubernetes Manifest (can be read similar to the docker-compose.yml) ;)

---
kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: app
  labels:
    app.kubernetes.io/name: app
spec:
  serviceName: app-headless
  replicas: 1
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app.kubernetes.io/name: app
  template:
    metadata:
      labels:
        app.kubernetes.io/name: app
    spec:
      containers:
        - name: app
          image: "juanluisbaptiste/otrs:6.0.40"
          resources:
            limits:
              memory: 5Gi
          env:
           - name: OTRS_DB_NAME
             value: otrs
           - name: OTRS_DB_HOST
             value: mariadb
           - name: OTRS_DB_USER
             value: otrs
           - name: OTRS_DB_PASSWORD
             value: otrs
           - name: MYSQL_ROOT_USER
             value: root
           - name: MYSQL_ROOT_PASSWORD
             value: secret123
           - name: OTRS_LANGUAGE
             value: "de"
           - name: OTRS_TIMEZONE
             value: "UTC"
           - name: OTRS_ARTICLE_STORAGE_TYPE
             value: ArticleStorageFS
           - name: OTRS_SET_PERMISSIONS
             value: "skip-article-dir"
           - name: OTRS_NUMBER_GENERATOR
             value: "AutoIncrement"
           - name: OTRS_SENDMAIL_MODULE
             value: "SMTP"
           - name: OTRS_SMTP_SERVER
             value: "catcher.mailcatcher.svc.cluster.local"
           - name: OTRS_SMTP_PORT
             value: "25"
           - name: OTRS_BACKUP_ROTATION
             value: "2"
           - name: OTRS_BACKUP_TIME
             value: "20 20 * * *"
           - name: OTRS_DEBUG
             value: "yes"
           - name: TZ
             value: "Etc/UTC"
          volumeMounts:
           - name: otrs-storage
             mountPath: /opt/otrs/Kernel
             subPath: Kernel
           - name: otrs-storage
             mountPath: /opt/otrs/var/httpd/htdocs/skins
             subPath: skins
           - name: otrs-storage
             mountPath: /opt/otrs/addons
             subPath: addons
           - name: otrs-storage
             mountPath: /opt/otrs/db_upgrade
             subPath: db_upgrade
           - name: otrs-storage
             mountPath: /opt/otrs/var/article
             subPath: article
           - name: backups
             mountPath: /var/otrs/backups
             readOnly: false
           - name: backup-and-restore 
             mountPath: /otrs_backup.sh
             subPath: otrs_backup.sh
           - name: backup-and-restore
             mountPath: /otrs_restore_system_data.sh
             subPath: otrs_restore_system_data.sh
           - name: backup-and-restore
             mountPath: /otrs_restore_articles.sh
             subPath: otrs_restore_articles.sh
           - name: backup-and-restore
             mountPath: /otrs_restore_mysql.sh
             subPath: otrs_restore_mysql.sh
      volumes:
        - name: backup-and-restore
          configMap:
            name: backup-and-restore
            defaultMode: 0770
        - name: backups
          hostPath:
            path: "/mnt/nfs/backups/applications/production/k1/znuny/app-triggered"
            type: DirectoryOrCreate
  volumeClaimTemplates:
    - metadata: 
        name: otrs-storage
        labels:
          reddoxx/storageclass: "iscsi-pv-provisioner"
      spec:
        storageClassName: "iscsi-pv-provisioner"
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 100Gi
---
# Source: otrs/templates/statefulset.yaml
kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: mariadb
  labels:
    app.kubernetes.io/name: mariadb
spec:
  serviceName: mariadb-headless
  replicas: 1
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app.kubernetes.io/name: mariadb
  template:
    metadata:
      labels:
        app.kubernetes.io/name: mariadb
    spec:
      priorityClassName: "znuny-production"
      containers:
        - name: mariadb
          image: "juanluisbaptiste/otrs-mariadb:latest"
          resources:
            limits:
              memory: 5Gi
          env:
           - name: "MYSQL_ROOT_PASSWORD"
             value: secret123
           - name: TZ
             value: Etc/UTC
          volumeMounts:
           - name: mariadb-tuning
             mountPath: /etc/mysql/conf.d/rdx-tuning.cnf
             subPath: rdx-tuning.cnf
           - name: mariadb-var-lib-mysql 
             mountPath: /var/lib/mysql
             subPath: var-lib-mysql
          livenessProbe:
            tcpSocket:
              port: 3306
            initialDelaySeconds: 120
            periodSeconds: 120
      volumes:
        - name: mariadb-tuning
          configMap:
            name: mariadb-tuning
  volumeClaimTemplates:
    - metadata:
        name: mariadb-var-lib-mysql
        labels:
          reddoxx/storageclass: "iscsi-pv-provisioner"
      spec:
        storageClassName: "iscsi-pv-provisioner"
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 100Gi
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: mariadb-tuning
data:
  rdx-tuning.cnf: |
    [mysqld]
    innodb_buffer_pool_size=2G

Describe the issue

After upgrading, the login-page shows jquery errors.

Expected behavior

no problems when upgrading the minor version

Screenshots

[rdxmb]

Here are the logs of the container after adding

    $Self->{"Loader::Enabled::JS"} = 0;

k -n znuny-production logs app-0 | grep 404
10.244.7.142 - - [27/May/2022:09:42:14 +0000] "GET /favicon.ico HTTP/1.1" 404 209 "https://znuny.rdx-dev.com/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:100.0) Gecko/20100101 Firefox/100.0"
10.244.7.142 - - [27/May/2022:09:42:15 +0000] "GET /otrs-web/js/thirdparty/jquery-ui-1.12.1/jquery-ui.js HTTP/1.1" 404 250 "https://znuny.rdx-dev.com/otrs/index.pl" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:100.0) Gecko/20100101 Firefox/100.0"
10.244.7.142 - - [27/May/2022:09:42:15 +0000] "GET /otrs-web/js/thirdparty/nunjucks-3.0.1/nunjucks.js HTTP/1.1" 404 247 "https://znuny.rdx-dev.com/otrs/index.pl" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:100.0) Gecko/20100101 Firefox/100.0"
10.244.7.142 - - [27/May/2022:09:42:15 +0000] "GET /otrs-web/js/thirdparty/jquery-ui-1.12.1/jquery-ui.js HTTP/1.1" 404 250 "https://znuny.rdx-dev.com/otrs/index.pl" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:100.0) Gecko/20100101 Firefox/100.0"
10.244.7.142 - - [27/May/2022:09:42:15 +0000] "GET /otrs-web/js/thirdparty/nunjucks-3.0.1/nunjucks.js HTTP/1.1" 404 247 "https://znuny.rdx-dev.com/otrs/index.pl" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:100.0) Gecko/20100101 Firefox/100.0"

paths inside the container:

# find . | grep thirdparty | grep jquery
./var/httpd/htdocs/skins/Customer/default/css/thirdparty/ui-theme/jquery-ui.css
./var/httpd/htdocs/skins/Agent/default/css/thirdparty/ui-theme/jquery-ui.css
./var/httpd/htdocs/js/thirdparty/jquery-migrate-3.0.0
./var/httpd/htdocs/js/thirdparty/jquery-migrate-3.0.0/jquery-migrate.js
./var/httpd/htdocs/js/thirdparty/jquery-browser-detection
./var/httpd/htdocs/js/thirdparty/jquery-browser-detection/jquery-browser-detection.js
./var/httpd/htdocs/js/thirdparty/jquery-3.5.1
./var/httpd/htdocs/js/thirdparty/jquery-3.5.1/jquery.js
./var/httpd/htdocs/js/thirdparty/jquery-ui-touch-punch-0.2.3
./var/httpd/htdocs/js/thirdparty/jquery-ui-touch-punch-0.2.3/jquery.ui.touch-punch.js
./var/httpd/htdocs/js/thirdparty/jquery-ui-1.13.1
./var/httpd/htdocs/js/thirdparty/jquery-ui-1.13.1/jquery-ui.js
./var/httpd/htdocs/js/thirdparty/jquery-validate-1.16.0
./var/httpd/htdocs/js/thirdparty/jquery-validate-1.16.0/jquery.validate.js
./var/httpd/htdocs/js/thirdparty/jquery-pubsub
./var/httpd/htdocs/js/thirdparty/jquery-pubsub/pubsub.js
./var/httpd/htdocs/js/thirdparty/ckeditor-4.17.1/samples/old/jquery.html
./var/httpd/htdocs/js/thirdparty/ckeditor-4.17.1/adapters/jquery.js
./var/httpd/htdocs/js/thirdparty/ckeditor-4.17.1/plugins/codesnippetgeshi/dev/geshi/geshi/jquery.php
./var/httpd/htdocs/js/thirdparty/jquery-tablesorter-2.28.14
./var/httpd/htdocs/js/thirdparty/jquery-tablesorter-2.28.14/jquery.tablesorter.js
./var/httpd/htdocs/js/thirdparty/jquery-jstree-3.3.7
./var/httpd/htdocs/js/thirdparty/jquery-jstree-3.3.7/jquery.jstree.js

[rdxmb]

Upgrade image: "juanluisbaptiste/otrs:6.0.28" -> "juanluisbaptiste/otrs:6.0.37" seems to be ok

[juanluisbaptiste]

Probably the same issue #118

[rdxmb]

I had read that before, but with

> Upgrade is successful only if I manually create a file "current_version" in volume mounted to /opt/otrs/Kernel/ and put information about current version in it (6.0.37). After upgrade file contains new version (6.0.38).

in #118 (comment) I thought that I have to create the file in ${OTRS_ROOT}/current_version . However, here it already exists ...

With 96fe692 it seems that it has to be copied to ${OTRS_CONFIG_DIR}/current_version

I will try that.

Thank you very much!

This seems a bit confusing to me. I've just realized that ${OTRS_CONFIG_DIR} is /opt/otrs/Kernel and not /opt/otrs/Kernel/Config/

# ls -la /opt/otrs/Kernel/current_version 
-rw-rw---- 1 otrs apache 7 May 27 10:59 /opt/otrs/Kernel/current_version
# cat /opt/otrs/Kernel/current_version
6.0.37

# grep current_version_file /functions.sh 
  local current_version_file="${OTRS_CONFIG_DIR}/current_version"
  cat $current_version_file
  if [ -e ${current_version_file} ] && [ ${OTRS_UPGRADE} != "yes" ]; then
    current_version=$(cat ${current_version_file})
      echo ${new_version} > ${current_version_file}

So upgrade from 6.0.37 to 6.0.40 should be possible now. I will try.

[rdxmb]

Update:

Upgrade from 6.0.37 to 6.0.40 (with the files existing like above) does also result in the jquery-errors.

[rdxmb]

Upgrade from juanluisbaptiste/otrs:6.0.37 to juanluisbaptiste/znuny:latestalso results in the jquery-errors. So 96fe692 does not seem to fix this.

[juanluisbaptiste]

I was able to reproduce the error when upgrading from 6.0.28 to 6.0.40, but if I do first upgrade from 6.0.28 to 6.0.33 which was the first Znuny version, and then to 6.0.40 I do not get the jquery error. I also found an error in the fix for issue #118 which would overwrite the value of /opt/otrs/Kernel/current_fileafter a minor upgrade, which would make the minor version upgrade procedure to run each time the container was started.

I will push a fix shortly.

[juanluisbaptiste]

@rdxmb were you able to test?

[rdxmb]

@juanluisbaptiste

Can you tell me what you want me to test exactly? Actually we are running juanluisbaptiste/otrs:6.0.37 in production.

Maybe I've copied the files manually, I do not remember ...

[root@app-0 otrs]# pwd
/opt/otrs
[root@app-0 otrs]# cat current_version 
6.0.37
[root@app-0 otrs]# cat Kernel/current_version 
6.0.37

[juanluisbaptiste]

@rdxmb , well, test the bug that you reported on this issue, and see if it is present for you.

[rdxmb]

I just pushed an updated 6.0.40 image.

from #118 (comment)

That was the information I was missing...

Ok, I am going to test that tag next week, probably on Monday.

[vadonka]

Fresh deploy 6.0.28 (otrs version) works
Upgrade to 6.0.33 (znuny version) works
Upgrade to 6.0.39 (latest znuny version what not break with the jquery error) works
Upgrade to 6.0.40 or above jquery error
Fresh deploy 6.0.40 or above missing perl modules error and apache not start

The jquery issue still not fixed and the old fresh install not working issue (#97) reintroduced.

[juanluisbaptiste]

@vadonka thanks for the debugging, I will point latest to 6.0.39 for now and also delete the images >= 6.0.40, while this issue is resolved.

[vadonka]

@vadonka thanks for the debugging, I will point latest to 6.0.39 for now and also delete the images >= 6.0.40, while this issue is resolved.

You welcome!
What did you change after the 6.0.39? I can look your commits maybe i noticed whats wrong. Im a dev by myself too.

[juanluisbaptiste]

The issue was introduced when working on this fix for minor updates. In a few words, what is happening is that because of that fix, the contents of the /Kernel directory is not being copied to /opt/otrs/Kernel on first run. Check the diffs of the commits for an idea of where to work.

[Vertganti]

We were stuck on this issue for a few days now and finally found the cause and a solution. The problem is that the Config subdirectory of the Kernel directory contains configuration about the used JS modules, including the path of the modules. Since most module names include the version (e.g. jquery-ui-1.12.1) the configuration has to be updated when a new version of the module is used.

The check_host_mount_dir function in functions.sh will only copy the default configuration if the config dir is empty, i.e. when you do a fresh deploy. Since the installed JS module versions are specified correctly in the Config subdirectory files in this case the jquery errors do not occur.

If the config dir already exists, nothing will be copied. The old configuration files in the Config subdirectory will still list the same JS module versions as before. If you upgrade to a version with updated JS modules, e.g. Znuny 6.0.40, you will get the jquery errors because the module paths point to the old versions which do not exist. Additionally you might miss Znuny modules that were added in newer versions, such as here or here.

As a solution we modified the check_host_mount_dir function to always copy the entire default configuration of the running Znuny version. If the configuration directory already exists we keep the Config.pm file.

Original function:

function check_host_mount_dir() {
  #Copy the configuration from /Kernel (put there by the Dockerfile) to $OTRS_CONFIG_DIR
  #to be able to use host-mounted volumes. copy only if ${OTRS_CONFIG_DIR} doesn't exist
  if ([ "$(ls -A ${OTRS_CONFIG_MOUNT_DIR})" ] && [ ! "$(ls -A ${OTRS_CONFIG_DIR})" ]) || [ "${OTRS_UPGRADE}" == "yes" ] || [ ${_MINOR_VERSION_UPGRADE} == true ];
  then
    print_info "Found empty \e[${OTRS_ASCII_COLOR_BLUE}m${OTRS_CONFIG_DIR}\e[0m, copying default configuration to it..."
    mkdir -p ${OTRS_CONFIG_DIR}
    cp -rfp ${OTRS_CONFIG_MOUNT_DIR}/* ${OTRS_CONFIG_DIR}
    if [ $? -eq 0 ];
      then
        print_info "Done."
      else
        print_error "Can't move OTRS configuration directory to ${OTRS_CONFIG_DIR}" && exit 1
    fi
  else
    print_info "Found existing configuration directory, Ok."
  fi
}

Modified function (including the fix mentioned here):

function check_host_mount_dir() {
  #Copy the configuration from /Kernel (put there by the Dockerfile) to $OTRS_CONFIG_DIR
  #to be able to use host-mounted volumes. keep Config.pm if ${OTRS_CONFIG_DIR} exists
  if ([ "$(ls -A ${OTRS_CONFIG_MOUNT_DIR})" ] && ([ ! "$(ls -A ${OTRS_CONFIG_DIR})" ] || [ "$(ls -A ${OTRS_CONFIG_DIR})" == "current_version" ])) || [ "${OTRS_UPGRADE}" == "yes" ] || [ ${_MINOR_VERSION_UPGRADE} == true ];
  then
    print_info "Found empty \e[${OTRS_ASCII_COLOR_BLUE}m${OTRS_CONFIG_DIR}\e[0m, copying full default configuration to it..."
  else
    print_info "Found existing configuration directory, copy default configuration but keep existing Config.pm..."
    cp -rfp ${OTRS_CONFIG_DIR}/Config.pm ${OTRS_CONFIG_MOUNT_DIR}/Config.pm
  fi
  mkdir -p ${OTRS_CONFIG_DIR}
  cp -rfp ${OTRS_CONFIG_MOUNT_DIR}/* ${OTRS_CONFIG_DIR}
  if [ $? -eq 0 ];
    then
      print_info "Done."
    else
      print_error "Can't move OTRS configuration directory to ${OTRS_CONFIG_DIR}" && exit 1
  fi
}

Since we only have modified the configuration in the Config.pm file this works well for our setup. If you have modified other configuration files you will have to adapt the approach to consider your modified file(s). It would probably be best to use the config files of the updated Znuny version as a base and add your custom configuration instead of using the old files, as there might have been additions to the default configuration.