Template expansion may fail in argo 3.5.10 and 3.5.11

[mist714]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

Template expansion may fail in argo 3.5.10 and 3.5.11．

It doesn't fail every time, but seems to occur when there is a certain amount of load.
Also, we manage multiple clusters, and even though they are the same version, there are some clusters where the problem occurs and some where it doesn't.

This problem does not exist in v3.4.6.
I discovered a problem when upgrading from 3.4.6 to 3.5.10, and tried 3.5.11, but the same error occurred.

Version(s)

v3.5.10,v3.5.11

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: sample-
spec:
  entrypoint: entrypoint
  templates:
  - name: entrypoint
    steps:
    - - name: generate-param
        template: generate-param
    - - name: with-param
        template: with-param
        arguments:
          parameters:
          - name: item
            value: '{{item}}'
        withParam: '{{steps.generate-param.outputs.result}}'
  - name: generate-param
    container:
      image: busybox
      command: [echo, '["foo", "bar", "baz"]']
  - name: with-param
    inputs:
      parameters:
      - name: item
    container:
      image: busybox
      command: [echo, '{{inputs.parameters.item}}']

While this workflow works well in many cases, there are some cases where it can fail:

NAME           STATUS      AGE   DURATION   PRIORITY   MESSAGE
sample-twvvt   Failed      38s   20s        0          withParam value could not be parsed as a JSON list: {{steps.generate-param.outputs.result}}: invalid character '{' looking for beginning of object key string
sample-ft2vd   Failed      39s   10s        0          withParam value could not be parsed as a JSON list: {{steps.generate-param.outputs.result}}: invalid character '{' looking for beginning of object key string
sample-hsbz8   Failed      40s   10s        0          withParam value could not be parsed as a JSON list: {{steps.generate-param.outputs.result}}: invalid character '{' looking for beginning of object key string

Logs from the workflow controller

Nothing showed up in the controller log.

Logs from in your workflow's wait container

Since the template expansion failed, init containers are not starting either.

[Joibel]

Can you ensure you've got pod RBAC correct: https://argo-workflows.readthedocs.io/en/release-3.5/workflow-rbac/

Can you provide logs from the wait container of the generate-param step?

Are you running the 3.5.11 executor in your created pods?