Test failures while running test-cmd-debug test suite on ppc64le

[sarveshtamba]

I have made changes to the scripts/openshiftci-presubmit-all-tests.sh and scripts/configure-installer-tests-cluster.sh for Power on the lines of the Z changes.
Also done the setup required like creating the secret to pull redhat registry images. Currently I am trying to run the scripts/openshiftci-presubmit-all-tests.s and seeing the below issues for test-cmd-debug test suite:

Summarizing 5 Failures:

[Fail] odo debug command tests odo debug on a nodejs:latest component [It] should expect a ws connection when tried to connect on default debug port locally
/usr/local/go/src/github.com/openshift/odo/tests/helper/helper_run.go:34

[Fail] odo debug command tests odo debug on a nodejs:latest component [It] check that machine output debug information works
/usr/local/go/src/github.com/openshift/odo/tests/helper/helper_run.go:34

[Fail] odo debug command tests odo debug on a nodejs:latest component [It] should expect a ws connection when tried to connect on different debug port locally and remotely
/usr/local/go/src/github.com/openshift/odo/tests/helper/helper_run.go:34

[Fail] odo debug command tests odo debug info should work on a odo component [It] should start a debug session and run debug info on a closed debug session
/usr/local/go/src/github.com/openshift/odo/tests/helper/helper_run.go:34

[Fail] odo debug command tests odo debug info should work on a odo component [It] should start a debug session and run debug info on a running debug session
/usr/local/go/src/github.com/openshift/odo/tests/helper/helper_run.go:34

Ran 5 of 177 Specs in 66.287 seconds
FAIL! -- 0 Passed | 5 Failed | 0 Pending | 172 Skipped


Ginkgo ran 1 suite in 1m11.122493463s
Test Suite Failed
make: *** [Makefile:284: test-cmd-debug] Error 1

Attaching detailed test run and some post failure investigation steps -
odo_test-cmd-debug_failures_ppc64le.txt

[sarveshtamba]

/area system-pz

[sarveshtamba]

cc:- @amitkrout ^^

[kadel]

/kind test

[amitkrout]

Same failures are reported in #3491 and #3497.
/close

[openshift-ci-robot]

@amitkrout: Closing this issue.

In response to this:

Same failures are reported in #3491 and #3497.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sarveshtamba]

@amitkrout Facing the following issues in test-cmd-debug test suite:-

Summarizing 4 Failures:

[Fail] odo debug command tests odo debug info should work on a odo component [It] should start a debug session and run debug info on a closed debug session
/usr/local/go/src/github.com/openshift/odo/tests/helper/helper_http.go:50

[Fail] odo debug command tests odo debug on a nodejs:latest component [It] should expect a ws connection when tried to connect on default debug port locally
/usr/local/go/src/github.com/openshift/odo/tests/helper/helper_http.go:50

[Fail] odo debug command tests odo debug info should work on a odo component [It] should start a debug session and run debug info on a running debug session
/usr/local/go/src/github.com/openshift/odo/tests/helper/helper_http.go:50

[Fail] odo debug command tests odo debug on a nodejs:latest component [It] should expect a ws connection when tried to connect on different debug port locally and remotely
/usr/local/go/src/github.com/openshift/odo/tests/helper/helper_http.go:50

Ran 5 of 184 Specs in 365.520 seconds
FAIL! -- 1 Passed | 4 Failed | 0 Pending | 179 Skipped


Ginkgo ran 1 suite in 6m10.374469509s
Test Suite Failed
make: *** [Makefile:332: test-cmd-debug] Error 1

Attaching the detailed spec failure log -
odo_test-cmd-debug_failures_ppc64le_1.txt

[sarveshtamba]

/reopen

[openshift-ci-robot]

@sarveshtamba: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[prietyc123]

Failure here is with port-forward

Running odo with args [odo debug port-forward --local-port 46097 --context /tmp/094321299]
error while requesting: Get http://localhost:46097: dial tcp [::1]:46097: connect: connection refused
[odo] I0820 02:30:04.665636   28967 preference.go:209] The path for preference file is /tmp/094321299/config.yaml
[odo] I0820 02:30:04.665672   28967 util.go:734] HTTPGetRequest: https://raw.githubusercontent.com/openshift/odo/master/build/VERSION
[odo] I0820 02:30:04.665829   28967 util.go:755] Response will be cached in /tmp/odohttpcache for 1h0m0s
[odo] I0820 02:30:04.665979   28967 util.go:768] Cached response used.
[odo] Started port forwarding at ports - 46097:5858
try 1 of 12
error while requesting: Get http://localhost:46097: EOF
[odo] E0820 02:30:09.914754   28967 portforward.go:400] an error occurred forwarding 46097 -> 5858: error forwarding port 5858 to pod 404f5d2b14e482084843e7956674e0a470022426fbf2fcf619b0008433239aaf, uid : exit status 1: 2020/08/20 06:30:09 socat[3081252] E connect(5, AF=2 127.0.0.1:5858, 16): Connection refused
try 2 of 12
[...]
[odo] I0820 02:31:04.471938   28967 odo.go:72] Could not get the latest release information in time. Never mind, exiting gracefully :)
Last output from http://localhost:46097:
Deleting project: ioqtwquywm
Running odo with args [odo project delete ioqtwquywm -f]
[odo] I0820 02:31:05.785982   29021 util.go:734] HTTPGetRequest: https://raw.githubusercontent.com/openshift/odo/master/build/VERSION
[odo] I0820 02:31:05.786156   29021 util.go:755] Response will be cached in /tmp/odohttpcache for 1h0m0s
[odo] I0820 02:31:05.977446   29021 util.go:768] Cached response used.
[odo]  ✓  Deleted project : ioqtwquywm
[odo]  ⚠  Warning! Projects are deleted from the cluster asynchronously. Odo does its best to delete the project. Due to multi-tenant clusters, the project may still exist on a different node.
[odo] I0820 02:31:06.127310   29021 odo.go:72] Could not get the latest release information in time. Never mind, exiting gracefully :)
Deleting dir: /tmp/094321299


• Failure [139.593 seconds]
odo debug command tests
/usr/local/go/src/github.com/openshift/odo/tests/integration/cmd_debug_test.go:19
  odo debug info should work on a odo component
  /usr/local/go/src/github.com/openshift/odo/tests/integration/cmd_debug_test.go:107
    should start a debug session and run debug info on a closed debug session [It]
    /usr/local/go/src/github.com/openshift/odo/tests/integration/cmd_debug_test.go:131

    Failed after 12 retries. Content in http://localhost:46097 doesn't include 'WebSockets request was expected'.

@kadel Not sure this is happening due to the wrong port-forward that you mentioned here - #3840 . Can you please look into this.

[amitkrout]

[odo] Started port forwarding at ports - 46097:5858
try 1 of 12
error while requesting: Get http://localhost:46097: EOF
[odo] E0820 02:30:09.914754 28967 portforward.go:400] an error occurred forwarding 46097 -> 5858: error forwarding port 5858 to pod 404f5d2b14e482084843e7956674e0a470022426fbf2fcf619b0008433239aaf, uid : exit status 1: 2020/08/20 06:30:09 socat[3081252] E connect(5, AF=2 127.0.0.1:5858, 16): Connection refused
try 2 of 12
[...]

[odo] Started port forwarding at ports - 46097:5858
try 1 of 12
error while requesting: Get http://localhost:46097: EOF
[odo] E0820 02:30:09.914754   28967 portforward.go:400] an error occurred forwarding 46097 -> 5858: error forwarding
 port 5858 to pod 404f5d2b14e482084843e7956674e0a470022426fbf2fcf619b0008433239aaf, uid : exit status 1: 
2020/08/20 06:30:09 socat[3081252] E connect(5, AF=2 127.0.0.1:5858, 16): Connection refused
try 2 of 12
[...]

@prietyc123 Port forward is happening in the right port but the connection has been refused while requesting for http://localhost:46097. I have never seen this in our prow CI. @sarveshtamba what is the frequency of this issue you are hitting ?

[sarveshtamba]

[odo] Started port forwarding at ports - 46097:5858
try 1 of 12
error while requesting: Get http://localhost:46097: EOF
[odo] E0820 02:30:09.914754 28967 portforward.go:400] an error occurred forwarding 46097 -> 5858: error forwarding port 5858 to pod 404f5d2b14e482084843e7956674e0a470022426fbf2fcf619b0008433239aaf, uid : exit status 1: 2020/08/20 06:30:09 socat[3081252] E connect(5, AF=2 127.0.0.1:5858, 16): Connection refused
try 2 of 12
[...]

[odo] Started port forwarding at ports - 46097:5858
try 1 of 12
error while requesting: Get http://localhost:46097: EOF
[odo] E0820 02:30:09.914754   28967 portforward.go:400] an error occurred forwarding 46097 -> 5858: error forwarding
 port 5858 to pod 404f5d2b14e482084843e7956674e0a470022426fbf2fcf619b0008433239aaf, uid : exit status 1: 
2020/08/20 06:30:09 socat[3081252] E connect(5, AF=2 127.0.0.1:5858, 16): Connection refused
try 2 of 12
[...]

@prietyc123 Port forward is happening in the right port but the connection has been refused while requesting for http://localhost:46097. I have never seen this in our prow CI. @sarveshtamba what is the frequency of this issue you are hitting ?

Seeing this frequently - almost every time I run this test suite.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[sarveshtamba]

/remove-lifecycle rotten

[kadel]

/remove-lifecycle rotten
/lifecycle frozen

[kadel]

@sarveshtamba do you still face the same issue with odo v3?