[CSIT-1962] 3n-icx, 3na-spr: Udpquicscale tests sometimes fail with various symptoms

[vvalderrv]

Description

While this has to point to a VPP bug, that bug is probably older.

The behavior was uncovered when CSIT converted the suite to do RX on both workers (on sibling lcores of the same physical CPU code).

Also, that [0] change has decreased the performance, but that performance was already invisible in trending due to CSIT-1935.

Among the symptoms, I have seen client not responding [1], server (deferred) failure [2], and even VPP crashes. Usually a crash does not create a core, but once [3] it did (an assert in quicly, but looks time-related, so no direct bug).

[0] 40977: fix(hoststack): Do not limit number of RX queues | https://gerrit.fd.io/r/c/csit/+/40977

[1] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2406-3n-icx/98/log.html.gz#s1-s1-s1-s1-s5-t1-k2-k4-k14

[2] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2406-3n-icx/102/log.html.gz#s1-s1-s1-s1-s13-t1-k2-k4-k18

[3] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2406-3n-icx/110/log.html.gz#s1-s1-s1-s1-s7-t1-k3-k4-k1

Assignee

Vratko Polak

Reporter

Vratko Polak

Comments

• vrpolak (Fri, 15 Nov 2024 12:30:55 +0000): Still present [4] in rls2410.

[4] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2410-3na-spr/44/log.html.gz#s1-s1-s1-s1-s5-t1-k2-k4-k14

Original issue: https://jira.fd.io/browse/CSIT-1962

[vvalderrv]

Still present [4] in rls2410.

[4] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2410-3na-spr/44/log.html.gz#s1-s1-s1-s1-s5-t1-k2-k4-k14