[CSIT-1960] 2n-zn2: AVF xxv710 sometimes loses one direction of traffic, mostly with geneve

[vvalderrv]

Description

This was previously mentioned as a secondary symptom of CSIT-1800. But that ticket is mostly about VPP crashes (not seen recently), while this symptom (NDRPDR failure but no crash) still can [0] happen, albeit rarely. Interestingly, all tests within the suite failed the same way.

No clear relation to number of tunnels is visible yet.

More investigation needed for proper VPP bug-report.

[0] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2406-2n-zn2/54/log.html.gz#s1-s1-s1-s3-s1-t1-k2-k9-k14

Assignee

Vratko Polak

Reporter

Vratko Polak

Comments

• vrpolak (Wed, 13 Nov 2024 13:27:04 +0000): > "l3 mac mismatch" errors leading to punt.

First time I see this symptom outside geneve (still on zen2). It happened [3] in ip4base test, so perhaps it is possible in all AVF xxv710 tests on zn2. Changing title.

[3] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2410-2n-zn2/37/log.html.gz#s1-s1-s1-s2-s10-t1-k3-k7-k1-k1-k1-k8-k14-k1-k1-k1-k1

• vrpolak (Wed, 13 Nov 2024 12:52:57 +0000): In rls2410, this happened [2] for 16Tun Geneve suite.

[2] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2410-2n-zn2/22/log.html.gz#s1-s1-s1-s3-s1-t1-k2-k9-k14

• vrpolak (Wed, 14 Aug 2024 12:04:47 +0000): Very rare. Last time seen in trending as MRR regression (also for 16tnl). Telemetry [1] shows "l3 mac mismatch" errors leading to punt.

[1] https://logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/955/log.html.gz#s1-s1-s1-s3-s1-t1-k2-k9-k9-k14-k1-k1-k1-k1

Original issue: https://jira.fd.io/browse/CSIT-1960

[vvalderrv]

> "l3 mac mismatch" errors leading to punt.

First time I see this symptom outside geneve (still on zen2). It happened [3] in ip4base test, so perhaps it is possible in all AVF xxv710 tests on zn2. Changing title.

[3] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2410-2n-zn2/37/log.html.gz#s1-s1-s1-s2-s10-t1-k3-k7-k1-k1-k1-k8-k14-k1-k1-k1-k1

[vvalderrv]

In rls2410, this happened [2] for 16Tun Geneve suite.

[2] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2410-2n-zn2/22/log.html.gz#s1-s1-s1-s3-s1-t1-k2-k9-k14

[vvalderrv]

Very rare. Last time seen in trending as MRR regression (also for 16tnl). Telemetry [1] shows "l3 mac mismatch" errors leading to punt.

[1] https://logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/955/log.html.gz#s1-s1-s1-s3-s1-t1-k2-k9-k9-k14-k1-k1-k1-k1