How to tune the TX performance?

[jxc98728]

I use this branch (DPDK 20.11) and my CPU is Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz.

For the FPGA design, my config.tcl in corundum is:

# Queue manager configuration
dict set params EVENT_QUEUE_OP_TABLE_SIZE "32"
dict set params TX_QUEUE_OP_TABLE_SIZE "32"
dict set params RX_QUEUE_OP_TABLE_SIZE "32"
dict set params TX_CPL_QUEUE_OP_TABLE_SIZE [dict get $params TX_QUEUE_OP_TABLE_SIZE]
dict set params RX_CPL_QUEUE_OP_TABLE_SIZE [dict get $params RX_QUEUE_OP_TABLE_SIZE]
dict set params EVENT_QUEUE_INDEX_WIDTH "6"
dict set params TX_QUEUE_INDEX_WIDTH "13"
dict set params RX_QUEUE_INDEX_WIDTH "8"
dict set params TX_CPL_QUEUE_INDEX_WIDTH [dict get $params TX_QUEUE_INDEX_WIDTH]
dict set params RX_CPL_QUEUE_INDEX_WIDTH [dict get $params RX_QUEUE_INDEX_WIDTH]
dict set params EVENT_QUEUE_PIPELINE "3"
dict set params TX_QUEUE_PIPELINE [expr 3+([dict get $params TX_QUEUE_INDEX_WIDTH] > 12 ? [dict get $params TX_QUEUE_INDEX_WIDTH]-12 : 0)]
dict set params RX_QUEUE_PIPELINE [expr 3+([dict get $params RX_QUEUE_INDEX_WIDTH] > 12 ? [dict get $params RX_QUEUE_INDEX_WIDTH]-12 : 0)]
dict set params TX_CPL_QUEUE_PIPELINE [dict get $params TX_QUEUE_PIPELINE]
dict set params RX_CPL_QUEUE_PIPELINE [dict get $params RX_QUEUE_PIPELINE]

# TX and RX engine configuration
dict set params TX_DESC_TABLE_SIZE "32"
dict set params RX_DESC_TABLE_SIZE "32"

# Scheduler configuration
dict set params TX_SCHEDULER_OP_TABLE_SIZE [dict get $params TX_DESC_TABLE_SIZE]
dict set params TX_SCHEDULER_PIPELINE [dict get $params TX_QUEUE_PIPELINE]
dict set params TDMA_INDEX_WIDTH "6"

# Interface configuration
dict set params PTP_TS_ENABLE "0"
dict set params TX_CPL_FIFO_DEPTH "32"
dict set params TX_CHECKSUM_ENABLE "1"
dict set params RX_RSS_ENABLE "1"
dict set params RX_HASH_ENABLE "1"
dict set params RX_CHECKSUM_ENABLE "1"
dict set params TX_FIFO_DEPTH "32768"
dict set params RX_FIFO_DEPTH "131072"
dict set params MAX_TX_SIZE "9214"
dict set params MAX_RX_SIZE "9214"
dict set params TX_RAM_SIZE "131072"
dict set params RX_RAM_SIZE "131072"

My U250 NIC is attached with a Mellanox CX6-Dx NIC. I use pktgen on CX6-Dx to generate traffic and use U250 corundum to receive traffic. I assign only one single core for Rx side and the MTU is 1500. The single-core Rx performance is around 93Gbps.

- Ports 0-1 of 2   <Main Page>  Copyright(c) <2010-2021>, Intel Corporation
  Flags:Port        : -------Sngl       :0
Link State          :       <UP-100000-FD>     ---Total Rate---
Pkts/s Rx           :            7,694,144            7,694,144
       Tx           :                    0                    0
MBits/s Rx/Tx       :             93,560/0             93,560/0
Pkts/s Rx Max       :            7,713,280            7,713,280
       Tx Max       :                    0                    0
Broadcast           :                    0
Multicast           :                    0
Sizes 64            :                    0
      65-127        :                    0
      128-255       :                    0
      256-511       :                    0
      512-1023      :                    0
      1024-1518     :          188,001,803
Runts/Jumbos        :                  0/0
ARP/ICMP Pkts       :                  0/0
Errors Rx/Tx        :                  0/0
Total Rx Pkts       :          183,882,923
      Tx Pkts       :                    0
      Rx/Tx MBs     :          2,236,016/0
Pattern Type        :              abcd...
Tx Count/% Rate     :        Forever /100%
Pkt Size/Tx Burst   :            64 /   32
TTL/Port Src/Dest   :       64/ 1234/ 5678
Pkt Type:VLAN ID    :      IPv4 / TCP:0001
802.1p CoS/DSCP/IPP :            0/  0/  0
VxLAN Flg/Grp/vid   :     0000/    0/    0
IP  Destination     :          192.168.1.1
    Source          :       192.168.0.1/24
MAC Destination     :    00:00:00:00:00:00
    Source          :    67:c6:69:73:51:ff
PCI Vendor/Addr     :    1234:1001/d9:00.0
-- Pktgen 21.03.1 (DPDK 20.11.10)  Powered by DPDK  (pid:623744) --------------

However, on the Tx size, the single-core is quite poor, which is only around 31Gbps

- Ports 0-1 of 2   <Main Page>  Copyright(c) <2010-2021>, Intel Corporation
  Flags:Port        : -------Sngl       :0
Link State          :       <UP-100000-FD>     ---Total Rate---
Pkts/s Rx           :                    0                    0
       Tx           :            2,584,000            2,584,000
MBits/s Rx/Tx       :             0/31,421             0/31,421
Pkts/s Rx Max       :                    0                    0
       Tx Max       :            2,610,336            2,610,336
Broadcast           :                    0
Multicast           :                    0
Sizes 64            :                    0
      65-127        :                    0
      128-255       :                    0
      256-511       :                    0
      512-1023      :                    0
      1024-1518     :                    0
Runts/Jumbos        :                  0/0
ARP/ICMP Pkts       :                  0/0
Errors Rx/Tx        :                  0/0
Total Rx Pkts       :                    0
      Tx Pkts       :           32,831,072
      Rx/Tx MBs     :            0/399,225
Pattern Type        :              abcd...
Tx Count/% Rate     :        Forever /100%
Pkt Size/Tx Burst   :          1500 /   32
TTL/Port Src/Dest   :       64/ 1234/ 5678
Pkt Type:VLAN ID    :      IPv4 / TCP:0001
802.1p CoS/DSCP/IPP :            0/  0/  0
VxLAN Flg/Grp/vid   :     0000/    0/    0
IP  Destination     :          192.168.1.1
    Source          :       192.168.0.1/24
MAC Destination     :    00:00:00:00:00:00
    Source          :    67:c6:69:73:51:ff
PCI Vendor/Addr     :    1234:1001/d9:00.0
-- Pktgen 21.03.1 (DPDK 20.11.10)  Powered by DPDK  (pid:624726) --------------

Is there any way or trick to tune or choose proper hardware design parameter for better Tx performance? Thx!