OpenPiton configuration of CVA6 cannot boot Linux on UltraScale+ FPGA

[MikeOpenHWGroup]

The cv64a6_imafddcv_sv39_config_pkg.sv configuration of the CVA6 (also known as the OpenPiton configuration) cannot boot Linux. The specific behaviour is that the core hangs on the user mode boundary while booting Linux. A crash at the user mode boundary points to MMU behaviour, but this has not been validated.

The configuration used for booting Linux in the CI flow is cv32a6_imac_sv32_config_pkg.sv, also known as the the 10xEngineers configuration.

The difference between these configurations is listed in the Tables below:

Lines 1..47

The following are all scalar localparams:

Lines	cv32a6_imac_sv32_config_pkg.sv	cv64a6_imafddcv_sv39_config_pkg.sv
13c13	localparam CVA6ConfigXlen = 32;	localparam CVA6ConfigXlen = 64;
15c15	localparam CVA6ConfigRVF = 0;	localparam CVA6ConfigRVF = 1;
26		localparam CVA6ConfigBExtEn = 0;
27	localparam CVA6ConfigHExtEn = 0;	localparam CVA6ConfigHExtEn = 0;
28	localparam CVA6ConfigVExtEn = 0;	localparam CVA6ConfigVExtEn = 1;
41c42	localparam CVA6ConfigDcacheByteSize = 32768;	localparam CVA6ConfigDcacheByteSize = 16384;
42c43	localparam CVA6ConfigDcacheSetAssoc = 8;	localparam CVA6ConfigDcacheSetAssoc = 4;
45c46	localparam CVA6ConfigDcacheIdWidth = 3;	localparam CVA6ConfigDcacheIdWidth = 1;
46c47	localparam CVA6ConfigMemTidWidth = 4;	localparam CVA6ConfigMemTidWidth = 2;

Lines 66..67

Localparams of type cache_type_t:

Lines	cv32a6_imac_sv32_config_pkg.sv	cv64a6_imafddcv_sv39_config_pkg.sv
66c67	localparam config_pkg::cache_type_t CVA6ConfigDcacheType = config_pkg::HPDCACHE;	localparam config_pkg::cache_type_t CVA6ConfigDcacheType = config_pkg::WT;

Lines 72..155

The following values are all members of localparam config_pkg::cva6_user_cfg_t cva6_cfg

Lines	cv32a6_imac_sv32_config_pkg.sv	cv64a6_imafddcv_sv39_config_pkg.sv
74c75	VLEN: unsigned'(32),	VLEN: unsigned'(64),
79c80	NrCommitPorts: unsigned'(2),	NrCommitPorts: unsigned'(1),
92c93	RVB: bit'(1),	RVB: bit'(CVA6ConfigBExtEn),
93c94	ZKN: bit'(1),	ZKN: bit'(0),
123c124	NrNonIdempotentRules: unsigned'(1),	NrNonIdempotentRules: unsigned'(2),
124c125	NonIdempotentAddrBase: 1024'({64'b0}),	NonIdempotentAddrBase: 1024'({64'b0, 64'b0}),
125c126	NonIdempotentLength: 1024'({64'h8000_0000}),	NonIdempotentLength: 1024'({64'b0, 64'b0}),
146c147	InstrTlbEntries: int'(2),	InstrTlbEntries: int'(16),
147c148	DataTlbEntries: int'(2),	DataTlbEntries: int'(16),
148c149	UseSharedTlb: bit'(1),	UseSharedTlb: bit'(0),

[Jbalkind]

To be clear, we only observed this issue on UltraScale+ FPGAs and we've actually had a user who ran on vcu118 or similar without the issue so I think there's a specific timing component to the issue