Kepler GPU not advertising ReBAR capability on PCIe bus

[terminatorul]

Hi @Xelafic, are you still around please ?

I tried to enable ReBAR on my very low end Kepler GPU (GK208, GeForce GT 710), and it looks like ReBAR is not advertised on the PCI 2.0 bus (see image, also CPU-Z report attached). This is true even after seting straps bits.

I rememeber the original discussion where you showed me a 2GB BAR on your GT 640 (Kepler GK107).

Can you tell me how did you get it working ? Did you do something special ? I see the straps bits should be the same ... did you see any differences ?

@xCuri0 do you know what can I do if ReBAR capability is not present in the PCI configuration space ?

X399-AORUS-GAMI-GT710.txt

[xCuri0]

@terminatorul I think Xelafic said Pascal and older only expose the strap bits, the BAR size changes using them.

[terminatorul]

Looks like @Xelafic really got a Keppler card going, see the last screenshot in the message here: xCuri0#89 (reply in thread)

[xCuri0]

@terminatorul isn't that same as what you are doing now ? just changing strap bits and getting large BAR size

[terminatorul]

Now I am setting the straps bits, and the the GPU still has no ReBAR extended capability in the PCI configuration space

[xCuri0]

@terminatorul xelafic also did not get resizable bar you can see in the screenshot.

What you will find though is that device manager shows the GPU has increased BAR size. And if the driver working correctly with it the vulkan memory pool in GPU-Z will too

[terminatorul]

Oh, I see, this must explain the empty output in GPU-Z, in the screenshot from @Xelafic.

For my board it stil didn't work 'though, Device Manager says "This device is not using any resources because it has a problem" ..., and it shows the device with an exclamation mark !

:(

[xCuri0]

Will have to wait for Xelafics response then maybe Kepler needs to be configured a bit different.

[ghost]

Hi @terminatorul , you've been busy!

Kepler uses offset BAR0 offset 0x101000 and bits 17:14 IINM to give a BAR1 range of 64MiB to 64GiB. AFAIK nvidia Geforce Resizable BAR capability started with Turing. Models before Turing can have BAR1 resized directly by their straps as there is no override via Resizable BAR capability.

FWIW I also tested an early Maxwell GTX 750Ti and that behaves similar to Pascal using BAR0 offset 0x10100c for strap bits.

BTW the claimed author of GPU-Z is asking for someone to test Turing Resizable BAR with a new GPU-Z build? See xCuri0#118 (comment) I think better it works with systems enabled by your software foremost. Or perhaps @UnidentifiedTag can do it?

[terminatorul]

Nice to hear from you again :)

Looks like straps 0 bits 17:14 do not quite match the envytools page at https://envytools.readthedocs.io/en/latest/hw/io/pstraps.html, most likely you meant bits 17:16, with a max of 512 MiB size. Do you have some different information about this ?

I have a possible scenario now:

So straps bits for BAR1 size are grouped in 2 parts, that are added together:

• straps0 (0x101000) bits 17:16
• straps1 (0x10100C) bits 22:20

NvStrapsReBar driver uses different sets depending on the target BAR size:

• first set (straps 0 bits 17:16) is used for a BAR size up to 256MB
• mostly the second set is used for larger BAR sizes, and the first set is kept at the default of 10 (that originally indicates 256 MiB)

Now the GK208 chips works when setting BAR1 size from 128 to 256, and does not work for higher values.

So it indicates straps0 bits are working, and straps1 bits are triggering the error in the Device Manager. I will check this again when I get back to my Keppler GPU...

But my straps0 bits (17:16) only allow for 512 MiB BAR size max. Can you check if this is right please ? So how did you get 2 GiB then ?

(P.S: Turing - with a g, instead of Turin...)

[terminatorul]

Looks like the stock BAR size for this card is 128 GiB instead of the usual 256 GiB.

I can set up to 2 GiB BAR with Microsoft Basic Display Adapter, and I can set 256 GiB (instead of 128 GiB) with the NVIDIA driver 474.64

Maybe my GT 710 card is just too low end for large BAR sizes :(

[xCuri0]

I think you meant MiB ?

[terminatorul]

Yes, indeed

[ghost]

Does it show 128MiB under Resizable BAR on GPU-Z? Suggested last year it would be nice to see BAR sizes for all cards instead of 'unsupported GPU'. They didn't seem to be aware of cards existing with sizes other than 256MiB! Don't know if something was done or not.

[terminatorul]

No, I used the output of NvStrapsReBar.exe to see the 128MB BAR. NvStrapsReBar.exe enumerates the memory resources from Device Manager, and picks the larger memory size as the current GPU BAR size.

I also sugested to the GPU-Z maintainer / developer to show a large BAR size even if it is not resizable. The answer was the API used by GPU-Z, to read the BAR size, no longer works if the BAR is not resizable (so without ReBAR enabled). GPU-Z is closed source, so I don't know what API it uses to read PCI BAR size.

[terminatorul]

BAR1 tests on GT640 using offset 0x101000 and bits 17:14

BAR0 Reg	Bits	 BAR1
??101000	17:14	 Size
================================
8040049e	0000	 64MiB
8040449e	0001	128MiB
8040849e	0010	256MiB    Default
8040c49e	0011	512MiB
8041049e	0100	  1GiB
8041449e	0101	  2GiB
8041849e	0110	  4GiB
8041c49e	0111	  8GiB
8042049e	1000	 16GiB
8042449e	1001	 32GiB
8042849e	1010	 64GiB
8042c49e	1011	128GiB
8043049e	1100	256GiB
8043449e	1101	512GiB
8043849e	1110	  1TiB
8043c49e	1111	  1TiB

Not sure why the last 2 are the same, maybe a limit? Still, should be more than enough for the 2GiB of VRAM 😀

@Xelafic have you considered reporting these findings to envytools ?

[ghost]

Given the straps data hasn't been updated in years on envytools I believe the team has moved on to other agendas. Probably they are already aware but just don't have the resources and time to add it. Given that, it would seem inappropriate to mention it. Saw your post on VRAM from BAR0 BTW. Thanks, would never have found it myself with such a convoluted method!

Since we are talking about straps, FWIW I have seen an Nvidia post showing a Quadro K600 with nvidia-smi memory command supported. The straps on the GT640 also allow changing the device ID so I configured the GT640 to report as a K600 which allowed installing the Quadro drivers which worked well with the card but still didn't produce nvidia-smi data for memory 😞 Perhaps something needed in the vbios? I didn't try flashing the K600 vbios to the GT640 as it seemed too incompatible.