EDIT*** It's become clear the issue isn't vRAM related, but more than likely an incompatibility between the system BIOS & GPU's with moden UEFI firmware

EDIT2*** So the issue turned out to be the Dell BIOS is incompatible with GPU's that include UEFI compatibility (basically anything in the last 10ish years). The fix is to remove the UEFI image from the GPU vBIOS forcing it back to compatibility mode and it boots fine now. So says the 2GB P400 Quadro I now have running in my R210ii Plex Server.

-------------------------------------------------------------------

Trying to enable all PCIe GPU's in Dell Servers.

Currently only some cards work, and from my testing and research the BIOS is looking for a PCIe GPU with <= 1GB RAM at boot. Any more than 1GB vram and the system won't post.

Ref: https://forum.level1techs.com/t/dell-pow...on-found/134343

With UEFITool I was able to enable PCIe Hot-Plug on my R210 ii which has allowed me to test that GPU's with >1GB do in fact work by hot-plugging the GPU after Windows has loaded. If a GPU with >1GB is installed at POST, the system halts with a static cursor _ and doesn't boot. I have even managed to insert the card after the device check and before the POST screen and have it recognised in the EFI Setup Screens, so clearly there is some arbitrary check stopping the use of these cards at boot.

There are many Dell server models with a similar issue, so once a patch is found it will hopefully work with other models. My specific case for testing is as follows...

Model: Dell PowerEdge R210 ii
BIOS Version: 2.8.0 (Home Lab, don't need the performance hit of the later patches).
I have multiple system boards & an SPI flasher should we need it.

2.8.0 Download
https://www.dell.com/support/home/au/en/...oweredge-r210-2

2.10.0 Download (Includes Spectre & Meltdown patches for those that need them)
https://www.dell.com/support/home/au/en/...oweredge-r210-2

I've had a look around IDA, tho my assembly knowledge is limited and figured I should ask for some pointers. I don't believe the lock is Device ID related.

If a solution is found, I would be most interested to know how it was discovered.

---------------------------

@Lost_N_BIOS does this sound like a challenge you'd be keen to help me work through?

My biggest hurdle is working out which module detects PCIe devices. I assume that's where this check takes place. I'm comfortable enough working through the assembly in IDA once I have an idea of where to look.

Wow, I've seen hot-plug PCIE option in many BIOS, but never knew someone could really do that and not release the magic smoke

Since you have programmer, send me a dump of the BIOS, if there is two chips dump both and send in one zip, I will compile them to inspect/edit.

* Edit -From your list at the level1techs site, looks like all GPU's you tried that work are old/legacy (non-UEFI vBIOS), and all that fail are probably Legacy/UEFI vBIOS (both in vBIOS Image)
So, it may be failing due to UEFI in vBIOS. Do you have any modern UEFI vBIOS capable GPU's that are below the 1GB vRAM you suspected.

I doubt any whitelist or block/allow list by GPU chipset, model, or brand could be used, simply based on the random ones that do work

Maybe... If we are lucky, I can add UEFI image to vBIOS on one of your known currently compatible GPU's, then you can test above theory
Please send me vBIOS dump from these two cards. You can dump with GPU-z or use programmer, up to you
XFX HD 5450 512MB DDR3 (HD-545X-YRH2 V2.1)
BFG GTX 260 (BFGRGTX260896OCB)

Reverse test of the above could be done too, remove UEFI image in vBIOS of any of these (if none have UEFI image and are only legacy vBIOS, then theory shot down right there)
Zotac mini GTX 1070
Club 3D R9 280x
Palit GTX 1060 6GB

Still reading your post there, so more thoughts may get edited in here shortly
This may also be a simple BIOS setting change needed, based on the error info you found (system resource configuration error)

lol, I cooked the PCIe bus on my first board, and a GPU also . So i bought 2 more spares just in case :) . Hot-Pluging is normally quite safe tho, my problem was power modding the PCIe riser to bypass the 25W power limit and I shorted the 12v rail.

Anywho... BIOS dump is attached. Only 1 chip.

That list on level1tech's isn't mine, but rather someone else's notes on the same problem.

I have looked through all the Available & Hidden BIOS options and nothing that remotely resembles gfx helped.

I have a GT710 2GB, GT530 2GB, RX2600PRO 512mb on hand, but nothing 'new' < 1GB . I have others, but those are the only three that fit in the ~25W power budget this server has for its PCIe slot.

My ultimate Goal is to install a P400 Quadro to serve as a Plex Transcoder.

I did have one other idea... can we mod a vBIOS to report only 1GB RAM? that would also test the VRAM theory

Ok, had another idea. I can buy identical GT710's in 1GB & 2GB variants. That's the easiest way I can think of testing the vRAM theory unless we can change the reported vRAM on the card I already have.

Gigabyte GV-N710D5-1GL & GV-N710D5-2GL

So I've installed a 1GB GT 710 and same issue. System halt with the _ cursor. There goes my 1GB vRAM theory, super glad I asked before wasting time in IDA.

Guess that leaves the UEFI vBIOS incompatibility.

Is that something we can add support for in the Dell BIOS?

Some Firmwares allow you to decide whenever you want to prioritize loading UEFI or legacy BIOS Option ROMs, or even disable loading an Option ROM alltogether. Do yours have something similar to this? https://www.thomas-krenn.com/de/wikiDE/i..._pcie_oprom.php
Perhaps one of the reasons why I love Supermicro so much is precisely because they often allows you to disable loading Option ROMs with a PCIe Slot level granularity: https://hardforum.com/threads/disable-op...-board.1984211/
...albeit some Workstation ASUS ones can, too: https://www.thomas-krenn.com/de/wikiDE/i...nfiguration.php
That can help in some obscure Option ROM compatibility issues, like it seems that yours is.

At some point I tested if I could not initialize a Video Card for VGA Passthrough purposes. With the IGP disabled and disabling loading the Option ROM of the PCIe Slot where the discrete Video Card was, the Motherboard beeped with an error code that pointed to a VBIOS not found error, precisely as intended. It could still pass POST and the video output turned on as soon as Linux GPU Drivers kicked in.

@zir_blazer that would certainly make life easier. Alas no options like that here, not even hidden options as far as I can tell.

Attached are all the options (Available & Hidden) I can access. (Extracted with H2OEZE)

Also, because I like to try stuff myself... i've attempted to remove the GOP driver from the GT 710 I have with no luck. I think i'm stuck on the checksum not matching. My attempt is attached.

@Lost_N_BIOS so your nudge in the direction of UEFI compatibility was what I needed.

I have now successfully removed the UEFI image from my P400 Quadro and it boots perfectly.

I'm still interested to know if there is anything we can do to the Dell BIOS so any card can be used without having to mod it first? But for now this is working great.

Also, what's involved in unlocking all the BIOS options I can see in H20EZE for this BIOS? Is it a simple patch or more involved?

@theunknown

This is a problem I've had for a couple of months now. Was just looking to tackle it again, and found your post. THe magic of GOogle.

Do you have instructions or a link to how you removed the UEFI image from the P400 Quadro? Have you tried it with a GT 710?

I'm looking at an R210 ii (Riverbed modded back to stock), and trying to get the GT 710 working in it.

@wan0net

It will work on any GPU I expect. I couldn't find instructions for Nvidia cards so had to teach myself the basics of a vBIOS to get it done, in the end it's super easy. I'll post a tutorial in the next couple days.

EDIT* Incidentally, if you don't have an iDRAC6 Enterprise module yet, have a read of this to save you countless hours also :)

https://www.reddit.com/r/homelab/comment...ac_no_boot_fix/

Thanks, much appreciated!

@theunknown - So, to clarify, you've blown away the 1GB as suspect, correct? And now, think maybe what I mentioned about UEFI vBIOS could be part of the problem?
Ohh, wait! So, you solved it And what I guessed about the vBIOS UEFI+Legacy image was correct? If yes, then, yes, it may possibly be resolved by changing a BIOS setting

Do you have a flash programmer? If yes, dump the BIOS and send to me (if there is two BIOS chips, dump both and send to me)
I HATE INSYDE BIOS.... There, that's out of the way. Unlocking BIOS menus may be possible, maybe not for me, but Insyde Pro's at BIOS-Mods.com forum may be able to do it for you if I can't
Changing BIOS options, you can see or not, can be done easily with H22OUVE (dump vars, edit, flash back) - Or by dump BIOS, edit setup + NVRAM, flash back, similar/same as how you enabled hot-plug

As for unlocking menus, settings etc, I'd need to see full images of your BIOS first (mainly, just show me one main image of each main page, so I can see what you can see now)
Then any areas you want full options you may be missing show me the entire section like advanced >> Submenu with settings you want that are missing
I can always easily make hidden settings visible to you in any section you can already see, but sometimes I can't make a menu section visible to you if it's not already (Sometimes I can though, I'll have to look after I see your images first)

Looking through the BIOS options using the HDR from stock EXE, I see you can change the following settings and deal with this without having to edit vBIOS
EFI Device First (Default Enabled) >> Set Disabled *may fix issue alone
If that fails, set this >> Boot Mode (Default UEFI) >> Set >> BIOS (= Legacy)

Your BIOS, from what I can see in HDR, which may not be all, does not offer granular enable/disable of UEFI/Legacy rom loading, so above may be your only two possible things to play with in attempt to get around this without removing GOP image from GFX crads vBIOS

@wan0net - if you do the vBIOS edit, the part to worry about here is flashing of mod BIOS, is that possible on your model/series (especially with nvdia).
You'll have to google about that and your card, your cards series etc. If you can't flash mod vBIOS on your card then you can't dump vBIOS and edit it to remove the UEFI/GOP image.

If you find you can, and you simply need edited vBIOS, post your vBIOS here and I can remove GOP for you

To you both, simply setting the legacy is last flag may fix this too, or setting legacy is only bit may as well, unsure about this, testing required.

@Lost_N_BIOS

Yes I was able to get it working by removing the EFI vBIOS Image.

I tried the "Boot Mode" BIOS Option without success, but completely missed "EFI Device First". I'll try that when I get home & report back, that would be a much simpler fix if it works.

I'll also reply with the BIOS Menu info you've mentioned.

Make sure you change the option in NVRAM/VSS too (not just in setup), otherwise the change may not be applied.
Easiest way to make those changes is dump current vars with UVE, make your desired edit, then program back. This changes NVRAM/VSS (current in use settings)

Dump
H2OUVE.exe -gv vars.txt

Write
H2OUVE.exe -sv varsM.txt

Use setup IFR to get varstore from the setting you are changing, then locate that varstore by GUID at top of IFR, then find it in the dumped vars (there may be multiple copies, or two different sections with same GUID, edit both)
On the IFR per the setting you want to edit, use the variable from >> VarStoreInfo (VarOffset/VarName): 0x85 << to edit the vars at that byte

Example for above >> VarStoreInfo (VarOffset/VarName): 0x85, VarStore: 0x1234
VarStore: 0x1234 >> = VarStore: VarStoreId: 0x1234 [A04A27F4-DF00-4D42-B552-39511302113D], Size: 0x3CB, Name: SystemConfig << So we are editing "SystemConfig" generally, but more specifically GUID A04A27F4-DF00-4D42-B552-39511302113D and it may be in Vars under System Config and also Custom, or other names
VarStoreInfo (VarOffset/VarName): 0x85 << in vars, edit byte 85

If you need edited example, give me your vars dump, I will edit something and show you what I edited, then I'm sure you will see and get it (if you don't already understand )

I'm running 2.10 and I don't appear to have that EFI Device First (Default Enabled) >> Set Disabled - I have Set Boot Retry which appeared to work, and then on repeat reboots I got nothing from the card.

I'll try and connect my card up to another PC and try and dump the vBIOS.