Troubleshooting • Re: Raspberry Pi 5 always boots from NVMe SSD, even with BOOT_ORDER=0xf41

Remove the SD card and set the boot order again.

I haven't tried this yet. So I shutdown the pi with BOOT_ORDER=0xf416, removed the sd card, booting from NVMe only and changed BOOT_ORDER to 0xf41.Now shutting down again, inserting the sd card, so that NVMe and sd are connected again. Unfortunately, it's still booting from NVMe:

some pcie hats cant be auto-detected by the firmware
so you need a config entry in the rpi-eeprom-config to make it detect the nvme, even with nvme in the boot-order
i forget what the entry is, but it should be in the forum search

It's called PCIE_PROBE=1 and documented here: https://www.raspberrypi.com/documentati ... -from-pcie
According to the docs, it's required for non-HAT+ devices. I already found this before and was a bit confused since some posts said it's required, others won't even mention it. The official documentation don't explain much about this flag. Other sources only shortly write it enables enumeration over the PCIe bus. Which seems confusing to me, since there is only a single PCIe connector. I can only guess that it would be theoretically possible that a PCIe hat itself has another PCIe connector, so that we can connect 2 or more devices using the same port.

So I first tried to boot from NVMe without setting PCIE_PROBE=1 to figure out more and it worked. I don't see how this directly affects booting from SD card. But to be sure, I tried to set PCIE_PROBE=0 in hope it may helps:After rebooting, it has been applied. But the RPI is still booting from NVMe:

Also relavant to OP viewtopic.php?p=2189368#p2189368

https://www.raspberrypi.com/documentati ... bling-pcie

I read this section of the documentation twice and as I understood them, BOOT_ORDER=0xf41 means: Try sd card first, then alternatively USB storage devices. If both failed, restart. While 0xf416 adds NVMe before them. 0xf416 worked, the pi has booted from NVMe and ignores the (also bootable) sd card. But 0xf41 still result in booting from sd card, even if this boot order doesn't even contain NVMe. I can't see that I missed anything that results in doing something wrong.

I slept over this issue and thought how this could be possible. My guess was that the files may be written in the wrong direction. So I had a look at the mount points again and saw something strange:The root partition is mounted on the NVMe SSD, but /boot/firmware on the sd card? Yesterday (before trying out the suggestions) this was not the case, I included an lsblk call in my original post:BOOT_ORDER is written to the firmware, not the OS drive. But before, it's written to /boot/firmware:Is it maybe something like rpi-eeprom-config writes it's config to device A, while it's read from device B during boot? Or even never read, when pending updates are written to device A and the RPI is looking for configuration changes on device B during boot.

I mounted the sd card root partition to have a look in the configuration of the .bin files in /lib/firmware:The file pieeprom-2024-01-22.bin doesn't exist there, since I checked for firmware upgrades and installed one available at the end of my original post. So this update was applied on the NVMe ssd. I think this leads into another source of error, so that I should re-flash the RPIOS on the card, boot from it and copy the new RPIOS installation to the NVMe SSD to let both know about the updated firmware.

Statistics: Posted by L1nux — Sun Feb 04, 2024 11:15 am

---