Yup- over the weekend we got two other people on the issue, so a lot of discussion took place. I think we have a true line of attack now.I see the github issue has got very busy...
Alright- if we force all DMA allocations via the IOMMU as uncached, then we can probably safely mark CACHED as supported in the kernel with minimal issues (other than performance, which with amdgpu seems to be quite respectable on a Radeon HD 7870). Honestly, the single grouping support of the IOMMUs shouldn't be a major issue, since Its-rad on the GitHub issue was able to get IOMMU grouping working with both IOMMU2 and IOMMU5 on the Pi (though he said IOMMU5 seems to work better). I think with the hardware we already have present on the D0 stepping should be enough to get this working, at least with a single group per IOMMU.For the first question, probably none of those. The IOMMUs are close to SDRAM. They operate on uncached memory and pay no attention to protection flags.* They do support partial TLB flush during unmap.
Performance from what I've seen with amdgpu is fine, even without any form of caching on the PCIe PHY or IOMMU side. Honestly it was more a question of "is the IOMMU coherent with the ARM caches", which it would seem from my own testing that it is, in fact, not. I appreciate the additional details though, since that gives me and everyone else a bit more to build off of.Addresses above 64GB can be routed back through the VPU L2 cache. This is not coherent with ARM caches and you basically never want to do this.
Yeah that sounds about right- since without isolated MSI when any PCIe device triggers the message signaled interrupt it'll fire the generic IRQ handler which can be running anywhere- the host kernel, a guest kernel, etc. It probably doesn't matter too much but it's more a matter of curiosity. So, the IOMMU is a single group device from what I can tell here- if you need isolation you have to use another IOMMU (if even possible since it seems most of them are for specific peripheral sets).I guess the Linux nomenclature for "isolated MSI" roughly maps to "MSI controller enforces DeviceID authenticity" - neither 2711 nor 2712 MSI(x) targets pay attention to this sideband data. Same behaviour as the IOMMU - they just operate on address + data.
Okay, so if I have this straight: the Pi 5's PCIe bus is mostly compliant, except it cannot access unaligned memory addresses and it doesn't snoop the ARM cache so it's not coherent either. If I have this right, then that's all I really need to know to get to the bottom of these driver issues. We can add the PCIe bus and all of its devices to an IOMMU group, then in the IOMMU DMA memory allocation function, we can force all allocations to be uncached, so we don't run into these coherency issues. From what I saw from the amdgpu driver, with the TTM allocator set to uncacheable memory, the driver works fine, but when allocating cacheable memory, I get a "ring gfx test failed (-ETIMEDOUT)" (unsurprisingly). The behavior is the same with the IOMMU group enabled (but cacheable memory allocations disabled), suggesting that either the IOMMU is 1. not allocating memory or 2. allocating cacheable memory or 3. not configured correctly and writes are not going to the PCIe bus. This has given us a lot to work on so we'll see how this progresses in the next week, but I think we have our work cut out for us (even if it seems quite possible).Pi 5 PCIe "unaligned" support - in the outbound direction this is a function of the MMIO space's memory attributes assigned by the CPU's MMU - ARM convention is to use Device memory for PCIe BARs so no unaligned accesses are permitted. In the inbound direction, unaligned accesses are decomposed into the required memory ops by the PCIe RC.
Statistics: Posted by srgb565 — Mon Sep 08, 2025 5:51 pm