For some reason, when allocating memory from user space linux, I have a problem with the write response in AXE Memory Mapped to PCI Express module

Question

I have a PC as a root complex and Xilinx fpga device as an end point that writes via PCI Express to root complex RAM using DMA and a linux driver to it. I have a problem that I can't solve in any way. For some reason, when allocating memory from user space, I have a problem with the write response in the AXI Memory Mapped to PCI Express module.

For example, during the execution of this code, my recording PCI via dma is constantly paused.

 const int numAllocations = 10000; 
for(int i = 0; i < 1000000; ++i) {
    std::vector<Dummy*> allocations;

    for (int i = 0; i < numAllocations; ++i) {
    Dummy* ptr = new Dummy();
    allocations.push_back(ptr);
    }

I get the addresses that I give to the fpga in the driver using the function:

        fdev->dma.buffers_kernel[i] = dmam_alloc_coherent(
        &pdev->dev,
        fdev->dma.buffer_size,
        &fdev->dma.buffers_device[i],
        GFP_KERNEL);

I have already tried to disable IOMMU in the OS and in the BIOS, but it did not help. Someone can explain why this is happening. After all, there should be plenty of RAM and PCI speeds. I think the problem has something to do with the server RAM memory.

Hi I'd like to help, but I'm not quite following the question. Can you elaborate on what the problem is exactly? If the first code snippet is the one with the problem, please give more details with what context the code is running in (is this kernel driver code? code running on an FPGA soft core? etc.) and what you're observing vs. what you expect. Its also not clear how "the addresses that I give to the fpga" ties into the top snippet. — GandhiGandhi
– GandhiGandhi, Commented Apr 15 at 15:07
I've done similar things before. Since you already have a driver, using custom ioctl calls, have the driver do the [internal] mmap equivalent. It can set up a userspace mapping that a simple mmap can't. It can disable caching on the memory range, etc. Because the driver knows the most about the device, it can do whatever is necessary to facilitate whatever the userspace app needs (e.g.) map the device into kernel address space, map device buffer(s) to userspace app, map the device to CPU memory (if needed), etc. Also, I'd avoid std::vector et. al. at least at first. — Craig Estey
– Craig Estey, Commented Apr 15 at 18:22

Alon Alush · Accepted Answer · 2025-04-15 15:04:34Z

0

Don't allocate DMA buffers in user space. Only use kernel-allocated, DMA-safe memory for PCIe DMA. User space can access the data after the transfer, but not as the primary buffer.

answered Apr 15 at 15:04

Alon Alush

1,9588 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

For some reason, when allocating memory from user space linux, I have a problem with the write response in AXE Memory Mapped to PCI Express module

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related