Evolving the page_pool API

Today mlx5 is not using the DMA-mapping part and is basically using page_pool as a fast alloc-side cache (see later why it is fast), and a return point for XDP-redirect. Packets send towards netstack still have a refcnt based recycle scheme, as a driver-layer on-top of page_pool.

The original idea was to modify put_page() to catch pages with refcnt==1 and recycle those. This was rejected upstream, but meanwhile a hook have been created at the exact spot we need.

The function calls used is called: put_devmap_managed_page().

It is used by HMM (Heterogeneous Memory Management), which is used by device memory like GPU on board memory. The DAX system also leverage this via type MEMORY_DEVICE_FS_DAX.

The question is (1) can page_pool also leverage this, (2) is the performance good enough.

TODO investigate: The page "zonenum" must be ZONE_DEVICE, which semantic is unclear, more info needed. Can this type of page be used for "normal" network stack delivery?

The code that end-up being called is: __put_devmap_managed_page(page); The callback in __put_devmap_managed_page(), is implemented by calling: page->pgmap->page_free(page, page->pgmap->data);

From struct-page the part containing this area looks like:

struct {        /* ZONE_DEVICE pages */
        /** @pgmap: Points to the hosting device page map. */
        struct dev_pagemap *pgmap;
        unsigned long hmm_data;
        unsigned long _zd_pad_1;        /* uses mapping */
};

TODO: Read include/linux/memremap.h and figure out, (a) that struct dev_pagemap is used for, and (b) what it means and what requirements are associated with being a MEMORY_DEVICE_PRIVATE type using ZONE_DEVICE.

For storing the DMA-addr (dma_addr_t), we could use the hmm_data area.

In mm/hmm.c the callback pgmap->page_free is assigned to hmm_devmem_free, and pagemap->data is the instance of the devmap itself. This could fairly easily be adapted for calling page_pool_put_page(), where the pagemap->data will be the instance of the page_pool.

The level and use of indirect calls (e.g. page->pgmap->page_free) is slightly concerning (in a spectre-v2 context).

Feasibility: (not needed for page_pool as is) Investigate if pages can be VMA mapped into userspace. There are indications that this is not possible, given the ZONE_DEVICE and page->pgmap share page->mapping area.

Another possible return hook is when the SKB (sk_buff) is freed.

SKB freeing basically all goes through __kfree_skb(), and following the code path expanding code-inside for reaching freeing the page addr:

void __kfree_skb(struct sk_buff *skb)
{
        skb_release_all(skb) {
                if (likely(skb->head)) {
                        skb_release_data(skb) {
                                skb_free_head(skb) {
                                        if (skb->head_frag) {
                                                skb_free_frag(skb->head) {
                                                        page_frag_free(addr);
                                                }
                                        }
                                }
                        }
                }
        }
}

Here the skb->head_frag bit is used for saying that the packet data is allocated from a page (both covering page fragments and order-0 pages).

We need some extra info at __kfree_skb() time, in-order to know what page_pool the data-page need to be returned to, and this info can only be derived from the SKB. Extending struct sk_buff (SKB) is a very sensitive topic, and is in-general not allowed. Thus, we have to reduce the info needed in the SKB to the absolute minimum.

The DMA-addr (8-bytes) could be stored in the struct-page at page->private.

The XDP-redirect system, uses struct xdp_mem_info for storing the necessary info in the (struct) xdp_frame. The xdp_mem_info is 8 bytes, and could be reduced further, (1) 'type' can easily be made smaller (given enum xdp_mem_type is smaller), and (2) is artificially capped at MEM_ID_MAX=0xFFFE.

struct xdp_mem_info {
        u32                        type;                 /*     0     4 */
        u32                        id;                   /*     4     4 */
        /* size: 8, cachelines: 1, members: 2 */
        /* last cacheline: 8 bytes */
};

Reusing xdp_mem_info for storing the info, would allow us to reuse the xdp_return_frame() API more or less directly.

Crazy idea: Use one bit in SKB, saying this originates from an xdp_frame, and via skb->head pointer, find offset to start of xdp_frame, and then use the struct xdp_mem_info, and return frame like/via xdp_return_frame() API.

The network stack current handling of packet-data pages is refcnt based.

So, what is the trick used by drivers that want to (1) keep the page DMA-mapped and (2) do faster recycle (than going through page-allocator)?. It obviously need to leverage a refcnt based scheme on the page.

The general idea is to elevate the refcnt on the page, before sending it through the network stack, and keep the page pointer on a tracking waiting-queue/list (often a queue). Thus, this waiting-queue contains pages that are in-flight, and there is no signal or callback when they are "done". To figure-out when a page is available for reuse, the driver must poll or revisit this waiting-queue and by observing the page refcnt know if the page can be reused now.

Most drivers poll or revisit the waiting-queue, when doing other work, like when needing to refill the RX-ring with fresh frames/pages. There are different strategies when head of the waiting-queue contains a page that is still in-flight, as it could mean several things. (1) could indicate too fast recycling and next pages are likely also not-ready for reuse, or (2) could be this page is being held by e.g. TCP stack for retransmit while the next page could be ready for reuse. Thus, there is a choice to either (a) keep the page for next round (hoping it will be reusable soon) or (b) release the page by DMA-unmapping and returning the page to the page-allocator.

When the page_pool interacts with XDP, it gets pages returned from XDP via the xdp_return_frame API hook/callback. Given network stack doesn't provide such hook, the mlx5 driver used its existing refcnt based recycle queue, on-top of page_pool. And page_pool was modified to fall-through to put_page() if refcnt is not equal 1. This double layer result in the page_pool cannot be responsible for the DMA-mapping.

An alternative to introducing a return-hook for the network stack, is to integrate a refcnt based waiting-queue inside page_pool, that is only used when frames/pages travel through the network stack. This is a hybrid approach, but integrating this would allow page_pool to keep the DMA-mapping.

The page_pool leverage the knowledge/requirement, that allocations MUST happen from NAPI context. (During driver init of RX ring, not in NAPI context, it is known that no concurrent users of this page_pool exist, thus it is still safe).

A NIC driver creates a page_pool per RX-queue. Combined with the protection provide by NAPI context (per RX-queue), allow page_pool to get pages from a completely unlocked array-style stack-queue (see struct pp_alloc_cache). It is difficult to get any faster than this.

Code from: __page_pool_get_cached()

if (likely(pool->alloc.count)) {
        /* Fast-path */
        page = pool->alloc.cache[--pool->alloc.count];
        return page;
}

Another optimization leveraged by page_pool is that, for frames that needs to be dropped while still running under the RX NAPI context, either for error cases or explicit drop due to XDP_DROP action. The API call page_pool_recycle_direct() can be used, which as described above, simply returns the page to an array stack (code in __page_pool_recycle_direct()).

One advantage is that even with XDP_REDIRECT, the redirect core-code can choose to drop frames and see almost the same drop performance as driver level code (via calling xdp_return_frame_rx_napi).

In __xdp_return there is a rhashtable_lookup() call on the xdp_mem_info->id, which looks like a performance issue, but it isn't. While we would like to amortized this via an explicit bulk return API, the lookup is surprisingly fast.

The reason behind this being fast, is that the hash function defined (xdp_mem_id_hashfn) is not really a hash function, as it simply returns the id as a key. Given we use a cyclic increasing ID, the key/id is unique and "hash-distributed" enough already.

When the RFC page_pool got ripped out and converted it to be used in network stack, the ALF (Array-based Lock Free) queue was dropped. Instead the ptr_ring was used and replaced the internal page_pool queue. The ptr_ring actually do have some performance advantages over ALF-queue, e.g. reduces the cross-CPU cache-coherency talk, and is faster cross CPU.

One disadvantage is that ptr_ring_produce (or ptr_ring_produce_bh) call takes a lock. And it is currently called per returned page, see __page_pool_recycle_into_ring(). This obviously is a scalability issue waiting to happen, when/if multiple CPUs want to return packet originating from the same RX-queue.

See No description for this link for benchmarks showing this is still very fast, in the non-lock congested case.

Thus, this need to be fixed/improved. The basic idea to address this is through bulking. But there are two ways to introduce (1) expose an explicit bulk return API, or (2) hide it in the page_pool API via clever lockless per CPU store (that return pages in a bulk).

Jesper have a lot of details for option (2), as a significant performance gain can be achieved by having knowledge about (and separating) what context the kernel is running in (softirq/bh, hardirq, process-context).

Benchmark based development notes: Proof-of-Concept (PoC) modified mlx5 driver and bq_xmit_all (devmap.c) to call xdp_return_frame (vs normally calling xdp_return_frame_rx_napi), xdp_redirect_map frame out a driver with ndo_xdp_xmit, but manually unload XDP from egress driver, causing driver ndo_xdp_xmit to return error, causing all xdp_frame to be freed/returned. Diff (1/14340444-1/12728465)*10^9 = -8.83 ns difference.

• The performance with xdp_return_frame was: 12,728,465 pps.
• The perf with xdp_return_frame_rx_napi was: 14,340,444 pps

Thus, in a non-lock congested case, the performance is still very good.

The base overhead of allocating and freeing an order-0 page in a tight loop, always hitting the PCP fast-path, measured with prototype kernel module page_bench01.

dmesg output
[ 1315.759541] page_bench01: alloc_pages order:0(4096B/x1) 223 cycles per-4096B 223 cycles
[ 1315.768139] time_bench: Type:alloc_pages_order_step Per elem: 223 cycles(tsc) 62.096 ns (step:0)
 - (measurement period time:0.062096443 sec time_interval:62096443)
 - (invoke count:1000000 tsc_interval:223548984)

Show alloc+free order-0 page cost: 223 cycles or 62.096 ns. This almost consume the entire budget for 10Gbit/s wirespeed which is 67.2 ns. Converting, 62.096 ns to packets per sec equal (1/62.096*10^9) 16,104,097 pps, which doesn't represent any real pps (just to relate this to pps numbers).