XEN_DOMCTL_claim_memory

This domctl command allows a privileged domain to stake a memory claim for a domain identical to XENMEM_claim_pages, but with support for NUMA-aware memory claims.

A claim entry with a node value of XEN_DOMCTL_CLAIM_MEMORY_NO_NODE stakes a claim for host memory, exactly like XENMEM_claim_pages does.

NUMA-aware memory claims

Memory locality is an important factor for performance in NUMA systems. Allocating memory close to the CPU that will use it can reduce latency and improve overall performance.

By claiming memory on specific NUMA nodes, toolstacks can ensure that they will be able to allocate memory for the domain on those nodes. This is particularly beneficial for workloads that are sensitive to memory latency, such as in-memory databases.

Note: The ABI supports multiple claims for future expansion. At the moment, Xen accepts a single claim entry (either a NUMA-aware or host-wide claim).

Implementation notes

As described in XENMEM_claim_pages, Xen keeps track of the number of claimed pages in the domain’s d->outstanding_pages counter.

Xen declares a NUMA-aware claim by assigning d->claim_node to a NUMA node, which declares that d->outstanding_pages is claimed on d->claim_node.

See Hypervisor documentation > Memory Management > Memory Claims for details on the API semantics and implementation details of the claims infrastructure of the Xen buddy allocator backing this hypercall.

Used functions & data structures

This diagram illustrates the key functions and data structures involved in the implementation of the domctl hypercall command XEN_DOMCTL_claim_memory:

        %% SPDX-License-Identifier: CC-BY-4.0
classDiagram

class xen_domctl {
    +uint32_t cmd
    +uint32_t interface_version
    +uint32_t domain
    +xen_domctl_claim_memory
}

class xen_domctl_claim_memory {
    +memory_claim_t* claims
    +uint32_t nr_claims
    +uint32_t pad
}

class memory_claim_t {
    +uint64_aligned_t pages
    +uint32_t node
    +uint32_t pad
}

class xc_domain_claim_memory["xc_domain_claim_memory()"] {
    +xc_interface* xch
    +uint32_t domid
    +uint32_t nr_claims
    +memory_claim_t* claims
}

class page_alloc_globals["xen/common/page_alloc.c"] {
    +unsigned long outstanding_claims
    +unsigned long node_outstanding_claims[]
}

class claim["DOMCTL_claim_memory"] {
    +int claim_memory(d, uinfo)
    +int domain_set_outstanding_pages(d, pages, node)
}

class domain["struct domain"] {
    +unsigned_int outstanding_pages
    +nodeid_t claim_node
}

xen_domctl_claim_memory o--> memory_claim_t
xen_domctl o--> xen_domctl_claim_memory
xc_domain_claim_memory ..> xen_domctl : populates
xc_domain_claim_memory ..> claim : calls via <tt>do_domctl()</tt>
claim ..> xen_domctl_claim_memory : reads
claim ..> domain : sets
claim ..> page_alloc_globals : updates outstanding claims

    

Diagram: Function and data relationships of XEN_DOMCTL_claim_memory

Call sequence diagram

The following sequence diagram illustrates the call flow for claiming memory for a domain using this hypercall command from an OCaml toolstack:

        %% SPDX-License-Identifier: CC-BY-4.0
sequenceDiagram

actor DomainBuilder
participant OcamlStub as OCaml stub for<br>xc_domain<br>claim_memory
participant Libxc as xc_domain<br>claim_memory
participant Domctl as XEN_DOMCTL<br>claim_memory
#participant DomainLogic as claim_memory
participant Alloc as domain<br>set<br>outstanding_pages

DomainBuilder->>OcamlStub: claims
OcamlStub->>OcamlStub: marshall claims -----> OCaml to C
OcamlStub->>Libxc: claims

Libxc->>Domctl: do_domctl

Domctl->>Domctl: copy_from_guest(claim)
Domctl->>Domctl: validate claim
Domctl->>Alloc: set<br>outstanding_pages
Alloc-->>Domctl: result
Domctl-->>Libxc: rc
Libxc-->>OcamlStub: rc
OcamlStub-->>DomainBuilder: claim_result
    

Sequence diagram: Call flow for claiming memory for a domain

Claim workflow

The following diagram illustrates a workflow for claiming and populating memory:

        %% SPDX-License-Identifier: CC-BY-4.0
sequenceDiagram

participant Toolstack
participant Xen
participant NUMA Node memory

Toolstack->>Xen: XEN_DOMCTL_createdomain
Toolstack->>Xen: XEN_DOMCTL_max_mem(max_pages)

Toolstack->>Xen: XEN_DOMCTL_claim_memory(pages, node)
Xen->>NUMA Node memory: Claim pages on node
Xen-->>Toolstack: Claim granted

Toolstack->>Xen: XEN_DOMCTL_set_nodeaffinity(node)

loop Populate domain memory
    Toolstack->>Xen: XENMEM_populate_physmap(memflags:node)
    Xen->>NUMA Node memory: alloc from claimed node
end

Toolstack->>Xen: XEN_DOMCTL_claim_memory(0, NO_NODE)
Xen-->>Toolstack: Remaining claims released

    

Workflow diagram: Claiming and populating memory for a domain

API example (libxc)

The following example demonstrates how a toolstack can claim memory before building the domain and then releasing the claim once the memory population is complete.

Note: memory_claim_t contains padding to allow for future expansion. Thus, the structure must be zero-initialised to ensure forward compatibility. This can be achieved by using the XEN_NODE_CLAIM_INIT macro, which sets the pages and node fields while zero-initialising the padding of the structure, zero-initialising the entire structure, or by using a compound literal with designated initialisers to set the pages and node fields while zero-initialising the padding of the structure.

#include <xenctrl.h>

int claim_guest_memory(xc_interface *xch, uint32_t domid,
                       uint64_t pages)
{
    memory_claim_t claim[] = {
      /*
       * Example 1:
       * Uses the ``XEN_NODE_CLAIM_INIT`` macro to zero-initialise the padding
       * and set the pages and node fields for a NUMA-aware claim on node 0.
       */
      XEN_NODE_CLAIM_INIT(pages, 0)  /* Claim memory on NUMA node 0 */
    };

    /* Claim memory from NUMA node 0 for the domain build. */
    return xc_domain_claim_memory(xch, domid, 1, claim);
}

int release_claim(xc_interface *xch, uint32_t domid)
{
    memory_claim_t claim[] = {
      /*
       * Example 2:
       * Uses a compound literal with designated initialisers to set the
       * fields to release the claim while zero-initialising the rest
       * of the structure for forward compatibility.
       */
      (memory_claim_t){
        /*
         * pages == 0 releases any outstanding claim.
         * The node field is not used in this case, but must be set to
         * XEN_DOMCTL_CLAIM_MEMORY_NO_NODE for forward compatibility.
         */
        .pages = 0,
        .node = XEN_DOMCTL_CLAIM_MEMORY_NO_NODE,
      }
    };

    /* Release any remaining claim once population is done. */
    return xc_domain_claim_memory(xch, domid, 1, claim);
}