libvirt error when enabling hugepages for guest

Per the libvirt documentation, in this section titled: Memory Backing.

The optional memoryBacking element may contain several elements that influence how virtual memory pages are backed by host pages.

hugepages

This tells the hypervisor that the guest should have its memory allocated using hugepages instead of the normal native page size. Since 1.2.5 it's possible to set hugepages more specifically per numa node. The page element is introduced. It has one compulsory attribute size which specifies which hugepages should be used (especially useful on systems supporting hugepages of different sizes). The default unit for the size attribute is kilobytes (multiplier of 1024). If you want to use different unit, use optional unit attribute. For systems with NUMA, the optional nodeset attribute may come handy as it ties given guest's NUMA nodes to certain hugepage sizes. From the example snippet, one gigabyte hugepages are used for every NUMA node except node number four. For the correct syntax see this.

nosharepages

Instructs hypervisor to disable shared pages (memory merge, KSM) for this domain. Since 1.0.6

locked

When set and supported by the hypervisor, memory pages belonging to the domain will be locked in host's memory and the host will not be allowed to swap them out. For QEMU/KVM this requires hard_limit memory tuning element to be used and set to the maximum memory configured for the domain plus any memory consumed by the QEMU process itself. Since 1.0.6

Example

<domain>
  ...
  <memoryBacking>
    <hugepages>
      <page size="1" unit="G" nodeset="0-3,5"/>
      <page size="2" unit="M" nodeset="4"/>
    </hugepages>
    <nosharepages/>
    <locked/>
  </memoryBacking>
  ...
</domain>

Since you do not say, I'm assuming you want to allocate all the memory to this particular guest. If so you could probably try omitting this section completely.

Alternative directions

I found these RHEL 5 & 6 steps in this article titled: How do I set up KVM guests to use HugePages? that show how to set this up as follows:

excerpt

Mount the HugeTLB filesystem on the host

You may use any mountpoint desired, here we have created /hugepages

  mkdir -p /hugepages
  mount -t hugetlbfs hugetlbfs /hugepages

This is also possible via an entry in /etc/fstab, for example

  hugetlbfs    /hugepages    hugetlbfs    defaults    0 0

Increase the memory lock limit on the host

Alter the following values in /etc/security/limits.conf depending on your required memory usage

  # Lock max 8Gb
  soft memlock 8388608
  hard memlock 8388608

Reserve HugePages and give the KVM group access to them

Alter to following lines in /etc/sysctl.conf depending on your required memory usage

  vm.nr_hugepages = 4096
  vm.hugetlb_shm_group = 36

Add HugePage backing to the KVM guest definition

Add the following to the guest config of an existing KVM guest. This can be done with virsh edit <guestname> or virsh define <guest.xml>

  <memoryBacking>
      <hugepages/>
  </memoryBacking>

Restart the host

This is required to re-allocate contigous memory to HugePages

Start a guest

Confirm the guest has HugePage backing Check the qemu-kvm process associated with that guest for the presence of -mem-path in the run command

  ps -ef | grep qemu

  root      4182     1  1 17:35 ?        00:00:42 /usr/libexec/qemu-kvm -S -M rhel5.4.0 -m 1024 -mem-prealloc
-mem-path /hugepages/libvirt/qemu -smp 1 -name vm1 -uuid 3f1f3a98-89f8-19ac-b5b5-bf496e2ed9be -no-kvm-pit-reinjection
-monitor pty -pidfile /var/run/libvirt/qemu//vm1.pid -boot c -drive file=/vmimages/vm1,if=ide,index=0,boot=on,cache=none
-drive file=,if=ide,media=cdrom,index=2 -net nic,macaddr=54:52:00:00:00:01,vlan=0 -net tap,fd=15,script=,vlan=0,ifname=vnet0
-serial pty -parallel none -usb -vnc 127.0.0.1:0 -k en-us

Confirm HugePage use on the system

Here we can see HugePages are being allocated at startup, as well as used/reserved for the guests

  cat /proc/meminfo | grep Huge

  HugePages_Total:    4096
  HugePages_Free:      873
  HugePages_Rsvd:      761
  Hugepagesize:       2048 kB

Root Cause

The default method of allocating memory for KVM guests is to use regular 4k pages. This can result in

  • large page tables which occupy unnecessary and inefficient amounts of memory
  • increased memory fragmentation which can slow down some kernel-based actions which require contigous memory (eg: disk writes, network access)
  • increasing page faults which can slow down all applications
  • risking swapping components of virtual guests out to disk which would cause a large performance hit

Using HugePages, page table sizes are dramatically reduced, contigous areas of memory are mapped, and HugePages cannot be swapped by design.

Note: These steps are not necessary with KVM on RHEL6, which uses Transparent HugePages to dynamically map contigous 2Mb areas of memory but also allows that memory to be broken up into 4k pages to be merged with KSM or swapped when the system is under memory pressure.

The above steps can be applied to RHEL6 if HugePages are desired over Transparent HugePages.


Answering my own question, in order to use hugepages with libvirt in Ubuntu, you just have to set KVM_HUGEPAGES=1 in the file /etc/default/qemu-kvm and restart.

This is related with this bugfix.

Tags:

Kvm

Recent Posts