Preserving HugeTLB Memory During Live Kernel Updates: A Step-by-Step Guide

Introduction

Live kernel updates—such as those enabled by the kexec handover and live update orchestrator features in Linux—allow system administrators to apply critical patches without rebooting. However, preserving large memory allocations provided by hugetlbfs (HugeTLB) during this process has been a challenging task. Recent developments, discussed at the 2026 Linux Storage, Filesystem, Memory Management, and BPF Summit (LSFMM+BPF) in a session led by Pratyush Yadav, focus on adding this capability. While the work is still ongoing, this guide outlines a conceptual approach to preserving HugeTLB pages across a live update, based on current kernel enhancements. Follow these steps to achieve minimal disruption while maintaining your large-page workloads.

What You Need

• A Linux kernel compiled with the following options enabled:
  • CONFIG_HUGETLBFS
  • CONFIG_KEXEC
  • CONFIG_LIVE_UPDATE_KEXEC_HANDOVER (or similar, depending on version)
• Root or sudo access on the target system.
• Tools: kexec-tools, hugetlbfs-aware utilities (e.g., libhugetlbfs), and a compatible bootloader.
• A test environment to validate before deploying to production.

Step-by-Step Instructions

Step 1: Verify Kernel and HugeTLB Support

Before attempting a live update, ensure your running kernel supports both hugetlbfs and the live update mechanisms. Check with:

cat /proc/cmdline | grep hugepages
ls /sys/kernel/kexec/  # should show handover-related files

If the hugepages kernel parameter is missing, add it and reboot. For live update orchestrator support, consult your distribution’s kernel configuration or enable CONFIG_LIVE_UPDATE when building. Without these, HugeTLB pages will not be preserved automatically.

Step 2: Mount hugetlbfs and Allocate Huge Pages

Mount the hugetlbfs filesystem to make HugeTLB pages available to applications:

mkdir -p /mnt/huge
mount -t hugetlbfs hugetlbfs /mnt/huge

Then reserve a number of huge pages (e.g., 2 MB each) via sysfs:

echo 10 > /proc/sys/vm/nr_hugepages

Check allocation: grep HugePages /proc/meminfo. The pages you allocate must be free (not in use) or explicitly marked for preservation. For this guide, we assume the live update orchestrator will attempt to preserve all free huge pages and any pages held by hugetlbfs files.

Step 3: Configure Live Update Parameters for HugeTLB Preservation

Set the kernel command line for the new kernel by passing the hugepages=preserve option (or a similar flag) to the kexec handover. For example, when preparing the kernel to be loaded:

kexec -l /boot/vmlinuz-new --initrd=/boot/initrd.img-new --reuse-cmdline --append="hugepages=preserve"

Some live update orchestrators provide a configuration file; add hugetlbfs_preserve=true. Ensure the new kernel also has hugetlbfs enabled and the same huge page size. If the new kernel lacks support, preservation will fail.

Step 4: Initiate the Live Update (kexec Handover)

Trigger the live update using the kexec system call or an orchestrator tool:

kexec -e   # or use a live update daemon

During this handover, the kernel saves a memory map of HugeTLB pages that are eligible for preservation. The new kernel, upon booting, should read this map and remap those pages without touching them, preserving the contents. Monitor the console or logs for messages like “hugetlbfs: preserved N pages”. If preservation fails, fall back to a full reboot.

Step 5: Verify Preservation After Update

Once the new kernel is running, confirm that your huge pages were retained:

• Check /proc/meminfo for HugePages_Free and HugePages_Rsvd. They should reflect the same counts as before the update.
• Inspect any applications that use hugetlbfs memory (e.g., databases, HPC workloads) to ensure they continue to operate.
• Run cat /sys/kernel/debug/hugetlbfs/preserved (if available) to see preserved page details.

If preservation succeeded, your live update maintained HugeTLB memory without data loss. If pages are missing, review logs and consider the tips below.

Step 6: Troubleshoot Common Issues

Preservation can fail due to:

• Incompatible kernel versions (e.g., one uses 2 MB pages, the other 1 GB).
• Pages that are pinned or in use by other mechanisms (like VFIO). The orchestrator may skip active mappings.
• Insufficient contiguous memory for the memory map.

To debug, enable verbose logging with kexec -d or check dmesg after the handover. If preservation fails repeatedly, revert to a full reboot with the new kernel and reconfigure huge pages manually.

Tips for Success

• Test in a non-production environment first, using the exact kernel versions and configuration you plan to use in production.
• Keep a fallback boot entry in case the live update fails; this ensures you can still boot the old kernel.
• Monitor memory pressure: preserving huge pages during a live update may temporarily increase memory fragmentation. Use tools like numactl to verify NUMA node balance.
• Stay updated with kernel patches and the live update orchestrator project. The feature is still under development (as of 2026), and fixes are being merged regularly.
• Document your huge page reservations so you can easily reconstruct them after a planned reboot.