kpatch on RHEL — patching kernel CVEs without the reboot, and where it falls short

A new kernel CVE drops at 9am AEST. The advisory is CVSS 8.8, exploitation is plausible, your CISO wants it fixed today, and you have 600 RHEL hosts. The traditional answer is "schedule a maintenance window, drain traffic, reboot the fleet, drink coffee." The newer answer, on RHEL since 7.6, is "load the kpatch module and the kernel is fixed in 30 seconds with no reboot."

This post is what we've learned running kpatch on customer fleets since it became GA: when it works brilliantly, where it falls down, and why we still schedule reboots even when kpatch is available. We run managed RHEL for customers ranging from financial services with strict change windows to SaaS shops that want zero-downtime patching, and kpatch lives in both worlds.

What kpatch actually does

kpatch is Red Hat's implementation of live kernel patching. The pattern is:

1. Red Hat builds a patch module (an .ko) that contains the diff between the running kernel and a fixed version of one or more functions
2. You install the kpatch-patch-<kernel-version> RPM
3. The kpatch daemon loads the module at boot, the module hot-patches the in-memory kernel by redirecting function calls to the patched versions
4. The kernel itself is unchanged on disk — the on-disk kernel still has the vulnerable code, but the running kernel doesn't execute it

When you eventually reboot, the kpatch module unloads, and the unpatched kernel runs until the kpatch service re-applies the patch at boot. For this reason, the on-disk kernel also needs updating at the next maintenance window — kpatch buys you time, it doesn't replace dnf update kernel.

Subscription model — read this before you plan

kpatch is not included in standard RHEL subscriptions. You need either:

• RHEL for SAP Solutions — includes kpatch for the SAP kernel
• Extended Update Support (EUS) — kpatch is included for some minor versions
• Add-on: Live Patching for RHEL — buy it separately, attaches to standard subscriptions

For most customers we onboard, the add-on is the answer. It's roughly an extra $200-300 USD per socket per year (check current Red Hat pricing for your region; this changes). Whether that's worth it depends entirely on the cost of your reboot windows. For a customer whose change windows are once a quarter, kpatch pays for itself the first time a high-severity CVE lands.

We've also seen customers buy kpatch and then never use it because they didn't realise it's a separate subscription. Audit your entitlements before you assume it's available.

What you can patch, what you can't

The headline rule: kpatch can patch functions. It cannot patch:

• Data structure layouts. If a fix changes the size of struct task_struct, kpatch can't deliver it. Reboot required.
• __init functions. These run once at boot and the memory is freed. Nothing to patch.
• Inline functions that have been inlined into multiple call sites. Sometimes patchable with consistency model tricks, often not.
• The very lowest layers of the kernel. Memory management primitives, the scheduler core. These exist but Red Hat is conservative about live-patching them.

In practice, about 70% of kernel CVEs in any given quarter ship with a kpatch. The other 30% need a real reboot. The advisory text (RHSA-xxxx) will tell you which: look for "This advisory is also available as a kpatch."

How we deploy kpatch

For managed RHEL customers, kpatch is enabled by default for any host running production workloads. The setup:

# Verify subscription includes Live Patching
subscription-manager repos --list-enabled | grep kpatch
 
# Install kpatch tooling
dnf install kpatch
 
# Enable and start the service
systemctl enable --now kpatch
 
# Install the patch for the currently running kernel
dnf install "kpatch-patch-$(uname -r | sed 's/\.x86_64$//')"

That last line is the actually-clever bit. The RPM name is keyed to the exact kernel version running. Once installed, the patch is loaded immediately and persists across reboots (the systemd unit re-applies it on boot).

Verify it loaded:

kpatch list

Output should show the loaded patches and which CVEs they address. We add a kpatch list check to our standard monitoring — if a host's kpatch state drifts from what Satellite says it should be, we get paged.

The 30-second emergency patch

The procedure when a critical CVE drops and the kpatch is available:

# 1. On a canary host, install the patch
dnf install "kpatch-patch-$(uname -r | sed 's/\.x86_64$//')"
 
# 2. Confirm it loaded
kpatch list
 
# 3. Run smoke tests against the canary
 
# 4. Roll it out across the fleet via Satellite remote execution
hammer host-collection update-packages \
  --organization "ACME" \
  --name "prod-all-rhel9" \
  --packages "kpatch-patch-5_14_0-503_38_1"

End-to-end, for a fleet of 600 hosts, that's roughly 45 minutes from "CVE drops" to "every host is patched." With reboots, the same operation is a multi-day exercise. For customers with strict uptime requirements, this is the killer feature.

The gotchas we've actually hit

Patch module fails to load after a kernel update

Most common failure: you update the kernel (let's say from 5.14.0-503.38.1 to 5.14.0-503.39.1), but you didn't update the kpatch RPM. The new kernel boots without the patch, the kpatch service tries to load the old module against the new kernel, fails, and your host is now unpatched. Worse, kpatch's default behaviour in this case is to silently fail and continue booting.

The fix: tie the kpatch RPM update to the kernel update in your content view. We do this by including both in the same Satellite content view filter, so they always promote together.

Patches stack

If two CVEs are patched within the same kernel release, you'll have two kpatch-patch-* RPMs installed simultaneously. They're meant to coexist, but we've seen edge cases where the second one fails to load because it touches a function the first one already patched. Workaround: remove the older patch RPM when the newer one is installed, since the newer one is cumulative.

Live patches sometimes get withdrawn

This is rare but it happens. Red Hat ships a kpatch, then discovers a regression, and withdraws it. If you've already applied the kpatch, removing the RPM doesn't unload the running patch — you have to kpatch unload it explicitly. We've added a step to our emergency response runbook that checks for withdrawn patches as part of post-CVE cleanup.

kdump and live patching don't always coexist gracefully

If you take a kernel crash dump on a host with active kpatches, the dump analysis is harder because the in-memory kernel doesn't match the on-disk kernel symbols. Make sure your debuginfo packages match the kpatched state, not just the base kernel. We've spent hours chasing a crash where the stack trace was nonsense because we forgot this.

When we still reboot

Some categories of update will always require a real reboot, kpatch or not:

• Microcode updates. Spectre/Meltdown-class issues need CPU microcode, which is loaded at boot. kpatch can't help.
• Major kernel version bumps. Going from RHEL 9.4 to 9.5's kernel is not something kpatch supports — that's a kernel install, not a kernel patch.
• Data structure changes. As above.
• Quarterly kernel rollover. We schedule a real reboot once a quarter regardless, to consolidate accumulated kpatches, install the latest on-disk kernel, and verify the host actually comes up cleanly. A host that hasn't been rebooted in 18 months is a host that's about to surprise you.

That last point is underrated. kpatch can extend uptime, but uptime is not a virtue in itself. We've seen hosts with 800 days of uptime that turned out to have stale init scripts, expired systemd units, and a /boot partition full of unused kernels. The reboot is also a test.

kpatch vs Ubuntu Livepatch — the real comparison

For customers running both, we get asked how kpatch compares to Canonical's Livepatch. The short version:

Both work. Both are production-ready. The choice is downstream of your distro choice, not the other way around. We've never had a customer pick RHEL specifically because of kpatch, nor pick Ubuntu specifically because of Livepatch.

What we ship by default

Every managed RHEL host under our cybersecurity-managed plan gets:

• kpatch installed and enabled, with the patch RPM for the current kernel
• Monitoring on kpatch list output — alerts if the expected patches aren't loaded
• Automated kpatch RPM updates tied to Satellite content view promotion
• A scheduled quarterly reboot to consolidate kpatches and verify cold-boot health
• A documented emergency CVE patching runbook with kpatch as the primary lever

If you're running RHEL in production and reboots are expensive, kpatch is genuinely transformative — but only if your subscription includes it and your patching pipeline knows about it. If neither of those is true, you're carrying unnecessary CVE exposure between maintenance windows.

Sudhanshu K. is a Senior Linux engineer at EdgeServers (RemotIQ Pty Ltd, ABN 91 682 628 128). She has been deploying RHEL kernels since the 2.6 days and has strong opinions about which kernel parameters belong in /etc/sysctl.d/.