0

I was tasked to setup automatic updates on some high-performance RHEL8 workstations & servers.
I don't like it, as a lot of things can go wrong with unattended updates, but well, it seems like provoking bugs and data losses is "safer" than having computers not updated at least weekly...

The target computers use zfs and kmod-nvidia (from zfsonlinux.org and elrepo.org respectively); both RPMs need a matching kernel "version", so the first safety measure I would like to implement is to ensure that when a new kernel is available then the installed zfs and kmod-nvidia (or their available update) are compatible with it.

For eg. just now there's a new kernel:

$ sudo dnf update --assumeno
Updating Subscription Management repositories.
Last metadata expiration check: 0:57:59 ago on Thu 28 Nov 2024 12:08:30 AM CET.
Dependencies resolved.
==========================================================================================================================================================
 Package                           Architecture           Version                                  Repository                                        Size
==========================================================================================================================================================
Installing:
 kernel                            x86_64                 4.18.0-553.30.1.el8_10                   rhel-8-for-x86_64-baseos-rpms                     10 M
Upgrading:
 bpftool                           x86_64                 4.18.0-553.30.1.el8_10                   rhel-8-for-x86_64-baseos-rpms                     11 M
 kernel-headers                    x86_64                 4.18.0-553.30.1.el8_10                   rhel-8-for-x86_64-baseos-rpms                     12 M
 kernel-tools                      x86_64                 4.18.0-553.30.1.el8_10                   rhel-8-for-x86_64-baseos-rpms                     11 M
 kernel-tools-libs                 x86_64                 4.18.0-553.30.1.el8_10                   rhel-8-for-x86_64-baseos-rpms                     10 M
 pam                               x86_64                 1.3.1-36.el8_10                          rhel-8-for-x86_64-baseos-rpms                    748 k
 python3-perf                      x86_64                 4.18.0-553.30.1.el8_10                   rhel-8-for-x86_64-baseos-rpms                     11 M
 webkit2gtk3                       x86_64                 2.46.3-2.el8_10                          rhel-8-for-x86_64-appstream-rpms                  28 M
 webkit2gtk3-jsc                   x86_64                 2.46.3-2.el8_10                          rhel-8-for-x86_64-appstream-rpms                 4.5 M
Installing dependencies:
 kernel-core                       x86_64                 4.18.0-553.30.1.el8_10                   rhel-8-for-x86_64-baseos-rpms                     43 M
 kernel-devel                      x86_64                 4.18.0-553.30.1.el8_10                   rhel-8-for-x86_64-baseos-rpms                     24 M
 kernel-modules                    x86_64                 4.18.0-553.30.1.el8_10                   rhel-8-for-x86_64-baseos-rpms                     36 M

Transaction Summary
==========================================================================================================================================================
Install  4 Packages
Upgrade  8 Packages

Total download size: 202 M

The installed kmod-nvidia seams compatible, as both the new kernel and the NVidia driver start with 4.18.0-553:

$ sudo dnf repoquery --requires kmod-nvidia | awk '$1 == "kernel"'
kernel >= 4.18.0-553.el8_10

But I don't know what to say about zfs, as there's no mention of any kernel requirement in its RPM:

$ sudo dnf repoquery --requires zfs
Updating Subscription Management repositories.
/bin/sh
/usr/bin/python3.6
libblkid.so.1()(64bit)
libblkid.so.1(BLKID_1.0)(64bit)
libc.so.6(GLIBC_2.28)(64bit)
libcrypto.so.1.1()(64bit)
libdl.so.2()(64bit)
libm.so.6()(64bit)
libm.so.6(GLIBC_2.2.5)(64bit)
libnvpair.so.3()(64bit)
libnvpair3(x86-64) = 2.1.15-1.el8
libnvpair3(x86-64) = 2.1.15-2.el8
libpthread.so.0()(64bit)
libpthread.so.0(GLIBC_2.12)(64bit)
libpthread.so.0(GLIBC_2.2.5)(64bit)
libpthread.so.0(GLIBC_2.3.2)(64bit)
librt.so.1()(64bit)
librt.so.1(GLIBC_2.3.3)(64bit)
libtirpc.so.3()(64bit)
libudev.so.1()(64bit)
libudev.so.1(LIBUDEV_183)(64bit)
libuuid.so.1()(64bit)
libuuid.so.1(UUID_1.0)(64bit)
libuutil.so.3()(64bit)
libuutil3(x86-64) = 2.1.15-1.el8
libuutil3(x86-64) = 2.1.15-2.el8
libz.so.1()(64bit)
libzfs.so.4()(64bit)
libzfs5(x86-64) = 2.1.15-1.el8
libzfs5(x86-64) = 2.1.15-2.el8
libzfs_core.so.3()(64bit)
libzpool.so.5()(64bit)
libzpool5(x86-64) = 2.1.15-1.el8
libzpool5(x86-64) = 2.1.15-2.el8
openssl
rtld(GNU_HASH)
sysstat
systemd
util-linux
zfs-kmod = 2.1.15

How can I verify that the installed (or available update of) zfs is compatible with the new kernel?

Improve this question

1 Answer 1

Reset to default
4

The kernel dependencies are not specified in the main zfs package, but in zfs-dkms or kmod-zfs packages respectively, depending on which variant repository you're using.

From OpenZFS documentation at openzfs.github.io:

By default the zfs-release package is configured to install DKMS style packages so they will work with a wide range of kernels. In order to install the kABI-tracking kmods the default repository must be switched from zfs to zfs-kmod. Keep in mind that the kABI-tracking kmods are only verified to work with the distribution-provided, non-Stream kernel.

dnf config-manager --disable zfs
dnf config-manager --enable zfs-kmod
dnf install zfs

Since your list of dependencies for zfs already includes zfs-kmod, it seems you are already doing this.

However, unlike kmod-nvidia, the kmod-zfs package available in that variant repository tracks kernel dependencies at the kernel symbol level, not at the kernel package version level:

rpm -q --requires kmod-zfs-2.1.15-2.el8.x86_64.rpm

kmod-zfs-2.1.15-2.el8.x86_64
/bin/sh
/bin/sh
/bin/sh
/usr/sbin/depmod
/usr/sbin/depmod
/usr/sbin/weak-modules
/usr/sbin/weak-modules
kernel(PDE_DATA) = 0xea2257cf
kernel(__alloc_disk_node) = 0x75d19f57
kernel(__alloc_pages_nodemask) = 0x8ab32967
kernel(__alloc_percpu) = 0x949f7342
kernel(__bitmap_weight) = 0x63c4d61f
kernel(__check_object_size) = 0x88db9f48

... <quite a lot of kernel symbol dependencies> ...

kernel(zlib_inflateEnd) = 0x107e5878
kernel(zlib_inflateInit2) = 0x35a88f28
kernel(zlib_inflate_workspacesize) = 0xce5ac24f
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(FileDigests) <= 4.6.0-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rpmlib(PayloadIsXz) <= 5.2-1
zfs = 2.1.15

On the other hand, the zfs-dkms package specifies its dependencies relative to the kernel-devel package, and can accept a wider range of kernel versions:

rpm -q --requires zfs-dkms*.rpm

zfs-dkms-2.1.15-2.el8.noarch
/bin/sh
/bin/sh
/bin/sh
diffutils
diffutils
dkms >= 2.2.0.3
dkms >= 2.2.0.3
dkms >= 2.2.0.3
dkms >= 2.2.0.3
gcc
gcc
kernel-devel >= 3.10
kernel-devel >= 3.10
kernel-devel <= 6.7.999
kernel-devel <= 6.7.999
make
make
perl
perl
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(FileDigests) <= 4.6.0-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rpmlib(PayloadIsXz) <= 5.2-1

With DKMS, the kernel module (re)building is triggered by the kernel package post-install phase before the bootloader update step, and if it fails, it should be treated as a kernel package installation failure. In other words, if DKMS cannot provide a compatible kernel module for the new kernel, the installation of the new kernel won't reach completion.

With kmod-zfs, the kernel symbol dependencies should block a non-compatible kernel RPM from installing until the ZFS can also be updated to match the new kernel.

In other words, you have an equivalent of the safety measure you're envisioned already in place.

Improve this answer
4
  • Thank you for the great answer; zfs-kmod is listed in the dependencies of zfs but zfs-dkms is the one in use. If a kernel update fails because of zfs-dkms incompatibility, won't the RPM DB be a somewhat broken state? Commented Nov 28, 2024 at 9:47
  • I don't currently have a RPM-based system with DKMS where I could test this situation and it's been too long since I encountered it for real, but the package manager should generally treat the new zfs-incompatible kernel as a package that failed installation/post-install steps, and automatically roll it back and retry at the time of the next update. If it causes the RPM database to break, something else is wrong. Commented Nov 28, 2024 at 10:14
  • I haven't observed this behaviour on RHEL 9. If a newly installed kernel is not compatible with the kABI expected by the currently-installed zfs-kmod package, nothing stops kernel from being upgraded. Rather, the symlink in /lib/modules/KERNELVER/weak-updates/zfs/zfs/zfs.ko simply isn't created. Once you upgrade kmod-zfs` to a version that is compatible with the newer kernel's kABI, the symlinks in the older kernels' weak-updates directories get removed, and the symlink in the newer kernels' weak-updates directories get created. Commented Apr 17 at 16:21
  • [continued] I'm looking into writing a script that monitors whether, for each directory immediately present in /lib/modules, there is a non-broken zfs.ko symlink somewhere within the directory tree; if not then raise an alert. Commented Apr 17 at 16:26

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.