How to isolate cpu cores, even from kernel space, at boot?

I have a faulty Ryzen 5900X desktop CPU. Previously, I somewhat tamed its faulty cores via isolcpus=2,3,14,15 kernel parameter in GRUB2 (see https://blog.cbugk.com/post/ryzen-5850x/).

However, on Proxmox 8.2, I have set up a CEPH cluster. It had crippling performance of around 2 MB/s. Redone the cluster got 20 MB/s speed while cloning a template VM. I was suspecting my use of second-hand enterprise SSDs but even fresh ones did it (with or without NVMe DB cache).

But, when I checked my faulty cores (2,3,14,15) they were being used. The moment I turn down the computer with 5900X, transfer speed jumps to around 100 MB/s on the remaining two nodes. Networking is 10G between each-node, iperf previously had shown 6G throughput, ~~it cannot be the bottle-neck.~~ It was the damn cabling.

Some duckduckgo-ing later found out isolcpus= works for user space but not for kernel space.

watch -n1 -- "ps -axo psr,pcpu,uid,user,pid,tid,args --sort=psr | grep -e '^ 2 ' -e '^ 3 ' -e '^ 14 ' -e '^ 15'" (source) gives:

  2  0.0     0 root          27      27 [cpuhp/2]
  2  0.0     0 root          28      28 [idle_inject/2]
  2  0.3     0 root          29      29 [migration/2]
  2  0.0     0 root          30      30 [ksoftirqd/2]
  2  0.0     0 root          31      31 [kworker/2:0-events]
  2  0.0     0 root         192     192 [irq/26-AMD-Vi]
  2  0.0     0 root         202     202 [kworker/2:1-events]
  3  0.0     0 root          33      33 [cpuhp/3]
  3  0.0     0 root          34      34 [idle_inject/3]
  3  0.3     0 root          35      35 [migration/3]
  3  0.0     0 root          36      36 [ksoftirqd/3]
  3  0.0     0 root          37      37 [kworker/3:0-events]
  3  0.0     0 root         203     203 [kworker/3:1-events]
 14  0.0     0 root          99      99 [cpuhp/14]
 14  0.0     0 root         100     100 [idle_inject/14]
 14  0.3     0 root         101     101 [migration/14]
 14  0.0     0 root         102     102 [ksoftirqd/14]
 14  0.0     0 root         103     103 [kworker/14:0-events]
 14  0.0     0 root         210     210 [kworker/14:1-events]
 15  0.0     0 root         105     105 [cpuhp/15]
 15  0.0     0 root         106     106 [idle_inject/15]
 15  0.3     0 root         107     107 [migration/15]
 15  0.0     0 root         108     108 [ksoftirqd/15]
 15  0.0     0 root         109     109 [kworker/15:0-events]
 15  0.0     0 root         211     211 [kworker/15:1-events]

Since Ceph uses kernel driver, I need a way to isolate cores from the whole system. Running PID 1 on-wards in a taskset is okay. I cannot use cset due to cgroups2. numactl is also okay.

With isolcpus I do not have apparent system stability issues, without that I would face secure connection errors on Firefox and OS installs would fail. But even that is not enough when using CEPH. And now I conclude that it could corrupt data unnoticed if this wasn't my homelab machine.

Can anyone suggest a way to effectively ban these faulty threads as soon as system allows to do so, permanently? (I better use the phrase CPU affinity in the post)

I was wrong, redone Cat6 cables just the right length, having cleared power cables earlier I can state intererence should be quite lower than earlier. The same error was there when I disabled half the cores on BIOS including the faulty ones. I get instant VM clones on CEPH pool now, thanks to nvme DB cache I suppose.

Also, the kernel threads on the cores are the ones used for scheduling processes, their PID and set of threads on those cores is constant with above watch command even during a VM clone on CEPH pool. So if no tasks are being scheduled, it might be working as intended.

Found these tangentially relevant readings interesting: migration - reddit, nohz - lwn.net