First of all, most of what comes up in search on the web has been deprecated. For example cgmanager is no longer supported on new systemd versions. Don't follow 99% of what comes up in web serches as far as using cpulimit, nice, cgset or other tools for this job. They either won't work at all as advertised (as in the case of cgroup management tools that expect you to create your own hierarchy) or won't get the job done without resorting to lots of hacks (as in the case of using 'nice' levels to manage whole groups of processes).
The good news is that along with those deprecations (and pursuing the traditional all-devouring octopus monster modus operandi of systemd) a default configuration is in place for everything on the system, and tweaking it for systemd services is trivial. Just add an overlay configuration to the service you want to limit:
$ sudo systemctl edit <servicename>
Add a section with whatever resource control values you want to override. In my case I came up with this:
[Service]
CPUWeight=20
CPUQuota=85%
IOWeight=20
MemorySwapMax=0
These values are not all necessary, but the first two answer the question as asked:
CPUWeight defaults to 100 for all processes on the system. Setting a low value still lets process use the CPU if nothing else is effectively keeping the system responsive for other tasks while not slowing down the results much. This is a arbitrary weight integer.CPUQuota is an absolute limit on how much CPU time is granted even if nothing else is going on. This is a percent value. In my case it wasn't really necessary to set this to fix the resource hogging issue. I ended up setting it anyway to keep the CPU temperature down when lots of CI jobs pile up.IOWeight is much the same as CPUWeight, in this case used to keep disks free for system tasks and only keep them busy with CI jobs when nothing else is going on.MemorySwapMax also isn't in scope for the question, in my case I ended up adding it because the ray traver (povray) running in some of the CI jobs seems to think using 30+ Gigs of swap in addition to the 30+ Gigs of RAM in this system is good idea just because it is there. It runs faster if you don't let it use it at all. This is probably something better configured in povray, but this way I don't have to police what happens inside of CI jobs and don't have to disable the system swap.
Lastly these values can be changed on the fly without restarting services by running systemctl daemon-reload. This is quite handy to watch the effect of the changes right away.
