[RFC] Add target module check before livepatch module loading

[wardenjohn]

I had faced a scenario like this:
There is a fuse.ko which is built as module of kernel source. However, our team maintain the fuse as oot module.
There is a bug of (name it as B1) the original fuse.ko. And our team fix B1 of fuse.ko as release it as oot module fuse_o1.ko.

Our system loaded fuse_o1.ko. Now, another team made a livepatch module base on fuse.ko to fix B2 bug.
They load this livepatch_fuse.ko to the system, it fixed B2 bug, however, the livepatch_fuse.ko revert the fix of fuse_o1.ko.
It expose the B1 bug which is already fix in fuse_o1.ko

The exposed B1 bug make fault to our cluster, which is a bad thing :(

I have a solution to handle this:

1. In kpatch-build, we would record the patched object, take the object of ko as a list of parameters.
2. Pass this ko list as parameter to create-klp-moudle.c
3. For each patched ko object, we should read its srcversion from the original module. If we use --oot-module, we would read the srcversion from the oot moudle version.
4. Store the target srcversion to a section named '.klp.target_srcversions'
5. When the kpatch module loading, we shoud check if section '.klp.target_srcversion' existed. If existed, we should check srcversion of the patch target in the system match our recorded srcversion or not. If thet are not match, refuse to load it. This can make sure the livepatch module would not load the wrong target.

This function can avoid livepatch from patching the wrong version of the function.

I am here waiting for Request For Comment. Before I do codes.

Thanks~~ ;)

Wardenjohn.

[joe-lawrence]

Hi @wardenjohn - in this scenario, I don't understand how livepatch-fuse.ko was ever activated if the system was loaded with the OOT fuse_o1.ko. Or is the OOT module also (confusingly) named "fuse"?

Also, are the module structures or other means to inspect srcversion exported? This might be something only the kernel livepatching core could and should do.

[wardenjohn]

@joe-lawrence
Right, they are both fuse but with different versions.
fuse_o1.ko is just here to tell you they are different object. In fact, all the module is named as fuse.ko. As you know, the code of fuse is in-tree code, but it can built as module with obj-m. Our team separated the code and maintain it in other repo.

The original fuse ko lie in /lib/modules/5.10.134-17.al8.x86_64/kernel
And the newer version of fuse lie in /lib/modules/5.10.134-17.al8.x86_64/updates

When the system start to run, if /lib/modules/5.10.134-17.al8.x86_64/updates exist, the module in /lib/modules/5.10.134-17.al8.x86_64/updates will load first.
In our scenario, we loaded module in /lib/modules/5.10.134-17.al8.x86_64/updates. But the livepatch module is built from the in-tree code. So, the livepatch module revert the fix in the newer version from /lib/modules/5.10.134-17.al8.x86_64/updates

Also, are the module structures or other means to inspect srcversion exported? This might be something only the kernel livepatching core could and should do.

I think we can read the structure the struct module which contains the srcversion inside.
Maybe we can read the srcversion from the ko file?

[joe-lawrence]

I think we can read the structure the struct module which contains the srcversion inside.

I couldn't find an exported API that would return the struct module for a given module name. If there is one, then you could try prototyping this with a pre-patch handler that failed out on a srcversion comparison.

Operationally, this scenario seems like a nightmare if multiple teams are creating their own livepatches -- who knows what the latest livepatch is for any given kernel/module/function? This RFC would help lock out the possibility, but essential force the teams to coordinate to avoid the situation in the first place :)

[wardenjohn]

I couldn't find an exported API that would return the struct module for a given module name. If there is one, then you could try prototyping this with a pre-patch handler that failed out on a srcversion comparison.

So I am now finding a way to get the srcversion from a module. I think read it from elf file may be an option.

Operationally, this scenario seems like a nightmare if multiple teams are creating their own livepatches -- who knows what the latest livepatch is for any given kernel/module/function? This RFC would help lock out the possibility, but essential force the teams to coordinate to avoid the situation in the first place :)

True. Because each team is actually responsible for relatively isolated business, if everyone uses the livepatch technology, this situation will occur. However, it is unreliable to expect everyone to consciously abide by an agreement. So we need to solve this problem technically.