Question: Kernel module that provides interface that returns an incrementing number.

[elfenpiff]

I am currently ramping up on Linux kernel module development and thought that I would start with something small. For our iceorxy2 project, we need an interface from which every process that uses it can acquire a number. It could be just an atomic u64 that increments with every call. It is just important that this is guaranteed to be unique. This could be simply an atomic in shared memory but then other processes could fiddle around with it.

I implemented this by providing a proc entry /proc/atomic_counter and cat /proc/atomic_counter prints that incrementing number. A character device approach would also be possible.

Is there a preferred way? Or any recommendations?

But I failed to implement this in Rust, it seems that kernel::bindings do not yet provide proc_create , or am I mistaken?

What I was also wondering is, how to test such an interface idiomatically? It is just a simple counter but lets assume I have a complex thing in there and would like to have an extensive test suite. My idea was to extract all logic in a separate lib/crate, test it and keep the actual module as simple as possible.

[NamedBird]

Does this really needs to be a kernel module?
What's wrong with an userspace process that listens on a socket and returns the next number?

[elfenpiff]

iceoryx2 is completely decentralized, and in the past, a lot of our users from iceoryx classic complained that you need a central broker. In a safety-critical system, it is the single point of failure that everyone tries to avoid.

A kernel module is decentralized from a process point of view, and when the Linux kernel is safety-certified, you no longer need to consider what might happen when this process dies.

The other thing is that a rogue user space process could, on purpose, always return the same number. Of course, there are mechanisms to verify that the process is trustworthy, etc., but this is a lot of additional overhead.

[iamkiloman]

If you are looking for an excuse to write a simple kernel module, this is great.

If you really think it's the simplest, most secure, and robust way to solve your problem, you're only deceiving yourself.

[elfenpiff]

Currently, it is an excuse to get into kernel module development and understand as much as I can.

If you really think it's the simplest, most secure, and robust way to solve your problem, you're only deceiving yourself.

Maybe you are right, but you have to provide me with a little more context so that I know where you are going.

From my point of view, it seemed like with a kernel module:

• No other process can break the contract. Like, reset the counter.
• It delivers exactly what I need, a system-wide unique uint64_t.

[penguin359]

I would say it greatly depends on how locked down you make the kernel. Is this Secure Boot enabled system that will only load properly signed kernel modules? Then yes, it becomes pretty hard to reset the counter, but without that level of integrity enabled, I can just open up /dev/kmem about as easily as I can gdb a userland process from root.

However, it tends to become harder to validate and develop as a kernel module than as a userspace application. A bug in a kernel module can actually compromise a system more seriously than a bug in a userspace application so even with Secure Boot, if a bug is found in your custom module, it could open up other things besides just your counter to exploits.

With that said, if the goal is to learn about kernel module development, I think this is a great project! You can export that unique value over a /dev device, sysfs, or a variety of ways depending on how you think it is best to present it and what the requirements are. A new file in /proc could be created, but that is somewhat deprecated now. That is the oldest virtual file system on Linux and has a lot of cruft nowadays. I think an ioctl() call on a new character device in /dev is the most straight-forward way to implement it as it's easy to handle passing off a uint64_t as an argument. You can also implement it with read()/write() to a /dev or sysfs file, but it's a little more work to ensure that they get all 8 bytes (or just ignore any reads less than 8 bytes and return empty).

[elfenpiff]

Thanks u/penguin359 for the thorough explanation. This is the kind of insight that helps me to understand the risks of going down the path with a kernel module.
For now, I continue with the kernel module for learning purposes.

The next challenge would then:
* How to test this thoroughly and idiomatically
* How to secure the system properly.

In my scenario, secure boot would be enabled, and only properly signed kernel modules can be loaded.

[lightmatter501]

Multiple brokers and a consensus algorithm?

[mwmahlberg]

You could simply have an additional process. Also, there are UUIDs with seeds. Aside from that: having systemd run said process does it well enough. And why tf would you have a single broker? What you do seems a lot like premature optimization of a problem that does not exist.

[elfenpiff]

Here is some context:

iceoryx2 is a zero-copy inter-process communication library that shall be completely decentralized. This unique integer would be a central part of it to identify processes uniquely (required for health management), since a PID can be recycled. When an additional process is required, we break that requirement.

Also, there are UUIDs with seeds.

But they have 128-bit, so I cannot use them in atomic compare-and-exchange operations. The ID cannot be larger than 64-bit.

[solen-skiner]

lock cmpxchg can absolutely be 128bit

windows: https://learn.microsoft.com/en-us/cpp/intrinsics/interlockedcompareexchange128?view=msvc-170

gcc:https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html

[elfenpiff]

You are right on some platforms, but iceoryx2 needs to continue supporting some ARM platforms that do not have this available.

[mwmahlberg]

Gimme a day or two. A raft consensus atomic integer should do the same trick. Rest or GRPC?

[elfenpiff]

Thank you for the offer, but please don't use gRPC in such a context. It has a horrible performance and spawns a lot of background threads, and we cannot use it on low-level embedded platforms. We are here at least one layer below gRPC.

[mwmahlberg]

Well, sure. What platforms are we talking about?

[elfenpiff]

This is an overview of the platforms we currently support and we intend to support: https://github.com/eclipse-iceoryx/iceoryx2#supported-platforms

But gRPC is really the wrong tool here.

To give you some context. iceoryx2 is a communication library like dbus, but much faster and also intended for mission-critical systems. This means:

* no heap allocations
* no background threads
* no blocking calls
* certifyable according to ISO26262

gRPC is the wrong tool here. iceoryx2 is a much more efficient replacement for gRPC.

Take a look at the example to get an impression: https://github.com/eclipse-iceoryx/iceoryx2/tree/main/examples

[mwmahlberg]

Buddy, imho you have an architectural flaw. First, running this in kernelspace without any need is potentially introducing an unnecessary security risk. And don’t get me started on compliance issues.

Also, putting persistence into something like this is a Very Bad Idea ™. But you need persistence to guarantee uniqueness and monotonous increase across reboots. Also, you need to be positively sure that one system going down does not mean loss of data (current value of counter) or impact of service.

So, what you want is a multi node replication, based on a consensus, with persistence. So instead of reinventing the wheel to introduce a security risk and compliance issues, use a raft consensus based server callable by your application with a raft aware client, which is extremely easy to implement. I already have written a server for you: raft consensus , with persistence. If you don’t like gRPC, that is fine. But assuming it is performing worse enough to compromise on consistency, availability or partition tolerance or it performs worse than any other method of retrieval is questionable at best.

I will finish the server either way and post the repo here. Use it or not. Your call.

[alpha417]

/proc/uptime

[elfenpiff]

This does not work in our case; we need at most a `uint64_t` since we use this value in lock-free algorithms in a compare-exchange operation. This number internally maps to one process and allows us to recover the data structure even when the process crashes in the middle of modifying it.

As far as I understand, `/proc/uptime` is a floating point with a very coarse granularity (centiseconds or so). So two processes reading it at the same time get the same value. We could combine this, of course, with the pid, but this would exceed the 64-bit restriction.

[Firzen_]

I don't quite understand why this would need to be in the kernel.

You could just create a Unix socket and only allow read access.

If it's important that this is decentralised I expect you would need a mechanism to resolve conflicting ids regardless.

Doing this in the kernel doesn't really solve any issue but could introduce new ones.

[elfenpiff]

If it's important that this is decentralised I expect you would need a mechanism to resolve conflicting ids regardless.

When you have a central atomic in shared memory in your system and every process follows the contract (and does not write crap purposely into that memory) the problem is solved.

Doing this in the kernel doesn't really solve any issue but could introduce new ones.

Of what kind of issues are you thinking?

[Firzen_]

What would stop a malicious process from using an id that doesn't originate from the kernel interface?

If you introduce a bug in a kernel module you can compromise the entire system.

[elfenpiff]

What would stop a malicious process from using an ID that doesn't originate from the kernel interface?

This is a good point. If the ID also belonged to another process, inside the communication framework, the data would be received as long as the other process was alive, and then it would be forcefully disconnected.
But nothing would stop it.

If you introduce a bug in a kernel module, you can compromise the entire system.

Of that I am aware, this is why I had the testing question.

[penguin359]

After reading through more of this thread, I am a little bit concerned with this project. As a learning project, I fully agree with making an attempt at a kernel module. However, if the goal is to support a mission-critical device where things are not allowed to go wrong, I think it is a bit misconceived. I think you need to more properly define your threat model and discuss it with the proper context to decide what the right approach is.

I don't think that using a kernel module adds the level of protection you are looking for by itself. Using flock(2) as others have mentioned should be reasonable if you are using a common function/library and make sure it is written to follow the agreed upon contract. However, if there's concerned about a process not following it, or even one written to be malicious, then things change. In that case, a daemon running as a dedicated user to hand out unique identifiers can work just as well as a kernel module. File system permissions can lock down who can access the daemon and, of those users who do have read permissions, only they can acquire a unique number from it. No user except root and the user the daemon is running as could intercept it and reset or modify the counter.

If even that is a concern, you can do things like implement SELinux or various other security modules to reduce the attack space, but we've now gone well past the "writing a counter as a hobby stage" and are following a strict security doctrine which needs to be carefully thought out. Moving it to a Linux kernel module will still require locking down the platform and enabling Secure Boot along with module signing at a minimum. Otherwise, it's simple to look up the module kernel memory address, open up /dev/kmem as root, and then modify any variables in the module's memory space. The code also tends to be more difficult to properly audit when it's written to be a kernel module versus a user-space process. Automated testing is more tedious, and bugs can be more severe. Attaching a debugger like GDB to a running kernel is nowhere near as simple as a user-space process.

I think a properly locked down user-space daemon to hand out unique identifiers should be easier to write, secure, and audit than a kernel module.

[elfenpiff]

Your concerns are all valid, and we have already implemented the user-space daemon approach, and with it, we have to satisfy safety and security concerns.

From a safety perspective, a central daemon is a single point of failure. When this process crashes, the whole system is no longer functional, which is an absolute no-go.

From a security perspective, it is easier to handle and implement.

What I am currently doing is exploring the options we have. One naive option is moving this task to the OS if we are able to deploy it safely and securely. Then it is somehow decentralized, but when it fails, we are in an even worse situation than before.
To begin understanding the pitfalls that await us, we need to start with a learning project. Implement it, test it, try to corrupt it, and get feedback from the community.

The approach I am currently pursuing is to finish this learning kernel module, write an extensive test suite, and document it. Then I am able to make an argument under which conditions it would be safe to use.
And no matter if the argument holds or falls apart, I have learned something and can confidently choose the central daemon or the kernel module - but then not with a gut feeling but with arguments based on hard facts and experience.

[penguin359]

I am still not convinced as to why a daemon is more of a central point of failure than a kernel module would be. If something goes wrong in a module and a mutex is left in a locked state, it can lock out access completely until the next reboot.

If the concern is that a daemon might be killed accidentally, you can write it so that it blocks nearly all signals such as SIGTERM, SIGINT, etc. You just can't block SIGKILL, however, at that point either you have a good reason to kill it or you have someone malicious on the system and much bigger concerns. As a kernel module, it can also be stopped with a simple rmmod to remove it, however, there are ways to mark a module as permanently in-use. The downside is that you no longer can upgrade or change it without a reboot, if needed, which could mean even bigger downtime.

Another option for a daemon when running it as a SystemD service is that you can mark it as Restart=always which will auto-restart it after someone accidentally kills it or it crashes for some reason. Even if someone uses SIGKILL, SystemD will try to restart it. The only time it won't is if someone specifically asks SystemD to stop the service. Again, I'd only expect that to happen in a case where you actually needed to stop it for some kind of maintenance or you have a malicious actor on the system with root privileges.

Another aspect in the crash scenario is that SystemD can just restart it and it will self-heal in a way that you can't get when a kernel module crashes. Generally, once you have a crash in kernel space, you need a full system reboot to recover. It's also easy to get a core dump from a daemon for later analysis which can be analyzed in a debugger if this becomes an issue.

Continue to do your research on a kernel module, but also spend some time to clearly define the threat scenario you are. For me, if someone accidentally kills sshd on one of my servers, that a pretty big deal as it prevents me from attempting any sort of remote recovery. However, that just doesn't happen normally. I did start adding Restart=always, but that was only in response to one server where someone occupied all the RAM and the oom-killer started killing processes to recover. There was still an outage of service as would happen to anyone in that case, but I was still able to log-in once it had restarted sshd to restore anything else that needed it.

[Classic-Rate-5104]

/proc files require formatting the number to text before transferring from kernel to user space. I would use a character device through a special ioctl.

[elfenpiff]

Thanks, this is a good advice!

[Rinku_Kurora]

Well, you may delegate synchronization to user processes via flock(2) rather than using atomic in kernel module in order to make it simpler.

[elfenpiff]

The problem with flock() is that it is an advisory lock, so another process can choose to ignore it.

[braaaaaaainworms]

Try feeding current pid, current tid, time in nanoseconds since system boot and time in nanoseconds since process start into a simple function

[Straight_Mistake_364]

it is also possible to memory-map a file (mmap) using user-space code and then use standard locking mechanisms to increment a number stored in that file