Chaos Computer Club - recent events feed

File descriptors for processes on Linux have been available for quite some time now. Userspace has adapted them widely. Over the last two years or so we've extended the abilities of pidfds significantly. This talk will go over all the new features and deep dive into their implementation and usage. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2025/talk/3BMJVH/

Going for minimal containers with restricted system calls and unprivileged users is the usual Kubernetes approach these days, and it works great for most web apps. However, the development of more complex infrastructure extensions frequently hinders application functionality. While looking for a solution to deploy virtiofsd in an unprivileged container for KubeVirt, we stumbled on seccomp notifiers. Seccomp notifiers are a kernel feature which monitors syscalls and get notifications to a userspace application when a syscall is executed. Alternative options involved either the use of a custom protocol using UNIX sockets or the deployment of virtiofs as a privileged component alongside the unprivileged VM. After our evaluation, the seccomp notifier turned out to be the simplest solution among all the choices. Unfortunately, the main constraint is the monitor's resilience after a restart, such as after a crash or an upgrade. This limitation forced us to back up to one of the less elegant approaches. But there is hope how this could be solved! The session will explain why seccomp notifiers are a lean solution to avoid extra userspace communication and synchronization, the current limitations and possible future solutions to overcome today’s challenges. Our experience will teach audiences several methods for dividing their privileged infrastructure. Utilizing virtiofsd as an actual example and a target application for KubeVirt integration and deployment. We will discuss the difficulties of using rootless containers in this session, as well as the design patterns, technologies, and tactics we thought about and ultimately chose to maintain or reject. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2025/talk/SPGAXS/

CentOS Stream is especially suited for production deployments. In these environments it's often common to develop improvements to distribution packages and want to contribute them upstream. Unfortunately, until very recently that required one to then maintain their own build and deployment pipeline for the packages, at least until the changes made their way into the distribution. CentOS Proposed Updates (CPU) SIG aims to bridge this gap - changes that have been submitted as merge requests can be built in this SIG, providing those who run Stream in production with access to needed updates while they make their way into CentOS Stream. We hope this will help increase collaboration between RHEL engineers, CentOS Stream contributors, and the rebuild community as well, especially those that have distributions derived from CentOS Stream directly (such as AlmaLinux with AlmaLinux OS Kitten), as everyone can focus on making improvements without reinventing their own build pipelines. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2025/talk/9QUZNY/

Coredumping on Linux has long been a nightmare. Currently two modes are supported: (1) Dumping directly into a file somewhere on the filesystem. (2) Dumping into a pipe connected to a usermode helper process spawned as a child of the system_unbound_wq or kthreadd. For simplicity I'm mostly ignoring (1). There's probably still some users of (1) out there but processing coredumps in this way can be considered adventurous especially in the face of set*id binaries. The most common option should be (2) by now. It works by allowing userspace to put a string into /proc/sys/kernel/core_pattern like: |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h The "|" at the beginning indicates to the kernel that a pipe must be used. The path following the pipe indicator is a path to a binary that will be spawned as a usermode helper process. Any additional parameters pass information about the task that is generating the coredump to the binary that processes the coredump. In the example core_pattern shown above systemd-coredump is spawned as a usermode helper. There's various conceptual consequences of this (non-exhaustive list): - systemd-coredump is spawned with file descriptor number 0 (stdin) connected to the read-end of the pipe. All other file descriptors are closed. That specifically includes 1 (stdout) and 2 (stderr). This has already caused bugs because userspace assumed that this cannot happen (Whether or not this is a sane assumption is irrelevant.). - systemd-coredump will be spawned as a child of system_unbound_wq. So it is not a child of any userspace process and specifically not a child of PID 1. It cannot be waited upon and is in a weird hybrid upcall which are difficult for userspace to control correctly. - systemd-coredump is spawned with full kernel privileges. This necessitates all kinds of weird privilege dropping excercises in userspace to make this safe. - A new usermode helper has to be spawned for each crashing process. On recent kernels a new mode has been added making use of AF_UNIX sockets. This talk will talk about the new modes, how they can be used, what advantages they have in comparison to the other modes, and look at technical implementation details. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2025/talk/MADB7R/

GNOME has collected some very old code over the years. During the recent GNOME 49 release, we've made some drastic cleanups. Most visibly, we've dropped support for X11 and gained many dependencies on systemd. Let's explore some of the what and why for these changes! Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2025/talk/FQE7QZ/

GNOME OS is a distribution based around systemd-sysupdate. This year, we finally created a live installer image using the same /usr partition as the installed OS. The main innovation however is the ability to install without the need to reboot. The user can start working while the installation is happening. This live image is built using systemd-repart. And the installer itself also uses systemd-repart. But systemd-repart is not the complete solution and we had to solve some challenges. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2025/talk/QRJVL3/

It has been 10 years since Axis Communications had a presentation at the systemd conference. Back then, we have shown how we have increased our product quality, stability and boot times by porting our platform to systemd. 10 years later, we had different challenges to keep the resource usages and boot times under control. We have started from bottom up and sliced our software for this purpose. This work also got us inspired to create virtual versions of our hardware products that we cluster deploy using systemd's nspawn. We have hundreds of engineers working on a software platform that is the base for different product types. It is a different challenge to keep the resource usages of different software composition when so many independent engineers collaborate together. We have applied a different strategy to keep our products as slim and as optimized as possible using different systemd principles like targets, slices, resource prioritization. As a side effect of this work, we have started from ground up and started to virtualize our products using systemd-nspawn. The next step for us was to figure out how in best way to cluster deploy our virtual products so that we can increase the quality of our end to end systems. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2025/talk/Z3NGWJ/

Many traditional container engines make use of the "subuid" concept and the "newuidmap" tool to implement a concept of "unprivileged" user-namespace containers on Linux. This approach has many shortcomings in my PoV, from both a security and scalability standpoint. Recent systemd versions provide a more powerful, more secure, mor scalable alternative, via systemd-nsresourced, systemd-mountfsd and other components. In this talk I want to shed some light on the problems with the "old ways", and in particular focus on what the "new ways" bring to the table, and how to make use of them in container runtimes. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2025/talk/E7FHPY/

Flatpak is the de-facto standard for distributing desktop applications across various Linux based systems. It also offers other advantages such as sandboxing. It is particularly useful for image based systems as it installs the applications into a separate location and doesn't try to modify the system. GNOME OS is GNOME's development, testing and QA operating system. It builds the latest and greatest in-development versions of the GNOME desktop and core applications. It is also Linux based system that tries to fully embrace the systemd ecosystem. The applications are however built into the system. While this might be great for testing the apps as they would be in most distros, we also want to build our Flatpak applications from the same build definitions and our users (or more correctly early adopters) prefer to use Flatpak for various reasons. In this talk we'll explore what other image based distributions do to provide Flatpak applications to their users, what users expect from "Flatpak applications" and the various proposals for implementing that in GNOME OS. We hope to be able to present our end result by the time of the conference. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2025/talk/98W9EX/

oo7-daemon is the new D-Bus Secret Service provider that aims to fully replace gnome-keyring. In this followup (continuation of my 2024 talk) lightning talk, I will go through the progress made, the challenges faced and the status of systemd credentials integration. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2025/talk/NFNFJS/

Marco will review the features available in the initramfs-tools ecosystem, the initrd generator used by Debian and Ubuntu, and how they can be implemented (or not) by adopting mkosi-initrd. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2025/talk/E989ZX/

At Meta, we've been looking to revamp our internal container runtime (Twine). Instead of maintaining all the low level container runtime code ourselves, we'd much prefer having more of this managed by systemd. This talk will go over what we did to make systemd transient units a suitable environment for running system containers (pid namespace support, cgroup namespace support, namespace delegation, ...), and why we went this route instead of reusing systemd-nspawn. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2025/talk/BBTJSF/

Simplify systemd management with `isd`! `isd` is a TUI offering fuzzy search for units, auto-refreshing previews, smart sudo handling, and a fully customizable interface for power-users and newcomers alike. If you ever became frustrated while typing: - `systemctl start --user unit-A.service` (manually starting a unit) - `systemctl status --user unit-A.service` (seeing that it failed) - `journalctl -xe --user -u unit-A.service` (checking the logs) - `systemctl edit --user unit-A.service` (updating the unit) - (repeat until problem is solved) `isd` could help. In this presentation, we will discuss the features that `isd` currently supports, the features that are planned for the future, and the experience of developing a TUI for `systemd` commands. I hope attendees will find the content engaging and practical. Audience participation is highly encouraged. I am especially eager to hear your thoughts, ideas, and feature requests. If you think a tool like `isd` might be unnecessary or redundant, I'd love to hear your perspective, too! Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2025/talk/GCV3PM/

Many OS artifacts today are still verified using proprietary, stateful keyring formats. With the "File Hierarchy for the Verification of OS Artifacts (VOA)" an attempt is made to rid the ecosystem of this limitation by implementing a generic lookup directory. With extensibility in mind, this unifying hierarchy currently provides integration for OpenPGP, with further integrations in planning. While working on improvements to the [ALPM](https://alpm.archlinux.page) ecosystem, the way packages and other OS artifacts are currently verified on Arch Linux has been evaluated. Noticing the extensive vendor lock-in to GnuPG and with today's widespread availability of [Stateless OpenPGP](https://wiki.archlinux.org/title/Stateless_OpenPGP) implementations in mind, a plan was hatched to create a more generic, stateless approach. A specification and implementation for the [UAPI group](https://uapi-group.org/) has been started to create a ["File Hierarchy for the Verification of OS Artifacts (VOA)"](https://github.com/uapi-group/specifications/pull/134). This approach is meant to be technology agnostic and allow further integrations, such as SSH and X.509. Follow along for an overview of what this specification is trying to improve upon and how today's tools could benefit from it in the future. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2025/talk/7DDSVZ/

For many years, firewall solutions on Linux have grown and evolved, without any major change, until eBPF. While eBPF can allow very fast and efficient packet filtering, the learning curve doesn't make it easily accessible to non-developers. bpfilter aims to bridge the gap between existing tools (nftables, iptables) and modern technologies such as eBPF. By translating filtering rules into native code, bpfilter abstracts the complexity behind cutting-edge kernel technologies while maintaining backward compatibility with existing solutions. Let's discuss about bpfilter and see it in action! Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2025/talk/JEVBTZ/

More than six months have passed since Daan tried to ~~shame~~ gently peer pressure Lennart to actually use the stuff he builds, via a FOSDEM talk: https://fosdem.org/2025/schedule/event/fosdem-2025-4057-particleos-can-we-make-lennart-poettering-run-an-image-based-distribution-/ Did he succeed? Is dogfooding standard practice now in the systemd development process? Or do things like randomly breaking logging in GNOME (*cough*) still happen from time to time? Join us for this talk to find out, and to apply yet more peer pressure. We will also spend some time talking about more boring and mundane topics, such as giving an overview of the current status of ParticleOS, and how we build it as a ready-to-consume and secure-by-default signed and self-enrolling appliance on the SUSE Open Build Service. https://github.com/systemd/particleos https://build.opensuse.org/package/show/system:systemd/particleos-fedora https://build.opensuse.org/package/show/system:systemd/particleos-debian Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2025/talk/QMYAMS/

Landlock is an unprivileged kernel feature that enables all Linux users to sandbox their processes. Complementary to seccomp, developers can leverage Landlock to restrict their programs in a fine-grained way. While Landlock can be used by end users through sandboxer tools, there is currently no well-integrated solution to define security policies tailored to system services. Although AppArmor and seccomp security policies can already be tied to a system unit, we aim to provide a more dynamic, standalone, and unprivileged option with Landlock. In this talk, we'll briefly explain what Landlock is and highlight its differences from other Linux security features (e.g., namespaces, seccomp, other LSMs). We'll then focus on the new configuration format we are designing for Landlock security policies, its characteristics, and how it could extend systemd units by taking into account runtime context (e.g., XDG variables). Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2025/talk/FXWDCF/

systemd-sysext provides a nice way to enhance a distribution with a read-only root filesystem without the need to reboot. But there is additional tooling necessary to manage the sysext images: * install an image which is compatible with the installed OS version * update installed images to the newest compatible version * rollback images in case of an OS rollback * cleanup unneeded images In this presentation I will talk about which tooling systemd itself provides for this (importctl, updatectl, ...) and what the benefits and disadvantages of this tools are compared with real world use cases. In the end I created an own, generic and distribution independent tool for this using systemd tools in the backend. Using openSUSE MicroOS as example I will demonstrate how we solved the problems with it and how we integrated it in our update stack. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2025/talk/8AA87L/

On image based desktops distributions such as Fedora Atomic desktops and Universal Blue, users are expected to run their graphical applications using Flatpaks and their command line ones using containers. But that approach does not work well for some applications that require more privileges, direct access to devices or kernel interfaces. With systemd system extensions (sysexts), it is possible to extend an image based system on demand. Sysexts come with a lot of advantages: they can be created out of arbitrary content (not only packages), are quickly enabled or disabled and can be built and shared independently of the main distribution channels. We will demonstrate how the Atomic Desktops can take benefit of sysexts to provide extensions such as virtual machine management (libvirt), alternative container runtimes (moby-engine or docker), IDE (VS Code) or debugging (gdb). We will also look at important details when building sysexts, the current limits when deploying them (SELinux policy modules, service management, RPM database update), what is currently blocking us from using it for more complex cases (kernel modules) and what we would need to properly manage and update them. Supporting examples for this talk: https://github.com/travier/fedora-sysexts Work in progress sysexts manager that targets managing sysexts on Bootable Container systems: https://github.com/travier/sysexts-manager Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2025/talk/DCVQLK/

All the big cloud providers provide your machines with a unique cryptographic identity that can be used to talk to their cloud services securely without having to manage or rotate any cryptographic secrets yourself. For example GCP has Service accounts and AWS has IAM roles. This ubiquity of cloud identity and the seamless integration with all the the services of these cloud providers is one of the reasons why they are so successful. SPIFFE (Secure Production Identity Framework For Everyone) tries to unify these concepts of workload identity in a vendor neutral framework. But how do we bootstrap our cryptographic identity securely when we are running things on our own hardware as opposed to on cloud? What is our bottom turtle? In this talk, I will show how I use Nix in combination with the swiss-army knife of tools provided by systemd (ukify, systemd-measure, systemd-repart, systemd-veritysetup-generator) to create reproducible images for which we can predict TPM measurements. Paired with a custom attestation plugin for SPIRE (the reference CA server for SPIFFE) that uses TPM remote attestation I can give each of my servers a unique identity encoded in a TLS certificate if and only if they were booted up with the software that I intended them to boot up with. This then allows me to have workloads talk to each other with mutual TLS without having to manage any keys or certificates myself. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2025/talk/X3ZSXV/

Provide a varlink service to access /etc/passwd and /etc/shadow so that no setuid and setgid binaries are necessary for this task. There are two independent "problems" which can be solved with the same idea: all files in /usr should be owned by root:root and no setuid binary should be needed. The first one is a requirement of image based updates of /usr to avoid UID/GID drift, the second one is a security feature wished by systemd developers and security teams. Currently most setuid binaries (or setgid binaries owned by group shadow) beside su and sudo only need this to read the shadow entry of the calling user. This task could be delegated to a systemd socket activated service which provides the user shadow entry for the calling user. In this talk I will present the why, the current implementation and feedback from security teams. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2025/talk/RUTE9Y/

systemd-confext is a lightweight overlay mechanism for /etc, allowing you to drop in a configuration extension ("confext") bundle and let systemd make it visible to your service as though it was already shipped with the base image. Building on the same extension magic as systemd-sysext, confext also introduces extra features tailored for the /etc use case, such as vpick-ing the newest version and the ability to pick up config revisions with a `systemctl reload`. This talk presents the changes to systemd-confext since [its debut at All Systems Go! 2023](https://cfp.all-systems-go.io/all-systems-go-2023/talk/XLQNDJ/), the lessons learned along the way to make it work, and how we leverage this capability at Microsoft already to deliver configuration payloads in production. Immutable Linux distributions offer stability and reproducibility, but at the cost of configuration changes needing time to build. A small configuration or system change can then require complete redeployment of the entire OS, adding friction to development work and impacting customers. This is not acceptable for certain Linux environments at Microsoft, where only seconds of planned downtime budget exist every year. systemd-confext is intended to be a signed, `dm-verity`-protected, live configuration update mechanism meant to address this issue. It provides a way to make quick additions in a secure and reliable way with minimum customer impact. Two years later, confext now supports host-level payloads to /etc and also works in the individual namespaces of services and portable units via `ExtensionImages=` and `ExtensionDirectories=` too. One of the recent significant additions is that systemd-confext has been integrated with `systemctl reload`, giving it the ability to pick up new configuration revisions during a reload, not just a restart. Combined with services that implement notify-reload, it is possible to simply drop a new versioned extension into the watched directory to trigger the reload flow and update the service config. We'll review how these features can be used and include a demo of a couple of use cases, including path-activated units to deploy config payloads in production at Microsoft. We'll also briefly discuss the namespace and mounting changes needed in the systemd codebase to make this integration work. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2025/talk/GSRYLR/

Let's have an open discussion with systemd developers who are at ASG and users in the audience. We will open with the developers saying what they plan to work on in the near future, and then allow questions / comments from the audience. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2025/talk/PXZGEL/

Haven’t you ever wanted to find ways to make your Rust code the most optimal in the world? I know how you feel. This is a talk, where I’d tell you how easy it is to profile your Rust software and how most often the solutions are trivial. This is a story of how I used a few readily-available Open Source tools to achieve huge optimizations in [zbus](https://crates.io/crates/zbus), a pure Rust D-Bus library. This was long journey but gains were worth the efforts. I will go through each single bottleneck found, how it was found and why it was a bottleneck and how it was optimized away. While attending this talk will by no means make you an expert in optimizations, it is my hope that by you will be able to relate to some of bottlenecks or solutions I will present (“hey”, I also do that in my code!”) and learn from my experience. Maybe afterwards, you can suggest an even better solution? Moreover, if you don’t already have any experience with profiling and optimizations, this talk should be a good introduction for that. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2025/talk/TYRJUG/

Communication is paramount in modern application development. This applies equally well to the process of writing applications and to the code itself. The complexity of the tasks ahead of us calls for a distributed and coordinated development effort, and this often manifests in our code: We design distributed, communicating systems to split complexity and responsibility among many people and teams, and at the same time meet the demand for ever faster systems. The last decade showed significantly increased popularity in API design, network protocols, and distributed computations. At the same time some of the most intriguing language research improves how multi-threaded applications synchronize and exchange information without sacrificing safety or performance. Between these two lies an almost forgotten world: Linux Inter-process communication (IPC) has lost ground to thread-communication and networking protocols. Let us look at how other operating systems have evolved their IPC layers, what new systems decide to go with, and why Linux IPC has not seen any major changes since the 90s. And finally, can we lift Linux IPC out of stagnation and catch up with everyone else? Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2025/talk/8WW7YH/

Same as every year, a lot has happened in the systemd project since last year's ASG. We released multiple versions, packed with new components and features. This talk will provide an overview of these changes, commenting on successes and challenges, and a sneak peak at what lies ahead. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2025/talk/B8RVCJ/