Chaos Computer Club - archive feed

How do you continually test and release new versions of systemd with confidence? Also, once released, how do you monitor PID 1 itself and your PID 1 usage across your server fleet? This talk dives into Meta’s way of answering these questions so we can minimize the risk of breaking changes and fun each systemd release brings us. Some of the technology in the talk is OSS, so you too, can join in on the fun knowing how your systemd usage is across your own infrastructure! This talk will dive into how Meta baseline’s our systemd usage across the fleet and use that data for CI, releasing and monitoring systemd. * Who am I + what do I work on * The common big monitoring hole many bare bone infrastructures have * PID 1 * PID 1 usage * Systemd @ meta * Imaging initrd * Initrd * Main os * Twine containers * Overview of OS image building and deployment @ meta * How we build images * How we provision servers * Chef’s role * What we check from our PID1 statistics to ensure a box is “healthy” enough to take workloads * Usage of hyperscale’s systemd-cd @ meta * What is systemd-cd * [https://sigs.centos.org/hyperscale/internal/ci/](https://sigs.centos.org/hyperscale/internal/ci/) * How do we use it * What issues has it found for us * Monitoring of meta’s systemd usage across the millions of hosts * Stats collected * Introduce monitord * Dbus (fun) vs. varlink * mention OSS alternative(s) found - explain why invented monitord * Introduce monitord-exporter * Show usage outside of meta (will be my small home infra + VPS’s) Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2024/talk/7APG3H/

This presentation introduces a novel approach to enhance the trust in SPIFFE by leveraging confidential computing technologies, specifically Confidential Virtual Machines. The presentation will provide an introduction to the realm of confidential computing, as well as an overview of SPIFFE/SPIRE. Armed with this knowledge we will demonstrate a practical example that integrates the AWS Instance Identity Document plugin with AMD SEV-SNP, showcasing the implementation challenges and solutions. SPIFFE is a framework to generate identities for software systems in dynamic and heterogeneous environments. SPIFFE Verifiable Identity Documents (SVIDs) enable us to be explicit about the trust we place in systems. However, the degree of trust we can place in SVIDs relies heavily on the soundness of the data gathering and verification process during node attestation. This presentation introduces a novel approach to enhance the trust in SVIDs by leveraging confidential computing technologies, specifically Confidential Virtual Machines (CVMs) such as AMD SEV-SNP or Intel TDX. These technologies enable us to track platform information directly in hardware, including firmware, boot loader, and kernel images, which are then signed with a key rooted inside the CPU itself. By incorporating hardware-protected platform information directly into the SVID generation process, we can significantly enhance the confidence placed in the resulting identity documents. Additionally, consumers of these SVIDs will be able to assert these properties before placing trust in a system. The presentation will provide an introduction to the realm of confidential computing, as well as provide an overview of SPIFFE/SPIRE, including the architecture of SPIRE agents and servers, the concept of workloads and SPIFFE SVIDs, and the role of node plugins in the attestation process. A practical example that integrates the AWS Instance Identity Document plugin with AMD SEV-SNP will be demonstrated, showcasing the implementation challenges and solutions. Through this presentation, attendees will gain insights into how confidential computing technologies can bolster the security of critical systems in an untrusted cloud environment, paving the way for more robust and resilient infrastructure in modern computing environments. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2024/talk/AG7L3K/

Yocto is a tool for building custom Linux distros. When you think about it, a container image is just a custom Linux distro. The distro (e.g. Alpine) is your base image and the customizations are the rest of your application or microservice. Like Podman, Yocto can generate a complete root filesystem in the form of an OCI container image. Originally targeted at bare metal, the Yocto configuration and build process seems complex when compared to the Containerfile approach of cloud native tools. Yocto's OpenEmbedded origins also mean that reduced image size, SBOM generation, license compliance, and reproducible builds were concerns early on in the project rather than afterthoughts. With security and risk of litigation now top of mind, this talk explains Yocto's uniquely layered and ultimately monolithic approach to solving these real-world software problems. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2024/talk/KZPRPN/

In this talk, I will discuss how Linux distributions can integrate and benefit from using systemd soft-reboot. Using openSUSE Tumbleweed as an example, I will show where and how it makes sense for traditional Linux distributions to use it and where the pitfalls are. With openSUSE MicroOS, we have a distribution with a read-only root file system that particularly benefits from a soft-reboot because a reboot is necessary after every update in order to change the root file system. However, this also requires special measures to ensure that it always functions smoothly. Afterwards I will talk about the requirements and solutions for services to survive a soft reboot and what's necessary to make the whole thing supportable. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2024/talk/YUAPMX/

postmarketOS was started with the lofty goal of enabling long term support for mobile phones and other devices with traditionally short lifespans, and doing so outside of the Android walled garden. This has inevitably resulted in a lot of upstream focused hardware bringup and development. Join us and learn what our community have been building, how we're running systemd on Alpine Linux and what we see in the future for postmarketOS. Through community driven efforts and collaboration, postmarketOS has grown into a highly adaptable platform which runs on anything from smartwatches and TVs to phones and laptops. In this talk, Caleb and Clayton discuss how our unique approach to tooling and package management have allowed such a small community to scale up to support hundreds of devices with more than 5 different bootloaders, over a dozen user interfaces, and now two init systems. They will cover: * A rough overview of the distro architecture * How device abstractions work in postmarketOS * Pmbootstrap and apk for fast developer iteration at a low cost * Systemd bootstrapping and current status * Our plan for an immutable postmarketOS (and request for feedback) Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2024/talk/LJXCKK/

This shows how to boot an [mkosi](https://github.com/systemd/mkosi) generated arm64 [Debian](https://debian.org) Image with [UKI](https://github.com/uapi-group/specifications/blob/main/specs/unified_kernel_image.md) and systemd-boot on a [u-boot](https://docs.u-boot.org/en/latest/develop/uefi/u-boot_on_efi.html) based EFI firmware with a [fTPM](https://github.com/microsoft/ms-tpm-20-ref/tree/main/Samples/ARM32-FirmwareTPM/optee_ta/fTPM) as a Trusted-Application in [OP-TEE](https://optee.readthedocs.io/en/latest/general/about.html) Embedded systems are very similar to IT managed PCs. A manufacturer of the device wants to ensure, that the system integrity is good, e.g. before unlocking secrets that allow accessing cloud services. Therefore the recent developments of the UAPI group and systemd are also very useful in the embedded world. This talk gives an overview of the involved software components and how they are combined. It shows how to build a firmware for an i.MX8MM that allows booting modern Linux images. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2024/talk/VZGAAG/

Why bother with Varlink IPC, and why now? The Varlink IPC has been around for a while, but recently we started using it heavily in systemd. In this talk I'd like to explain what Varlink IPC is, and why we are now adopting it so heavily. And I also want to explain why I think that Varlink is a good candidate as IPC of choice for any Linux software, both low-level and higher-level. We'll compare it with D-Bus in particular, and highlight where it shines (and where it doesn't shine so much). Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2024/talk/XSYMKW/

Container runtimes and other privileged system management tools have historically struggled with safely operating on a path within a directory tree controlled by a malicious user. [libpathrs][] is a library which makes it easy to do said path operations, as well as providing some other safe path-related utilities such as providing safe wrappers to operate on procfs files in a safe way. [libpathrs]: https://github.com/openSUSE/libpathrs As part of the kernel work on openat2(2) and continuing kernel work to make magic-links safer (against both confused deputy attacks and resource re-opening attacks), the need for a library to make it easy to do all sorts of VFS operations safely became obvious, and so [libpathrs][] was born. [libpathrs][] uses openat2(2) if available, but has a fallback to the old fashioned (and more finicky) method of doing safe-ish path resolutions. This talk will talk about how [libpathrs][] works and how it can help secure container runtimes and privileged system management tools against attacks, as well as touching on some ongoing kernel work which would allow for even more hardening. After the talk, slides will be available from [my site](https://www.cyphar.com/talks). [libpathrs]: https://github.com/openSUSE/libpathrs Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2024/talk/ZZFL7L/

Let's have an open discussion with systemd developers who are at ASG and users in the audience. We will open with the developers saying what they plan to work on in the near future, and then allow questions / comments from the audience. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2024/talk/YQZBGT/

A brief report about how we use io_uring in SLASH/fellow https://gitlab.com/uplex/varnish/slash, an always consistent, eventually persistent storage engine for Varnish-Cache. (FOSS, LGPL) Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2024/talk/U7GJJW/

Ensuring consistent and secure software builds is crucial in today's cloud-native environments. At Sidero Labs, we've developed a comprehensive approach to reproducible builds for Talos Linux using a variety of tools and techniques. This talk will explore our use of Docker Buildx, Kres, and other key components that contribute to our build system. We'll share insights into our methods, challenges faced, and solutions implemented, providing practical guidance for developers aiming to achieve reproducibility in their own projects. To achieve a fully reproducible stack, from the kernel and initramfs to the software we own and third-party software we build, we use multiple tools in our toolset: - Buildx: Provides a consistent environment for building software. - Kres: Our project scaffolding tool for generating and updating build instructions and dependencies. - Code Patches: Address issues in third-party projects that prevent reproducible builds. - Tests: Written by us to ensure and verify reproducibility. In this talk, we will cover each of these tools and techniques, providing examples and practical insights. You will learn how to apply these methods to achieve reproducible builds in your own projects, gaining a complete picture of our approach and how it can be adapted to your needs. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2024/talk/RYZJ9W/

Same as every year, a lot has happened in the systemd project since last year's ASG. We released multiple versions, packed with new components and features. This talk will provide an overview of these changes, commenting on successes and challenges, and a sneak peak at what lies ahead. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2024/talk/RLZEPD/

Ideas for improving systemd-boot Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2024/talk/DT3RCU/

A new way to develop on immutable Linux Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2024/talk/NSKLAR/

Integration testing environment for mixed HPC and cloud workloads Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2024/talk/XNQLTE/

This presentation will review how far Debian (and more generally, traditional distributions) is from supporting factory reset: what can work, what is missing and possible hacks^Wways to do it without starting a distribution-wide effort. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2024/talk/3K8NZT/

oo7-daemon (a temporary name based on the oo7 client library) project aims to provide a replacement for the gnome-keyring-daemon as the new D-Bus Secret Service provider in the GNOME desktop environment. In this talk I will go through the latest development plans and the progress made to integrate TPM backed credentials support to oo7-daemon using systemd per-user credentials as a backend. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2024/talk/8TMT9T/

A quick overview of how RAUC uses libcomposefs to handle new use-cases. Traditionally, RAUC focused on A/B updates for whole partitions, either by using filesystem images or tar archives. While the image-based OS approach has many benefits, there are scenarios where more loosely coupled components need to be handle in addition to the root filesystem. In RAUC, these can be handled with using the new "artifact updates" support. As a system might have many artifacts installed in parallel, such as for containers (systemd-nspawn or otherwise) and systemd-sysexts, efficient storage is important. In many cases, these are updated often, so download efficiency is important as well. After evaluating multiple alternatives, we've now decided to integrate composefs. Besides solving the requirements above, it additionally provides the same level of integrity protection as a dm-verity root filesystem, which is important in systems using secure boot. This talk will show how RAUC uses libcomposefs and the new use-cases supported by having an efficient content-addressed backing store with full authentication. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2024/talk/3DKX9V/

systemd introduced Portable Services support in 2018, as part of v239. This feature was covered at ASG 2018 and in a blog post published at the time: https://0pointer.net/blog/walkthrough-for-portable-services.html But a lot has changed in the past 6 years, and very crucial new features have been introduced, so it is time to have another look at this topic and see what has happened in the meanwhile, what new use cases have opened up, and what is coming in the near future. https://systemd.io/PORTABLE_SERVICES/ Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2024/talk/DGVBSC/

Probably the way systemd is thought of and used is mostly as a service manager, and a collection of tools built around the idea of “low level user space”. We rarely think of it as a library that can be used as part of any high level language or application. This talk will cover this aspect of systemd, and through the lens of pystemd, explore how applications can use (and abuse) systemd. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2024/talk/VAQPQW/

Arch Linux creates 2 cloud images, 2 vagrant images every month using custom bash scripts and requiring root for building. This talk will look at how these images can be created using mkosi, building them in CI, testing the build images and as a bonus; build reproducible? Project link: https://gitlab.archlinux.org/archlinux/arch-boxes Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2024/talk/QFUGLT/

On general purpose image based systems such as Flatcar and Fedora CoreOS, users are encouraged to run all their applications using containers. To make updates safe and predictable, the system is mounted as read only and local modifications are discouraged. While containers offer a lot of flexibility on Linux, there are still cases where installing binaries or running applications directly on the host operating system is preferred. For example to add kernel modules, use an alternative container runtime version, add more udev rules, etc. Some of those use cases could be addressed with statically linked binaries, but their management is manual and their usage creates new issues around updates, versionning, memory footprint and not everything can be statically compiled. Alternatively, one can build its own image but at non-negligeable maintenance costs. Systemd's system extensions (sys-ext) provide a mechanism to extend the content of the host while preserving the safety guarentees around updates. We will demonstrate how Flatcar, Fedora CoreOS and Atomic Desktops are leveraging sysext images to securely extend the OS. With practical examples and usecases (e.g Cluster API) learn how to install Python, Podman, Kubernetes, ZFS, everything at the same time, by composing your very own image with systemd-sysext. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2024/talk/HJLF3C/

With systemd tooling, including mkosi, it is possible to build an OS image that fulfills all checkmarks a modern image-based OS should have, but with a standard off-the-shelf distribution! This talk gives an overview for a possible workflow, including A/B updates and offline signed images and updates, in real-use. As a bonus, it is also self-replicating and uses as little configuration as possible, leveraging built-in systemd auto detection. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2024/talk/LJAYYL/

While software bills of materials become of increasing value to further trust in the software supply chain, generating high quality SBOMs still poses some challenges in some ecosystems due to the lack of proper tooling or accessible build metadata. In this talk, I'll explain and demonstrate how we can leverage the static dependency graph of functional package managers like Nix to generate very precise SBOMs, that can be relevant for running a service on any linux distribution thanks to systemd portable services. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2024/talk/7XGYDC/

An update on systemd's TPM features, i.e. what happened since last year, i.e. systemd-pcrlock, NvPCRs, and Varlink APIs. At last year's ASG I already did a systemd & TPM talk, and this is supposed to be a follow-up to that, with everything that happened since then, plus what's next and what's missing. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2024/talk/VQLZBT/

Containers have become the de facto choice for deploying most applications, and all of us benefit from the isolation, portability, and the surrounding ecosystem. In this talk we’ll take a deep dive into the world of bootable containers, using the same ideas, goals and technology for the host system (whether virtualized or bare metal). We’ll look at the bootc project under the github.com/containers umbrella and its current flagship distribution usage in the new Fedora/CentOS bootc project and initiative. We hope you are as excited as we are by taking cloud-native approaches down to the operating system level, and a key goal is finding points that can be shared with other components of the ecosystem, from the uapi-group.org to other container-based OSes. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2024/talk/LA9LXV/

You may have heard about this weird distribution, NixOS, that breaks compatibility with /usr. This talk explores the properties inherent to NixOS, focusing on its distinct approach to package management and system configuration. Learn how these principles combine with general upstream efforts at bringing TPM2, Secure Boot and more to your Linux distribution. Everything you wanted to know about why NixOS do things a certain way will be answered here. The idea is that you get out of this talk understanding the different compromises done by the NixOS community and what they get out of it. We will cover https://github.com/nix-community/lanzaboote which is a Rust UEFI stub similar to systemd-stub with fewer features but with one unique special feature for NixOS, similar to UKI addons. We will also do a status report of where NixOS stands in terms of adoption of systemd features such as systemd-pcrlock. Licensed to the public under https://creativecommons.org/licenses/by/4.0/de/ about this event: https://cfp.all-systems-go.io/all-systems-go-2024/talk/UQ3CYU/