modified: Tuesday 29 March 2016
Moving away from Arch Linux Part 1: Systemd
This is the first post in a multi-part series about moving away from Arch Linux and installing another distro.
- Part 1: Systemd and reasons for leaving (this page)
- Part 2: Looking at gentoo and Void Linux.
- Part 3: Experiences with Void Linux (still to be written)
Years ago I found Arch to be an amazing distribution. I loved the fact that the user was trusted to keep things running. Administrating the system was transparent with very little obfuscated automation. If something broke, I knew it was my fault, not some random script doing something unexpected. You even had to identify and install many dependencies yourself — packages would only list the deps that they need to run, not the deps required for all of their features to function.
This contrasted with my previous distro, Debian. If you installed a package that had a daemon, the daemon’s service would immediately be added and started. If you installed a package then every conceivable dependency would be installed along with it (one of my favourite apt-get flags was —no-recommends , especially when I was on a 2GB internet cap).
Today Arch still has some features that other distributions cannot beat it on. The Arch Linux wiki has become a defacto central hub for linux user documentation. The installation process is manual (especially useful given that I always break the assisted/gui installers). Luckily these plusses are mostly Arch agnostic: the wiki’s content is useful regardless of what distro you are running; and the installation instructions have taught me enough to be able to bootstrap other distributions as well.
My experiences with Arch Linux on both my desktop and my laptop have slowly gone downhill since my initial impressions. This post outlines why I want to move on and future posts will describe my journey migrating my desktop installation away from it. I plan to tackle my (somewhat more complicated) laptop another day.
When I first installed Arch Linux it used sysvinit as its init system. Systemd was a fabled futuristic init system that promised speed-ups and parallelisation. I was excited about it and tried to install it, but returned to the sysvinit system after that.
Arch Linux migrated from sysvinit to systemd some years ago. The sysvinit system is no longer supported.
Systemd is now the primary reason I want to distro-hop away from Arch: over time my issues with it have slowly built up and I no longer want to tolerate it being a component of my system.
Over the past few years systemd has become a core component of many Linux distributions. Now many popular distros will not function without systemd because their core packages depend on it.
It’s interesting that the Wikipedia page for systemd (at the time of writing) describes it as (only) an init system:
systemd is an init system used by some Linux distributions to bootstrap the user space and manage all processes subsequently, instead of the UNIX System V and Berkeley Software Distribution (BSD) init systems
I would say that’s quite inaccurate. ‘Init’ is one feature of the infinitely growing list of features that systemd provides. The freedesktop.org page has a better idea. I don’t expect you to read all this:
systemd is a suite of basic building blocks for a Linux system. It provides a system and service manager that runs as PID 1 and starts the rest of the system. systemd provides aggressive parallelization capabilities, uses socket and D-Bus activation for starting services, offers on-demand starting of daemons, keeps track of processes using Linux control groups, supports snapshotting and restoring of the system state, maintains mount and automount points and implements an elaborate transactional dependency-based service control logic. systemd supports SysV and LSB init scripts and works as a replacement for sysvinit. Other parts include a logging daemon, utilities to control basic system configuration like the hostname, date, locale, maintain a list of logged-in users and running containers and virtual machines, system accounts, runtime directories and settings, and daemons to manage simple network configuration, network time synchronization, log forwarding, and name resolution.
Over time this list has only got bigger and it appears that the project wants to continue expanding.
Bad experiences with systemd on my laptop
Disclaimer: my laptop’s processor architecture is armv7h, not x86. It’s a Series 2 Chromebook made by samsung .
I have had a lot of fun with this laptop. My /etc/issue file sums up the most interesting failures I’ve experienced:
Welcome to Clusterlizard, chamber \l. Please praise your selected deity(s) Summary of interesting failure stories since December 2013: - 'debug' on kernel command line -> systemd becomes very verbose -> random race conditions due to text output -> crash - filesystem death on SSD by power-cycling too rapidly too many times whilst debugging other issues - btrfs failure from trying to compile firefox ("ran out of inodes" even though btrfs does not use inodes in its design) - systemd update -> required kernel support for xattrs on folders -> boot hung - cracked the screen by carrying in my bag with Chinese take-away containers - systemd update -> prevented kernel from loading userspace wifi firmware - bootloader occasionally corrupts, fails to work until battled into a soft-reset after many attempts - upgraded many packages, write-cache filled memory, OOM -> system hung, filesystem left inconsistent - updated kernel -> post-install script dd'd bootloader over my root filesystem Misc self-inflicted - removed systemd-sysvcompat - custom init system: spawned gettys with wrong arguments -> no logins - repeat episodes of systemd firmware issues Happy birthday Clusterlizard! (2014) (2015)
Note that ‘Clusterlizard’ is the hostname of my laptop. I like giving my computers names interesting enough to remember.
I’ll go through the backstory for some of these issues, even though some of them are not relevant to systemd, because they’re still pretty entertaining.
The systemd verbosity issue was a well publicised problem that lead to some heated discussion on the linux kernel mailing lists:
Summary: if you had ‘debug’ on your kernel command line then systemd would pick this as a cue to also be verbose. Something to do with the sheer amount of messages being printed would cause crashes on bootup.
This bug inconsistently triggered for me and I was convinced that my hardware was flaky for some time. Eventually I discovered that the quirky unsigned version of u-boot (my bootloader) had a command-line interface that let me get rid of this kernel argument. This was not easy: the documentation for this variant was scarce and the ‘help’ command printed out several screen-heights of text without a way to scroll back up. Eventually I worked out that the variables ‘set’ in the environment included lists of commands for booting, and by luck the list of set variables fitted exactly into one screenheight. A bonus of fixing this problem was that I didn’t have to deal with the debug messages during runtime (which I did not want to see anyway!).
More fun was the issue where systemd refused to boot my system. I had recently updated systemd and the new copy wanted to use a feature (xattrs) that my kernel did not have. Rather than gracefully continuing, it would outright hang during boot.
For most of a year I held back from updating systemd on the laptop because the kernel verison in the Arch Linux ARM repositories was too old (and I didn’t want to compile one myself on this hardware). This lead to a whole pile of other packages being held back, until I was at a point where I could not update (nor easily install) anything.
This issue resolved itself and then repeated itself at a later point. The second time around firmware loading no longer functioned, breaking the wifi.
It turned out that the distro maintainers had abandoned my kernel’s package (linux-snow) and instead had moved onto another one that supported multiple variants of the ARM chromebooks (linux-peach). I discovered this by chance whilst searching through packages. I do not think I was alone in not knowing this as other users on the forums had the same issue.
Last year I went on a holiday to a remote mountainside village in Greece. Here internet was scarce and I had just discovered the replacement kernel package. “Ooh!” I went. “What could go wrong?”.
The new kernel package assumed a different partition layout to what I had. A post-installation script in it asked if it could ‘install’ onto my root partition. “Sure” I thought. I lost my root partition. Lesson: read other people’s scripts before trusting that you understand what they do.
A day’s work later at an internet cafe (does that even make sense?) I had a fresh install up and running. It’s worth noting that the wifi at internet cafes in rural villages in Greece is faster than wired internet in the suburbia of Australia.
Whilst at Greece I wrote my own init system to replace systemd on my laptop. It’s a shell script with some supporting utilities and it has been the best thing that I have ever done for my laptopping experience. Fast, simple and educational.
Systemd still remains installed as many programs will not function without libsystemd.so. Eventually I plan to hop away from Arch on this device too, but at the moment I have averted most of my problems with my own init.
Random stuff appearing in my boot sequence
Recently my desktop has been pausing for almost ten seconds with this item at bootup:
Feb 26 13:20:01 Shyarenris systemd: Starting Manage, Install and Generate Color Profiles... Feb 26 13:20:02 Shyarenris dbus: [system] Successfully activated service 'org.freedesktop.ColorManager' Feb 26 13:20:02 Shyarenris systemd: Started Manage, Install and Generate Color Profiles.
What? I don’t user colour profiles. I have never asked my computer to start a colour profiles service on bootup.
Someone has assumed that I want this and decided it should automatically be added. Congrats whoever you are, you’re now an honourary sysadmin of my home computer.
Hard dependencies in the last places you would expect
After removing the systemd package from my laptop one day, I discovered this gem:
I have no understanding of why pgrep would require systemd to function. I can plainly see it’s dynamically linked to libsystemd.so, but why I can’t fathom. I can also see the sanity oozing red from my tentacles. And circling the pigeons twice. Settling somewhere arid without overstops.
Apparently this is a widespread issue for users trying to move away from systemd. If you switch from udev (often/mostly/always integrated with systemd) to eudev (a fork) on Arch linux, then you will probably want to make use of the AUR package eudev-systemdcompat. Last time I looked this package simply provides the shared object files from the normal (x86) systemd packages. This ensures any programs with hard dependencies on systemd still function.
Seriously though. Grep. Systemd. What happens? A journalctl entry is made every time a user greps? Pgrep can dbus?
Purpose and direction
The fact systemd keeps obtaining new features scares me.
In real life: if I need something, I go and find it. I don’t wait for the product or service to come to me.
When the products or services does come to me, then I am very sceptical of it. Why has it come to me? Who is providing it? Why are they doing it?
In the case of systemd I feel that the project direction is costing me time and happiness. It should be a no-brainer to switch but until now my desire to just keep my computer running (and play games!) has been more than my desire to install a new operating system.
Part two will cover my experiences backing up my system and installing Void Linux, one of the (increasingly fewer) distros that does not use systemd. I’ll also write up a seperate post about the numerous things I’ve done on my ARM-based laptop (including my custom init).
Feel free to discuss (and disagree), just make sure you aware of the rules. If you are after some reading material that from the other side of the fence, Lennart Poettering (main systemd guy) has a blog.