modified: Tuesday 29 March 2016
author: Hales
markup: textile
Moving away from Arch Linux Part 1: Systemd
This is the first post in a multi-part series about moving away from Arch Linux and installing another distro.
- Part 1: Systemd and reasons for leaving (this page)
- Part 2: Looking at gentoo and Void Linux.
- Part 3: Experiences with Void Linux (still to be written)
Years ago I found Arch to be an amazing distribution. I loved the fact that the user was trusted to keep things running. Administrating the system was transparent with very little obfuscated automation. If something broke, I knew it was my fault, not some random script doing something unexpected. You even had to identify and install many dependencies yourself — packages would only list the deps that they need to run, not the deps required for all of their features to function.
This contrasted with my previous distro, Debian. If you installed a package that had a daemon, the daemon’s service would immediately be added and started. If you installed a package then every conceivable dependency would be installed along with it (one of my favourite apt-get flags was —no-recommends , especially when I was on a 2GB internet cap).
Today Arch still has some features that other distributions cannot beat it on. The Arch Linux wiki has become a defacto central hub for linux user documentation. The installation process is manual (especially useful given that I always break the assisted/gui installers). Luckily these plusses are mostly Arch agnostic: the wiki’s content is useful regardless of what distro you are running; and the installation instructions have taught me enough to be able to bootstrap other distributions as well.
My experiences with Arch Linux on both my desktop and my laptop have slowly gone downhill since my initial impressions. This post outlines why I want to move on and future posts will describe my journey migrating my desktop installation away from it. I plan to tackle my (somewhat more complicated) laptop another day.
Systemd
When I first installed Arch Linux it used sysvinit as its init system. Systemd was a fabled futuristic init system that promised speed-ups and parallelisation. I was excited about it and tried to install it, but returned to the sysvinit system after that.
Arch Linux migrated from sysvinit to systemd some years ago. The sysvinit system is no longer supported.
Systemd is now the primary reason I want to distro-hop away from Arch: over time my issues with it have slowly built up and I no longer want to tolerate it being a component of my system.
Background
Over the past few years systemd has become a core component of many Linux distributions. Now many popular distros will not function without systemd because their core packages depend on it.
It’s interesting that the Wikipedia page for systemd (at the time of writing) describes it as (only) an init system:
systemd is an init system used by some Linux distributions to bootstrap the user space and manage all processes subsequently, instead of the UNIX System V and Berkeley Software Distribution (BSD) init systems
I would say that’s quite inaccurate. ‘Init’ is one feature of the infinitely growing list of features that systemd provides. The freedesktop.org page has a better idea. I don’t expect you to read all this:
systemd is a suite of basic building blocks for a Linux system. It provides a system and service manager that runs as PID 1 and starts the rest of the system. systemd provides aggressive parallelization capabilities, uses socket and D-Bus activation for starting services, offers on-demand starting of daemons, keeps track of processes using Linux control groups, supports snapshotting and restoring of the system state, maintains mount and automount points and implements an elaborate transactional dependency-based service control logic. systemd supports SysV and LSB init scripts and works as a replacement for sysvinit. Other parts include a logging daemon, utilities to control basic system configuration like the hostname, date, locale, maintain a list of logged-in users and running containers and virtual machines, system accounts, runtime directories and settings, and daemons to manage simple network configuration, network time synchronization, log forwarding, and name resolution.
Over time this list has only got bigger and it appears that the project wants to continue expanding.
Bad experiences with systemd on my laptop
Disclaimer: my laptop’s processor architecture is armv7h, not x86. It’s a Series 2 Chromebook made by samsung .
I have had a lot of fun with this laptop. My /etc/issue file sums up the most interesting failures I’ve experienced:
Welcome to Clusterlizard, chamber \l. Please praise your selected deity(s)
Summary of interesting failure stories since December 2013:
- 'debug' on kernel command line -> systemd becomes very verbose -> random race conditions due to text output -> crash
- filesystem death on SSD by power-cycling too rapidly too many times whilst debugging other issues
- btrfs failure from trying to compile firefox ("ran out of inodes" even though btrfs does not use inodes in its design)
- systemd update -> required kernel support for xattrs on folders -> boot hung
- cracked the screen by carrying in my bag with Chinese take-away containers
- systemd update -> prevented kernel from loading userspace wifi firmware
- bootloader occasionally corrupts, fails to work until battled into a soft-reset after many attempts
- upgraded many packages, write-cache filled memory, OOM -> system hung, filesystem left inconsistent
- updated kernel -> post-install script dd'd bootloader over my root filesystem
Misc self-inflicted
- removed systemd-sysvcompat
- custom init system: spawned gettys with wrong arguments -> no logins
- repeat episodes of systemd firmware issues
Happy birthday Clusterlizard! (2014) (2015)Note that ‘Clusterlizard’ is the hostname of my laptop. I like giving my computers names interesting enough to remember.
I’ll go through the backstory for some of these issues, even though some of them are not relevant to systemd, because they’re still pretty entertaining.
The systemd verbosity issue was a well publicised problem that lead to some heated discussion on the linux kernel mailing lists:
http://lkml.iu.edu/hypermail/linux/kernel/1404.0/01327.html
http://lkml.iu.edu//hypermail/linux/kernel/1404.0/01331.html
Summary: if you had ‘debug’ on your kernel command line then systemd would pick this as a cue to also be verbose. Something to do with the sheer amount of messages being printed would cause crashes on bootup.
This bug inconsistently triggered for me and I was convinced that my hardware was flaky for some time. Eventually I discovered that the quirky unsigned version of u-boot (my bootloader) had a command-line interface that let me get rid of this kernel argument. This was not easy: the documentation for this variant was scarce and the ‘help’ command printed out several screen-heights of text without a way to scroll back up. Eventually I worked out that the variables ‘set’ in the environment included lists of commands for booting, and by luck the list of set variables fitted exactly into one screenheight. A bonus of fixing this problem was that I didn’t have to deal with the debug messages during runtime (which I did not want to see anyway!).
More fun was the issue where systemd refused to boot my system. I had recently updated systemd and the new copy wanted to use a feature (xattrs) that my kernel did not have. Rather than gracefully continuing, it would outright hang during boot.
For most of a year I held back from updating systemd on the laptop because the kernel verison in the Arch Linux ARM repositories was too old (and I didn’t want to compile one myself on this hardware). This lead to a whole pile of other packages being held back, until I was at a point where I could not update (nor easily install) anything.
This issue resolved itself and then repeated itself at a later point. The second time around firmware loading no longer functioned, breaking the wifi.
It turned out that the distro maintainers had abandoned my kernel’s package (linux-snow) and instead had moved onto another one that supported multiple variants of the ARM chromebooks (linux-peach). I discovered this by chance whilst searching through packages. I do not think I was alone in not knowing this as other users on the forums had the same issue.
Last year I went on a holiday to a remote mountainside village in Greece. Here internet was scarce and I had just discovered the replacement kernel package. “Ooh!” I went. “What could go wrong?”.
The new kernel package assumed a different partition layout to what I had. A post-installation script in it asked if it could ‘install’ onto my root partition. “Sure” I thought. I lost my root partition. Lesson: read other people’s scripts before trusting that you understand what they do.
A day’s work later at an internet cafe (does that even make sense?) I had a fresh install up and running. It’s worth noting that the wifi at internet cafes in rural villages in Greece is faster than wired internet in the suburbia of Australia.
Whilst at Greece I wrote my own init system to replace systemd on my laptop. It’s a shell script with some supporting utilities and it has been the best thing that I have ever done for my laptopping experience. Fast, simple and educational.
Systemd still remains installed as many programs will not function without libsystemd.so. Eventually I plan to hop away from Arch on this device too, but at the moment I have averted most of my problems with my own init.
Random stuff appearing in my boot sequence
Recently my desktop has been pausing for almost ten seconds with this item at bootup:
Feb 26 13:20:01 Shyarenris systemd[1]: Starting Manage, Install and Generate Color Profiles...
Feb 26 13:20:02 Shyarenris dbus[402]: [system] Successfully activated service 'org.freedesktop.ColorManager'
Feb 26 13:20:02 Shyarenris systemd[1]: Started Manage, Install and Generate Color Profiles.What? I don’t user colour profiles. I have never asked my computer to start a colour profiles service on bootup.
Someone has assumed that I want this and decided it should automatically be added. Congrats whoever you are, you’re now an honourary sysadmin of my home computer.
Hard dependencies in the last places you would expect
After removing the systemd package from my laptop one day, I discovered this gem:
I have no understanding of why pgrep would require systemd to function. I can plainly see it’s dynamically linked to libsystemd.so, but why I can’t fathom. I can also see the sanity oozing red from my tentacles. And circling the pigeons twice. Settling somewhere arid without overstops.
Apparently this is a widespread issue for users trying to move away from systemd. If you switch from udev (often/mostly/always integrated with systemd) to eudev (a fork) on Arch linux, then you will probably want to make use of the AUR package eudev-systemdcompat. Last time I looked this package simply provides the shared object files from the normal (x86) systemd packages. This ensures any programs with hard dependencies on systemd still function.
Seriously though. Grep. Systemd. What happens? A journalctl entry is made every time a user greps? Pgrep can dbus?
Purpose and direction
The fact systemd keeps obtaining new features scares me.
In real life: if I need something, I go and find it. I don’t wait for the product or service to come to me.
When the products or services does come to me, then I am very sceptical of it. Why has it come to me? Who is providing it? Why are they doing it?
In the case of systemd I feel that the project direction is costing me time and happiness. It should be a no-brainer to switch but until now my desire to just keep my computer running (and play games!) has been more than my desire to install a new operating system.
Coming soon
Part two will cover my experiences backing up my system and installing Void Linux, one of the (increasingly fewer) distros that does not use systemd. I’ll also write up a seperate post about the numerous things I’ve done on my ARM-based laptop (including my custom init).
Feel free to discuss (and disagree), just make sure you aware of the rules. If you are after some reading material that from the other side of the fence, Lennart Poettering (main systemd guy) has a blog.
I highly agree with everything in this article, that's why I'm using Void Linux. (PS: I'm not a spambot.)
I'm not convinced Jordyn! What is the sqrt( (-7) * (-7) )? How often do you type on the hands with your keyboard?
How are you going with Void? What do you like and dislike about the install process and day to day usage? Any particular personal dislikes?
(Note to self: engage with community better)
I can't comment much about systemd since my main distribution (GoboLinux) does not use it and I never had any need for it.
Poettering obviously attempts to make systemd his portfolio so systemd will continue to aggressively proliferate, not because
of any "technical need" but simply because systemd can be sold as a service. I assume that this is one huge reason why Red
Hat attempts to poison other linux distributions with it but the latter will be soon in the ~near future.
What to me is interesting is actually your comments about Archlinux. Many years ago, Archlinux was praised as something
for advanced users who do not want to compile everything from source (like via Gentoo, who also do not use systemd by
the way). Now with Archlinux having forced everyone into systemd, but also with other things such as the old creator
no longer being in charge (projects change with different people in charge), it seems as if they have gone away from
their old ideals.
I kept on hearing about VoidLinux and I think it is the "new" Archlinux, in spirit; possibly different enough to not
be called the "true" successor but they seem to have enough momentum going. Even folks in the ruby community like the
author of rack use VoidLinux now, so in some ways, VoidLinux because the new Archlinux with some similarities to
Gentoo. As for me, once a GoboLinuxer, always a GoboLinuxer, even though it'll remain a niche system. :)
Hales, I have no idea what 'sqrt( (-7) * (-7) )' even is. :p
Void is quite easy to install. the iso's are a bit dated so X.org doesn't work on my brand new hardware.
Void is quite nice, it's like arch back in it's glory days. the only 2 cons are that it doesn't have a equalizer or kde5 (yet).
Jordyn: what equaliser do you normally use? alsaequal?
Hales, I still use pulseaudio as I need dynamic audio output switching, and normally I use the LADSPA equalizer. Sure it's not as good as most other ones, but it works.
I also went the Void way until I found out that they cram non-free firmware down your troth without asking or letting you know.
Their latest stupidity goes far beyond the problems systemd generates in their so-called "We care about you" rant:
"Together with the statistics we’re starting to improve password security! Every Void system will get a PAM service that sends your password to one of our secured Amazon cloud servers and check it there against our password policies. We hope to use this information to help our users become as secure as they can be."
http://www.voidlinux.eu/news/2017/04/We-care-about-your-needs.html
That has got to be one of the lamest excuses for spyware that I have heard in a long time. If people accept this, then there is really something wrong in the world!
Jordyn: thankyou. I used to use alsaequal. I'm surprised that ALSA can't switch streams between outputs dynamically -- that just seems silly -- but I believe many of the design decisions of ALSA were based around performance, not
Erik: That's an April 1st post, don't panic :D
Bundling non-free firmware is done by many distros these days. I'm surprised that CPUs need firmware ('microcode'), but apparently that's a thing too.
In practice: remember that your computer is shipped with non-free firmware in it to begin with. Even if you use a distro without bundled NFF you will still be running the NFF that came in your computer. Arguably you could be running more if you use linux-nonfree, but likely overall a similar amount.
Woops, I need to finish my sentences.
... the design decisions of ALSA were based around performance, not being very flexible.
Regarding Erik's post Re: Void's "We Care About You" rant...
I didn't notice it at first, and was concerned about that.
Notice the date: It's an April Fools' joke.
Void has a nonfree repo, the firmware I couldn't care less about. And Erik, look at the date of that post.
> the only 2 cons are that it doesn't have a equalizer or kde5 (yet).
Why doesn't it have these things? Can't they be compiled even if they aren't in the official repositories? Pardon my noob questions!
I'm new to linux, trying to find a system-d free distro for my desktop and laptop. I'm leaning towards Gentoo for desktop, but laptop is old and I'm considering void linux.
Great write-up, thanks.
Thanks Seth.
>> the only 2 cons are that it doesn't have a equalizer or kde5 (yet).
>Why doesn't it have these things? Can't they be compiled even if they aren't in the official repositories? Pardon my noob questions!
Not at all.
Big desktop environments like KDE are a pain to compile. They have lots of deps, often lots of things to fix (related to the locations/names/etc of libraries in your specific distro), and at a minimum hours of compilation time. You can do it, but it's not fun, especially on laptops.
Even just applications that use Qt (the display toolkit used in KDE) are complex and big to compile. I don't think any GUI toolkits, apart from gtk-server (look this one up, it's pretty amazing) are nice on this front.
Equalisers: not sure. I think alsaequal is included with alsa plugins by default now (?). I have not had to use pulseaudio for years, and Void definitely has LADSPA/jack-compatible eqs (such as the nice parametric ones in 'calf') if you're into that.
Well, some guys like me try to have Arch without systemd. My projects use S6 supervision suite instead of systemd as init and service manager. This projects was started from more than 2 years now. Take a look on it, having Arch without system:D on it can be possible even for Kde, Xfce, Openbox,Jwm, Mate. An installer is also provided to install desktop or simply a minimal install without X server.
https://web.obarun.org
Oh dear this article is now a year old! The world of systemd has only gotten worse. I planned another writeup about the troubles it has caused me in the past year. Let's see if I can get my words together.
Eric: What is the community of Obarun like? If Obarun is 'Arch without systemd' then from a user's perspective it fills a similar niche to Void.