Rationale for switching from upstart to systemd?

Layman users shouldn't notice any change, by design. It's an init system, not something users traditionally interact with. It should completely replace the functionality provided by Upstart —and do a few extra things— but the only time a non-technical user will see this is when it goes wrong.

Users, sysadmins and developers who have been actively using and developing for Upstart are the people who need to address things. There is a migration document on the Ubuntu Wiki to help developers convert their init scripts, but users and sysops can carry on using Upstart by sticking with 14.04 (which is supported until 2019).

The reason and rationale for change really wasn't from Ubuntu's side. Canonical were happy enough with Upstart (their project) but many Debian users wanted to move to a modern init engine to get better concurrency at boot and better monitoring functionality across all services.

That meant a fight between various options (the rationales) and systemd eventually won.

Canonical went along with Debian because it's easiest and probably best. They get to drop a project and aren't fighting upstream. It also brings us in line with other distributions (Red Hat, Fedora, etc) who are also moving to systemd. More focus and less duplication of effort.

tl;dr To a non technical person, this shouldn't affect you at all. For Ubuntu it should mean less work and a better init system.


Could anyone adequately explain to a non-technical user how and if this affect us at all?

In theory, this shouldn't affect the non-technical end user who doesn't get involved in the nitty gritty of how the system actually works. In practice, there are a lot of things that you're going to see.

Here's an incomplete list:

  • If you had add-on softwares that used upstart job definition files for starting programs, they'll stop working. You'll have to install (and possibly write but more commonly just nick off someone else who has already written) systemd service unit files. Example: https://askubuntu.com/questions/613785
  • Various design assumptions by systemd developers about things like power management result in defaults that are at odds with what you might have become used to. The systemd developers have very definite ideas about what should happen in response to the the lid switches on laptops, for example.
  • If you are using the nvidia proprietary display driver, then there are various design decisions in systemd that affect you. Example: https://askubuntu.com/questions/613773
  • It's not really relevant when coming from upstart, as Ubuntu users have had a manual page telling them this for some years now, but I mention it for the non-Ubuntu users who might read this: Other Linux operating systems users coming from System 5 init+rc are bitten by the fact that systemd is only backwards compatible with System 5 rc. Like upstart, and indeed most other systems, it professes, and supplies, no backwards compatibility with System 5 init and its configuration file /etc/inittab.

    So people who followed the advice from from the 30-some years of people advising "Well, you can just edit this into /etc/inittab …", or people who use softwares that followed that advice now have softwares that don't start at bootstrap. Example: https://unix.stackexchange.com/a/196197/5132

  • You cannot get to single-user mode via the systemd shutdown command, as you could with previous shutdown commands. Aside from the fact that it's called rescue mode in systemd jargon, rescue mode is not considered a shut down state in the systemd worldview. It's regarded as a running state. shutdown now will power off the machine. It's systemctl rescue to reach single-user mode in the systemd world. Further reading: https://unix.stackexchange.com/a/196471/5132
  • Further to that last subject: If you haven't thrown away the idea of run levels already, now is the time to do so. Futher reading: https://unix.stackexchange.com/a/196014/5132
  • You're going to have to be careful about following general systemd advice found by random WWW browsing, because you "know" that "it's all systemd now". You're going to see people talking about running commands with the --user option to systemctl. That doesn't apply to Ubuntu (yet). upstart and systemd differ significantly in this area, and Ubuntu version 15 still uses the upstart per-session init rather than a systemd per-user instance. So https://superuser.com/a/860598/38062 won't apply, for example. ☺

As others have already noted here, in theory, this shouldn't affect the non-technical end user - and in theory there is no difference between theory and practice but in practice there is.

Clarification

I think few things posted here need some clarification:

It's an init system, not something users traditionally interact with.

It was the case with SysV init and with Upstart but it's not the case with systemd any more. It does a lot of things that users traditionally interact with:

  • https://cgit.freedesktop.org/systemd/systemd/tree/NEWS

It should completely replace the functionality provided by Upstart —and do a few extra things

Two things to clarify - first about completely replacing Upstart:

No SysV init scripts

One of the issues that people have with systemd is that it doesn't run SysV init scripts. So there's one example that it doesn't completely replace the functionality provided by Upstart.

This is something that we could rely on for over 30 years and traditionally you wrote SysV init scripts for maximum portability without repeating yourself (by writing multiple versions of the same scripts), which is not the case any more.

This shouldn't be a problem when using only packages from official repositories because presumably all the packages that used to have either SysV init or Upstart scripts would need to have their scripts rewritten before they get packaged.

It will be only a problem for people who happen to use any third-party or custom software that have their init scripts written for either SysV init or for Upstart and those will need the init scripts rewritten before upgrading to a system with systemd (or get the upstart installed, which is also an option, or migrate to a system that doesn't use systemd).

There is systemd-sysv-generator that is supposed to automatically translate SysV init scripts to systemd scripts but there are some bugs and a long list of explicit incompatibilities.

Now, the second clarification - about those few extra things:

Few extra things

Those "few extra things" that systemd is going to cover - according to A Perspective for systemd - What Has Been Achieved, and What Lies Ahead presentation by Lennart Poettering in 2014 at GNOME.asia - are the following:

  • init system
  • journal logging
  • login management
  • device management
  • temporary and volatile file management
  • binary format registration
  • backlight save/restore
  • rfkill save/restore
  • bootchart
  • readahead
  • encrypted storage setup
  • EFI/GPT partition discovery
  • virtual machine/container registration
  • container management
  • hostname management
  • locale management
  • time management
  • random seed management
  • sysctl variable management
  • console managment
  • introspection
  • auto discovery
  • plug and play
  • network management
  • systemd-networkd
  • DNS cache
  • mDNS responder
  • LLMNR responder
  • DNSSEC verification
  • IPC in the kernel
  • kdbus
  • sd-bus
  • time synchronisation with NTP
  • systemd-timesyncd
  • integration with containers
  • sandboxing of services
  • sandboxing of apps
  • OS image format
  • Container image format
  • App image format
  • GPT with auto-discovery
  • Stateless systems
  • instantiatable systems
  • factory reset
  • node initialisation and updates
  • integration with the cloud
  • service management across nodes
  • verifiable OS images all the way to the firmware
  • Boot Loading
  • Building the Internet’s Next Generation OS Unifying pointless differences between distributions

So going back to: "It's an init system, not something users traditionally interact with." - it has to be pointed out that the init system is just one item on that list.

And finally, the last thing I'd like to comment:

[T]he only time a non-technical user will see this is when it goes wrong.

Oh, what a relief. :)

Changes

Most notable changes for end users (other than the scripts themselves) is starting and stopping services and using commands like:

  • screen
  • tmux
  • nohup

which no longer work as expected. For example, nohup is a POSIX command to make sure that the process keeps running after you log out from your session. It no longer works on systemd. Also programs like screen and tmux need to be invoked in a special way or otherwise the processes that you run with them will get killed (while not getting those processes killed is usually the main reason of running screen or tmux in the first place).

This is not a bug, it is a design choice, so it is not likely to get fixed in the future. This is what Lennart Poettering has said about this issue:

In my view it was actually quite strange of UNIX that it by default let arbitrary user code stay around unrestricted after logout. It has been discussed for ages now among many OS people, that this should possible but certainly not be the default, but nobody dared so far to flip the switch to turn it from a default to an option. Not cleaning up user sessions after logout is not only ugly and somewhat hackish but also a security problem. systemd 230 now finally flipped the switch and finally by default cleans everything up correctly when the user logs out.

For more info see:

  • Systemd Starts Killing Your Background Processes By Default
  • Systemd v230 kills background processes after user logs out, breaks screen, tmux
  • Debian Bug #825394: systemd kill background processes after user logs out

Running screen

  • upstart: screen
  • systemd: systemd-run --user --scope screen

(Note: the behavior of "upstart" above is really anything except systemd, this is not upstart specific)

Starting job foo:

  • upstart: start foo
  • systemd: systemctl start foo

Stopping job foo:

  • upstart: stop foo
  • systemd: systemctl stop foo

Restarting job foo:

  • upstart: restart foo
  • systemd: systemctl restart foo

Listing jobs with their status:

  • upstart: initctl list
  • systemd: systemctl status

(See my answer to What are the pros/cons of Upstart and systemd? for more details that are out of the scope for this question.)

Logs

There is also a big difference in handling the logs because contrary to the Unix tradition the logs of systemd are stored in binary files in a custom format, so instead of:

cat /var/log/upstart/foo.log
tail -f /var/log/upstart/foo.log

you need to use special commands to access your logs:

sudo journalctl -u foo
sudo journalctl -u foo -f

Controversies

The introduction of systemd first to Debian and later to Ubuntu was not without controversies and vast opposition as is known to anyone who wrote one of the following articles:

  • Meet Devuan, the Debian fork born from a bitter systemd revolt (PCWorld)
  • Grab your pitchforks: Ubuntu to switch to systemd on Monday (The Register)
  • Linus Torvalds and others on Linux's systemd (ZDNet)
  • About the systemd controversy (Errata Security)

The official Debian position on systemd and the resulting controversy has led to the Exodus declaration in 2014 and ended with Ian Jackson's resignation.

The Init Freedom, Without-Systemd.org and Systemd-Free.org initiatives were born, with a lot of discussion on Hacker News.

Further reading

  • Sad news today: systemd-resolved to be deployed in Ubuntu 16.10 by Paul Wouters
  • Arguments against systemd on Without Systemd
  • List of incompatibilities of systemd with SysV (official docs)
  • Re-enable Upstart on Ubuntu 15.04 by Derrik Diener
  • systemd sysv init compatibility mode: how it works and troubleshooting when it breaks
  • What are the pros/cons of Upstart and systemd? on unix.stackexchange.com

Tags:

Upstart

Systemd

15.04

Recent Posts