generic methodology to debug ordering cycles in systemd

You can visualise the cycle with the commands systemd-analyze verify , systemd-analyze dot and the GraphViz dot tool:

systemd-analyze verify default.target |&
perl -lne 'print $1 if m{Found.*?on\s+([^/]+)}' |
xargs --no-run-if-empty systemd-analyze dot |
dot -Tsvg >cycle.svg

You should see something like this:

enter image description here

Here you can see the cycle: c.service->b.service->a.service->c.service

Color legend: 
    black     = Requires
    dark blue = Requisite
    dark grey = Wants
    red       = Conflicts
    green     = After

Links:

  • systemd-analyze(1)
  • dot(1)

Is there a command which would demonstrate where this link is derived from?

The closest you can do is systemctl show -p Requires,Wants,Requisite,BindsTo,PartOf,Before,After cvol.service, which will show the resulting (effective) dependency lists for a given unit.

is there a command, which would find the cycles and show where each link in the cycle originates?

To my knowledge, there is no such command. Actually systemd offers nothing to aid in debugging ordering cycles (sigh).

According to journalctl, cvol.service wants basic.service, except that it doesn't, at least not obviously.

First, the requirement dependencies (Wants=, Requires=, BindsTo= etc.) are independent of ordering dependencies (Before= and After=). What you see here is an ordering dependency cycle, i. e. it has nothing to do with Wants= etc.

Second, there is a number of "default dependencies" created between units of certain types. They are controlled by DefaultDependencies= directive in the [Unit] section (which is enabled by default).

In particular, unless this directive is explicitly disabled, any .service-type unit gets implicit Requires=basic.target and After=basic.target dependencies, which is exactly what you see. This is documented in systemd.service(5).


step 1: run verification command for default.target

systemd-analyze verify default.target

step 2: observe which service or target mentioned in the message "systemd Breaking ordering cycle by deleting job" and display it's complete dependency list

systemctl show -p Requires,Wants,Requisite,BindsTo,PartOf,Before,After <service or target name mentioned in the "breaking cycle" message>

step 3: look at "after" and "before" groups inside the service or target file usually defined in

/lib/systemd/system

and find the services or targets well known to be sequential but are in outbound order for this one.

example:

dbus.service

is usually marked "after"

multi-user.target

but "before"

sockets.target

such dependency could be easily observed by calling

systemctl list-dependencies default.target

however if the file

/lib/systemd/system/dbus.service

contain lines like:

Before=multi-user.target

or

After=sockets.target

or both simultaneously, means dbus.service is defined outbound and it's causing an systemd endless cycle.

the cure is simple - change word "After" to "Before" and vice-versa if necessary.

Tags:

Systemd