generic methodology to debug ordering cycles in systemd
You can visualise the cycle with the commands systemd-analyze verify
, systemd-analyze dot
and the GraphViz dot
tool:
systemd-analyze verify default.target |&
perl -lne 'print $1 if m{Found.*?on\s+([^/]+)}' |
xargs --no-run-if-empty systemd-analyze dot |
dot -Tsvg >cycle.svg
You should see something like this:
Here you can see the cycle: c.service->b.service->a.service->c.service
Color legend:
black = Requires
dark blue = Requisite
dark grey = Wants
red = Conflicts
green = After
Links:
- systemd-analyze(1)
- dot(1)
Is there a command which would demonstrate where this link is derived from?
The closest you can do is systemctl show -p Requires,Wants,Requisite,BindsTo,PartOf,Before,After cvol.service
, which will show the resulting (effective) dependency lists for a given unit.
is there a command, which would find the cycles and show where each link in the cycle originates?
To my knowledge, there is no such command. Actually systemd offers nothing to aid in debugging ordering cycles (sigh).
According to journalctl, cvol.service wants basic.service, except that it doesn't, at least not obviously.
First, the requirement dependencies (Wants=
, Requires=
, BindsTo=
etc.) are independent of ordering dependencies (Before=
and After=
). What you see here is an ordering dependency cycle, i. e. it has nothing to do with Wants=
etc.
Second, there is a number of "default dependencies" created between units of certain types. They are controlled by DefaultDependencies=
directive in the [Unit]
section (which is enabled by default).
In particular, unless this directive is explicitly disabled, any .service
-type unit gets implicit Requires=basic.target
and After=basic.target
dependencies, which is exactly what you see. This is documented in systemd.service(5).
step 1: run verification command for default.target
systemd-analyze verify default.target
step 2: observe which service or target mentioned in the message "systemd Breaking ordering cycle by deleting job" and display it's complete dependency list
systemctl show -p Requires,Wants,Requisite,BindsTo,PartOf,Before,After <service or target name mentioned in the "breaking cycle" message>
step 3: look at "after" and "before" groups inside the service or target file usually defined in
/lib/systemd/system
and find the services or targets well known to be sequential but are in outbound order for this one.
example:
dbus.service
is usually marked "after"
multi-user.target
but "before"
sockets.target
such dependency could be easily observed by calling
systemctl list-dependencies default.target
however if the file
/lib/systemd/system/dbus.service
contain lines like:
Before=multi-user.target
or
After=sockets.target
or both simultaneously, means dbus.service is defined outbound and it's causing an systemd endless cycle.
the cure is simple - change word "After" to "Before" and vice-versa if necessary.