This design document outlines the implementation of native systemd support in Ganeti by providing systemd unit files. It also briefly discusses the possibility of supporting socket activation.
Ganeti currently ships an example init script, compatible with Debian (and derivatives) and RedHat/Fedora (and derivatives). The initscript treats the whole Ganeti system as a single service wrt. starting and stopping (but allows starting/stopping/restarting individual daemons).
The initscript is aided by
daemon-util, which takes care of correctly
ordering the startup/shutdown of daemons using an explicit order.
Finally, process supervision is achieved by (optionally) running
ganeti-watcher via cron every 5 minutes.
ganeti-watcher will - among
other things - try to start services that should be running but are not.
The example initscript currently shipped with Ganeti will work with systemd’s LSB compatibility wrappers out of the box, however there are a number of areas where we can benefit from providing native systemd unit files:
systemd is the de-facto choice of almost all major Linux distributions. Since it offers a stable API for service control, providing our own systemd unit files means that Ganeti will run out-of-the-box and in a predictable way in all distributions using systemd.
systemd performs constant process supervision with immediate service restarts and configurable back-off. Ganeti currently offers supervision only via ganeti-watcher, running via cron in 5-minute intervals and unconditionally starting missing daemons even if they have been manually stopped.
systemd offers socket activation support, which may be of interest for use at least with masterd, luxid and noded. Socket activation offers two main advantages: no explicit service dependencies or ordering needs to be defined as services will be activated when needed; and seamless restarts / upgrades are possible without rejecting new client connections.
systemd offers a number of security features, primarily using the Linux kernel’s namespace support, which may be of interest to better restrict daemons running as root (noded and mond).
We propose to extend Ganeti to natively support systemd, in addition to
shipping the init-script as is. This requires the addition of systemd
unit files, as well as some changes in daemon-util and ganeti-watcher to
systemctl on systems where Ganeti is managed by systemd.
Systemd uses unit files to store information about a service, device,
mount point, or other resource it controls. Each unit file contains
exactly one unit definition, consisting of a
Unit an (optional)
Install section and an (optional) type-specific section (e.g.
Service). Unit files are dropped in pre-determined locations in the
system, where systemd is configured to read them from. Systemd allows
complete or partial overrides of the unit files, using overlay
directories. For more information, see systemd.unit(5).
We will create one systemd service unit per daemon (masterd, noded,
mond, luxid, confd, rapi) and an additional oneshot service for
ganeti-common.service). All services will
ganeti-common.service, which will thus run exactly once per
transaction (regardless of starting one or all daemons).
All daemons will run in the foreground (already implemented by the
-f flag), directly supervised by systemd, using
Restart=on-failure in the respective units. Master role units will
EXIT_NOTMASTER as a successful exit and not trigger
restarts. Additionally, systemd’s conditional directives will be used to
avoid starting daemons when they will certainly fail (e.g. because of
Apart from the individual daemon units, we will also provide three target units as synchronization points:
ganeti-node.target: Regular node/master candidate functionality, including
ganeti-master.target: Master node functionality, including
ganeti.target: A “meta-target” depending on
ganeti.targetitself will be
multi-user.target, so that Ganeti starts automatically on boot.
To allow starting/stopping/restarting the different roles, all units
will include a
PartOf directive referencing their direct ancestor
target. In this way
systemctl restart ganeti-node.target or
restart ganeti.target will work as expected, i.e. restart only the node
daemons or all daemons respectively.
The full dependency tree is as follows:
ganeti.target ├─ganeti-master.target │ ├─ganeti-luxid.service │ │ └─ganeti-common.service │ ├─ganeti-masterd.service │ │ └─ganeti-common.service │ └─ganeti-rapi.service │ └─ganeti-common.service └─ganeti-node.target ├─ganeti-confd.service │ └─ganeti-common.service ├─ganeti-mond.service │ └─ganeti-common.service └─ganeti-noded.service └─ganeti-common.service
The systemd unit files will be built from templates under
doc/examples/systemd, much like what is currently done for the
initscript. They will not be installed with
make install, but left
up to the distribution packagers to ship them at the appropriate
Systemd automatically creates a service for each SysV initscript on the
.service to the initscript name, except if a
service with the given name already exists. In our case however, the
initscript’s functionality is implemented by
Systemd provides the ability to mask a given service, rendering it
unusable, but in the case of SysV services this also results in
failure to use tools like
service. Thus we have
to ship a
/bin/true) of type
oneshot, that depends on
ganeti.target for these tools to
continue working as expected.
ganeti.target on the other hand will
be marked as
PartOf = ganeti.service for stop and restart to be
propagated to the whole service.
ganeti.service unit will not be marked to be enabled by systemd
(i.e. will not be started at boot), but will be available for manual
invocation and only be used for compatibility purposes.
daemon-util is used wherever daemon control is required:
In the sample initscript, to start and stop all daemons.
ganeti.backendto start the master daemons on master failover and to stop confd when leaving the cluster.
ganeti.bootstrap, to start the daemons on cluster initialization.
ganeti.cli, to control the daemon run state during certain operations (e.g. renew-crypto).
daemon-util uses two auxiliary tools for managing daemons
daemon, in this order of preference. In
order not to confuse systemd in its process supervision,
will have to be modified to start and stop the daemons via
in preference to
will require a basic check against run-time environment integrity:
Make sure that
systemdruns as PID 1, which is a simple check against the existence of
systemdknows how to handle Ganeti natively. This can be a check against the
Unless both of these checks pass,
daemon-util will fall back to its
Systemd offers support for socket activation. A daemon supporting socket-based activation, can inherit its listening socket(s) by systemd. This in turn means that the socket can be created and bound by systemd during early boot and it can be used to provide implicit startup ordering; as soon as a client connects to the listening socket, the respective service (and all its dependencies) will be started and the client will wait until its connection is accepted.
Also, because the socket remains bound even if the service is restarting, new client connections will never be rejected, making service restarts and upgrades seamless.
Socket activation support is trivial to implement (see sd_listen_fds(3)) and relies on information passed by systemd via environment variables to the started processes.