Translation(s): English - Español - Français - Italiano - 한국어 - Russian - Brasileiro - 简体中文

systemd - system and service manager

systemd is a system and service manager for Linux. It is the default Init system for Debian since Debian 8 ("jessie").

Its features include:

Systemd runs as a daemon with PID 1, but it is not monolithic: many of its features are optional and provided by other processes.

Contents

  1. systemd - system and service manager
    1. Units
    2. Targets
    3. Sockets
  2. Getting information on system status
    1. System status
    2. Starting, stopping, enabling and disabling services
    3. Creating or altering Units
  3. Security Hardening
    1. Showing the hardening settings in use
    2. Which settings to use
      1. Restricting Filesystem access
      2. General settings
      3. Restricting Capabilities
      4. System calls
    3. Security hardening and oneshot services
  4. Debugging
    1. Failed services
      1. Failed units
      2. Verify units
      3. systemd hangs on startup or shutdown
      4. Kernel debug without systemd debug in Jessie
  5. See also
    1. Bugs and Bug-Tracking-Systems
    2. Known Issues and Workarounds
      1. Shared bind mounts
      2. SSH session doesn't cleanly terminate on reboot/shutdown
      3. Missing startup messages on console(tty1) after the boot
      4. Virtual and serial console changes
      5. Orphaned processes
    3. Where to get help?
    4. Replacing systemd with another init system
    5. Debian Resources
    6. Other Resources

Units

Systemd manages every service with a unit. For example, SSH is controlled by ssh.service.

The unit determines the service by specifying:

Targets

There are several different types of unit to manage services, mount points, devices, sockets, or timers. See systemd.unit(5) for information. For example, Targets are used to start groups of units: at boot time, systemd simply activates the target default.target, which is usually an alias for another target such as graphical.target, and this causes all services needed to be started without the user needing to specify the order explicitly.

Sockets

Systemd can create and manage sockets(2) used for communication between services. For example, if a service uses a socket to communicate with client processes, then only socket needs to be created on system boot: the service itself can be started on demand, when the first connection is made. This allows systemd to start multiple client services that need the socket in parallel with the server, and to restart crashed servers without losing client connections (since the kernel will buffer data). Please see the upstream systemd page for more information.

Getting information on system status

systemctl is the main tool used to see and control the state of system. You can use it to enable or disable services, either permanently or just for the current session. See systemctl(1) for more details.

System status

Show all running services and the overall system status: 

$ systemctl status

List all running services: 

$ systemctl

List any failed units: 

$ systemctl --failed

Display the service for "example1": 

# systemctl cat example1

List all available units: 

$ systemctl list-unit-files

Starting, stopping, enabling and disabling services

Start the service "example1" immediately: 

# systemctl start example1

Stop the service "example1" immediately: 

# systemctl stop example1

Restart the service "example1" immediately: 

# systemctl restart example1

Shows the status of the service "example1": 

# systemctl status example1

Make "example1" be started on the next system boot: 

# systemctl enable example1

Remove "example1" from the services to be started on the next boot: 

# systemctl disable example1

Creating or altering Units

See Working with Units for information about editing existing units and systemd/Services for information about writing your own services.

Security Hardening

A significant benefit of systemd is the ability to restrict the access that services are granted to the system. Because most services need to use some features only available to root, a single compromised service can often allow an attacker to take over the whole system. Systemd gives you several options that can be used to limit what a service can do. This does not mitigate all issues: proper security still needs people to design services securely and implement the design correctly.

These hardening features include:

Showing the hardening settings in use

The following command shows a list of all units and rates them based on the hardening directives used (a lower score is better): 

systemd-analyze security

The following command shows the details of the ssh service: 

systemd-analyze security ssh.service

Generally you want to decrease the numbers as much as possible, without breaking functionality.

Which settings to use

If the service is controlling a traditional daemon (for example, a long-running program that runs in the background and which does not need to access files other than its configuration in /etc) you can usually use several hardening features. If it is a 'oneshot' program that runs on-demand and then exits there are some unfortunate issues with sending email that limit what can be used (see below).

Restricting Filesystem access

You can block services from accessing specific directories.

The following is recommended for every service that doesn't directly interact with user files. For examples, most services do not need to access any files in user home directories by default: 

ProtectHome=true

The following setting means each service sees its own version of /tmp, and cannot see files in /tmp created by other services or users. This can prevent a service being attacked by a user or another service via /tmp race conditions. It's unlikely to cause any problems: 

PrivateTmp=true

General settings

The following settings are good for protecting the overall OS installation. You can use usually use these: 

ProtectKernelLogs=true
ProtectControlGroups=true
ProtectKernelModules=true
MemoryDenyWriteExecute=true
ProtectHostname=true
LockPersonality=true
RestrictRealtime=true
DevicePolicy=closed
ProtectClock=true
RestrictSUIDSGID=true
ProtectKernelTunables=true
PrivateDevices=true

The following is good for daemons which need no networking. This allows Unix domain sockets by filesystem (EG /run/$DAEMON/whatever.sock) but not the abstract socket namespace. The risk of this is PAM services that use networking or other indirect network access. This is a good option for the sysadmin who is locking down their own system but risky for the package maintainer. 

PrivateNetwork=true

The NoNewPrivileges option prevents a daemon from running a SETUID program to gain privileges that are undesired. But there are some corner cases for this such as running a program that has a SE Linux automatic domain transition to LESS privileges can be denied. This cannot be used with services that want to send email via exim. 

NoNewPrivileges=true

Restricting Capabilities

The CapabilityBoundingSet controls the capabilities that a service can use. There is a full list of capabilities with their descriptions in the file /usr/include/linux/capability.h in the linux-libc-dev package, and some limited documentation in capabilities(7). Unfortunately, working out which capabilities a service needs is difficult: if something doesnt work, you blocked too much, but usually the logs don't indicate what was missing or even what capability was missing.

You can use an allowlist (space separated on a single line) or blocklist (prefix with a ~). Usually it's best to just list the capabilities that are to be permitted as it's a smaller set.

Here is an example of allowing a daemon to use DAC_OVERRIDE (allows a UID 0 process to read files even if the filesystem permissions would deny access), SETUID and SETGID (for dropping privileges), PTRACE, and memory locking. All other capabilities will be denied: 

CapabilityBoundingSet=CAP_DAC_OVERRIDE CAP_SETGID CAP_SETUID CAP_SYS_PTRACE CAP_IPC_LOCK

System calls

Systemd can restrict access to individual system calls. Filtering system calls has some overlap with restricting capabilities but it's good to restrict things in two ways. It usually isn't difficult to map error messages logged by a daemon to which groups of syscalls were filtered and then do a binary search for the most restrictive options.

The following example blocks access to some system calls that are known to be risky (such as the @clock group and the @mount group), some system calls that aren't usually needed (such as @cpu-emulation and @raw-io). 

SystemCallFilter=~@mount @cpu-emulation @debug @raw-io @reboot @resources @swap @module @obsolete @clock

Security hardening and oneshot services

(more to be added) If you have a Type=?OneShot service that wants to send email via sendmail or similar then only a few hardening settings can be used.

The following works with debian's mta in their default configuration - but the capabilities are risky! 

Type=oneshot
ExecStart=/usr/bin/myprog

# you _need_ to add this to allow the mail to be delivered by exim4 -
# otherwise it sits in the queue
ExecStart=sleep 10s

# Hardening options for this unit are limited because anything setting
# NoNewPrivilages will break exim's ability to deliver the email: exim
# wants to change user to transmit and deliver the email :(

# (If User=root is set then some of the options marked 'cannot set'
#  do become settable - but may not have any effect)

#User=root
# cannot set: DynamicUser=yes
# cannot set: PrivateUsers=true


ProtectSystem=strict

StateDirectory=whatever

# exim needs to spool and log the email
ReadWritePaths=-/var/spool -/var/mail -/var/log


## exim allows you to disable these next two ReadWritePaths
# but other mta may want to write other dirs (eg: courier needs /var/lib/courier)
ReadWritePaths=-/var/lib
# some mta deliver to /home/<user>/Maildir (may be needed for bounce messages)
# (for exim, ProtectHome=read-only is enough)
ProtectHome=false

PrivateTmp=true
PrivateMounts=true
# cannot set: PrivateDevices=true (unless we also set User=root)
# even with PrivateDevices, some of these four are needed or exim's log does not contain the '<=' and '=>' lines
# DevicePolicy=closed
DevicePolicy=strict
DeviceAllow=/dev/stdout w
DeviceAllow=/dev/stdin r
DeviceAllow=/dev/stderr w
DeviceAllow=/dev/null rw

ProtectProc=invisible
ProcSubset=pid

RemoveIPC=true

ProtectControlGroups=true
# cannot set: RestrictNamespaces=true
# cannot set: LockPersonality=true

# cannot set: ProtectKernelTunables=true
# cannot set: ProtectKernelModules=true
# cannot set: ProtectKernelLogs=true

# cannot set: ProtectHostname=true
# cannot set: ProtectClock=true
# cannot set: RestrictRealtime=true
# cannot set: MemoryDenyWriteExecute=true

# do not want: IPAddressDeny=any
# do not want: PrivateNetwork=true
# do not want, and cannot set: RestrictAddressFamilies=none

# cannot set: NoNewPrivileges=true
# cannot set: RestrictSUIDSGID=true

# cannot set: NoNewPrivileges=true
# cannot set: RestrictSUIDSGID=true
AmbientCapabilities=

# exim (and most mta) works with just these: needs to change ownership of mail-related files
CapabilityBoundingSet=CAP_SETGID
CapabilityBoundingSet=CAP_SETUID
CapabilityBoundingSet=CAP_FSETID
CapabilityBoundingSet=CAP_CHOWN
CapabilityBoundingSet=CAP_DAC_OVERRIDE
CapabilityBoundingSet=CAP_FOWNER

# cannot set: SystemCallArchitectures=native

# cannot set: SystemCallFilter=@system-service
# cannot set (mail is frozen): SystemCallFilter=~@privileged
# cannot set: SystemCallFilter=~@resources

Debugging

Failed services

Failed units

In some cases units enter a failed state. The statuscommand can be used to find out some details: 

$ systemctl status <UNITNAME>

Failed units can be manually cleared out: 

# systemctl reset-failed

Verify units

To check that we didn't misspell anything or used any unknown options when writing units, systemd-analyze 

$ systemd-analyze verify <UNITNAME-or-PATHTOUNIT>

No output means everything checks out.

systemd hangs on startup or shutdown

Sometimes it is necessary to investigate why systemd hangs on startup or on reboot/shutdown.

Solution #0: Remove "quiet" from Kernel command line (so called "cmdline" or "grub line")

Solution #1: Increase verbosity via cmdline: Add "systemd.log_target=kmsg systemd.log_level=debug"

Of course you can have a "temporary" persistent solution: 

[ /etc/default/grub ]
GRUB_CMDLINE_LINUX="systemd.log_target=kmsg systemd.log_level=debug" <--- Add here (by uncommenting you can easily switch to debug)

# update-grub

Solution #2: Increase verbosity via /etc/systemd/system.conf 

LogLevel=debug           <--- Uncomment this line and use "debug" (default: commented and "info")
LogTarget=syslog-or-kmsg <--- Uncomment this line (default: commented)

Solution #3: Boot an emergency shell: Add systemd.unit=rescue.target or just 1 (the number one) to the kernel command line.

Solution #4: Enable the debug shell: Run systemctl enable debug-shell.service. (You can do this in a chroot environment after booting a rescue system.) This starts a root shell on TTY 9.

HINT: "man systemd" and "man systemd-system.conf"

HINT: Extensive debugging information about systemd is on this FreeDesktop page.

HINT: How to check Kernel command line parameters/options? 

# cat /proc/cmdline

NOTE on LogLevel (see systemd(1) and systemd-system.conf(5)):

"Set log level. As an argument this accepts a numerical log level or the well-known syslog(3) symbolic names (lowercase): emerg, alert, crit, err, warning, notice, info, debug."

HINT: Keep a copy of /sbin/init from sysvinit package in case of rescue (so you can use init=/sbin/init.sysvinit in cmdline)! 

# cp -av /sbin/init /sbin/init.sysvinit <--- Before installing systemd-sysv package

See also https://fedoraproject.org/wiki/How_to_debug_Systemd_problems

Kernel debug without systemd debug in Jessie

Using the old "debug" kernel parameter in Jessie will turn on systemd debug logging as well as kernel debug logging. To get the old behaviour, do not use "debug", instead use the kernel parameter "loglevel=7".

See also

Bugs and Bug-Tracking-Systems

Known Issues and Workarounds

Shared bind mounts

The default behavior of bind mounts changes under systemd. The Linux kernel makes bind mounts of anything below / PRIVATE. Systemd changes this to SHARED.

Thus, when you do this: 

    mount --bind / $CHROOT
    mount --bind /dev/ $CHROOT/dev
    umount $CHROOT/dev

then /dev will be unmounted in your base/parent system as well!

What you can do now instead, is to: 

    mount --bind --make-rslave / $CHROOT
    mount --bind --make-rslave /dev/ $CHROOT/dev

this will propagate mount changes (also mount options) in the base/parent system into the $CHROOT but not from the $CHROOT back to the parent.

The rationale for the change of the default behavior can be found in bug 739593, particularly in Lenart's comment therein.

SSH session doesn't cleanly terminate on reboot/shutdown

If you happen to reboot/shutdown remote machine over ssh you may find out that your session isn't terminated properly, leaving you with the non-reacting terminal until a long timeout has been reached. There was bug 751636 about it. A workaround to this problem was to install: 

    apt-get install libpam-systemd

which would terminate the ssh session before the network was dropped. Please note, that that would require PAM to be enabled in sshd.

Missing startup messages on console(tty1) after the boot

With systemd console(tty1) is handled differently and if you used to check it to see how did your boot go now you'll see only couple of non-informative lines.

To be able to get full transcript of the system boot on your console you need to perform two steps.

1. Add to the kernel options systemd.show_status=1, for example via /etc/default/grub: 

    GRUB_CMDLINE_LINUX_DEFAULT="quiet systemd.show_status=1"

and run update-grub2.

2. Create file /etc/systemd/system/getty@tty1.service.d/noclear.conf with the content: 

[Service]
TTYVTDisallocate=no

to disable clearing of the terminal on getty invocation.

Virtual and serial console changes

Those used to change inittab to enable/disable virtual or serial consoles will notice that that file is gone from clean installs. This is all managed through systemd directly now. For example, you can enable a serial console on COM1 with: 

systemctl enable serial-getty@ttyS0.service
systemctl start serial-getty@ttyS0.service

However, it is generally preferable to add console=ttyS0 on the kernel commandline, since this also enables kernel output on reboots. For, e.g. GRUB, this is done by adding the following to /etc/default/grub: 

GRUB_CMDLINE_LINUX="console=ttyS0"

... and running update-grub. This will take effect only on the next reboot, however.

Note also that Linux supports multiple consoles for output - but only one for input - the last named (or default if none named) from the kernel commandline specifies which is used for input, and all are used for output. E.g. for GRUB: 

GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0"

That will, after running update-grub and rebooting, have console input from /dev/ttyS0 and output to both that and /dev/tty0 device. Note also that serial parameters can also be specified, e.g.: 

GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,9600n8"

Orphaned processes

Because it manages user sessions (taking over the role of X or other components), systemd may slightly change how processes survive a logoff. By default, when X shuts down all processes should exit and systemd will clean up the session, but there are some corner cases where certain processes don't cleanup after themselves properly.

You can configure how systemd manages leftover processes with the KillUserProcesses= parameter in logind.conf. By setting this to yes, processes will be forcibly killed when the session terminates. Note that this will break tools like screen or tmux, unless they are configured to run under a distinct user@.service unit and if enable-linger is set to yes in loginctl. A simple way to do this on the fly is to run the program in a "transient scope", using systemd-run: 

systemd-run --scope --user screen

Now, normally sessions should cleanup after themselves, and such issues should be fixed without having to revert to the KillUserProcesses=yes sledgehammer. A good way to list the affected processes is to group them by "control group", with the systemd-cgls command: 

systemd-cgls

Some known misbehaving applications:

Where to get help?

Systemd is a young project with a strong emphasis on solving problems in a distribution agnostic manner.

Debian specific channels include

Most other distributions also use systemd

Replacing systemd with another init system

See Init, under "Changing the init system - at installation time"

Debian Resources

Other Resources

CategoryPermalink ?CategorySystemd