Wunschkonzert, Ponyhof und Abenteuerspielplatz

Here's the ninth installment of my ongoing series on systemd for Administrators:

On /etc/sysconfig and /etc/default

So, here's a bit of an opinion piece on the /etc/sysconfig/ and /etc/default directories that exist on the various distributions in one form or another, and why I believe their use should be faded out. Like everything I say on this blog what follows is just my personal opinion, and not the gospel and has nothing to do with the position of the Fedora project or my employer. The topic of /etc/sysconfig has been coming up in discussions over and over again. I hope with this blog story I can explain a bit what we as systemd upstream think about these files.

A few lines about the historical context: I wasn't around when /etc/sysconfig was introduced -- suffice to say it has been around on Red Hat and SUSE distributions since a long long time. Eventually /etc/default was introduced on Debian with very similar semantics. Many other distributions know a directory with similar semantics too, most of them call it either one or the other way. In fact, even other Unix-OSes sported a directory like this. (Such as SCO. If you are interested in the details, I am sure a Unix greybeard of your trust can fill in what I am leaving vague here.) So, even though a directory like this has been known widely on Linuxes and Unixes, it never has been standardized, neither in POSIX nor in LSB/FHS. These directories very much are something where distributions distuingish themselves from each other.

The semantics of /etc/default and /etc/sysconfig are very losely defined only. What almost all files stored in these directories have in common though is that they are sourcable shell scripts which primarily consist of environment variable assignments. Most of the files in these directories are sourced by the SysV init scripts of the same name. The Debian Policy Manual (9.3.2) and the Fedora Packaging Guidelines suggest this use of the directories, however both distributions also have files in them that do not follow this scheme, i.e. that do not have a matching SysV init script -- or not even are shell scripts at all.

Why have these files been introduced? On SysV systems services are started via init scripts in /etc/rc.d/init.d (or a similar directory). /etc/ is (these days) considered the place where system configuration is stored. Originally these init scripts were subject to customization by the administrator. But as they grew and become complex most distributions no longer considered them true configuration files, but more just a special kind of programs. To make customization easy and guarantee a safe upgrade path the customizable bits hence have been moved to separate configuration files, which the init scripts then source.

Let's have a quick look what kind of configuration you can do with these files. Here's a short incomprehensive list of various things that can be configured via environment settings in these source files I found browsing through the directories on a Fedora and a Debian machine:

Additional command line parameters for the daemon binaries
Locale settings for a daemon
Shutdown time-out for a daemon
Shutdown mode for a daemon
System configuration like system locale, time zone information, console keyboard
Redundant system configuration, like whether the RTC is in local timezone
Firewall configuration data, not in shell format (!)
CPU affinity for a daemon
Settings unrelated to boot, for example including information how to install a new kernel package, how to configure nspluginwrap or whether to do library prelinking
Whether a specific service should be started or not
Networking configuration
Which kernel modules to statically load
Whether to halt or power-off on shutdown
Access modes for device nodes (!)
A description string for the SysV service (!)
The user/group ID, umask to run specific daemons as
Resource limits to set for a specific daemon
OOM adjustment to set for a specific daemon

Now, let's go where the beef is: what's wrong with /etc/sysconfig (resp. /etc/default)? Why might it make sense to fade out use of these files in a systemd world?

For the majority of these files the reason for having them simply does not exist anymore: systemd unit files are not programs like SysV init scripts were. Unit files are simple, declarative descriptions, that usually do not consist of more than 6 lines or so. They can easily be generated, parsed without a Bourne interpreter and understood by the reader. Also, they are very easy to modify: just copy them from /lib/systemd/system to /etc/systemd/system and edit them there, where they will not be modified by the package manager. The need to separate code and configuration that was the original reason to introduce these files does not exist anymore, as systemd unit files do not include code. These files hence now are a solution looking for a problem that no longer exists.
They are inherently distribution-specific. With systemd we hope to encourage standardization between distributions. Part of this is that we want that unit files are supplied with upstream, and not just added by the packager -- how it has usually been done in the SysV world. Since the location of the directory and the available variables in the files is very different on each distribution, supporting /etc/sysconfig files in upstream unit files is not feasible. Configuration stored in these files works against de-balkanization of the Linux platform.
Many settings are fully redundant in a systemd world. For example, various services support configuration of the process credentials like the user/group ID, resource limits, CPU affinity or the OOM adjustment settings. However, these settings are supported only by some SysV init scripts, and often have different names if supported in multiple of them. OTOH in systemd, all these settings are available equally and uniformly for all services, with the same configuration option in unit files.
Unit files know a large number of easy-to-use process context settings, that are more comprehensive than what most /etc/sysconfig files offer.
A number of these settings are entirely questionnabe. For example, the aforementiond configuration option for the user/group ID a service runs as is primarily something the distributor has to take care of. There is little to win for administrators to change these settings, and only the distributor has the broad overview to make sure that UID/GID and name collisions do not happen.
The file format is not ideal. Since the files are usually sourced as shell scripts, parse errors are very hard to decypher and are not logged along the other configuration problems of the services. Generally, unknown variable assignments simply have no effect but this is not warned about. This makes these files harder to debug than necessary.
Configuration files sources from shell scripts are subject to the execution parameters of the interpreter, and it has many: settings like IFS or LANG tend to modify drastically how shell scripts are parsed and understood. This makes them fragile.
Interpretation of these files is slow, since it requires spawning of a shell, which adds at least one process for each service to be spawned at boot.
Often, files in /etc/sysconfig are used to "fake" configuration files for daemons which do not support configuration files natively. This is done by glueing together command line arguments from these variable assignments that are then passed to the daemon. In general proper, native configuration files in these daemons are the much prettier solution however. Command line options like "-k", "-a" or "-f" are not self-explanatory and have a very cryptic syntax. Moreover the same switches in many daemons have (due to the limited vocabulary) often very much contradicting effects. (On one daemon -f might cause the daemon to daemonize, while on another one this option turns exactly this behaviour off.) Command lines generally cannot include sensible comments which most configuration files however can.
A number of configuration settings in /etc/sysconfig are entirely redundant: for example, on many distributions it can be controlled via /etc/sysconfig files whether the RTC is in UTC or local time. Such an option already exists however in the 3rd line of the /etc/adjtime (which is known on all distributions). Adding a second, redundant, distribution-specific option overriding this is hence needless and complicates things for no benefit.
Many of the configuration settings in /etc/sysconfig allow disabling services. By this they basically become a second level of enabling/disabling over what the init system already offers: when a service is enabled with systemctl enable or chkconfig on these settings override this, and turn the daemon of even though the init system was configured to start it. This of course is very confusing to the user/administrator, and brings virtually no benefit.
For options like the configuration of static kernel modules to load: there are nowadays usually much better ways to load kernel modules at boot. For example, most modules may now be autoloaded by udev when the right hardware is found. This goes very far, and even includes ACPI and other high-level technologies. One of the very few exceptions where we currently do not do kernel module autoloading is CPU feature and model based autoloading which however will be supported soon too. And even if your specific module cannot be auto-loaded there's usually a better way to statically load it, for example by sticking it in /etc/load-modules.d so that the administrator can check a standardized place for all statically loaded modules.
Last but not least, /etc already is intended to be the place for system configuration ("Host-specific system configuration" according to FHS). A subdirectory beneath it called sysconfig to place system configuration in is hence entirely redundant, already on the language level.

What to use instead? Here are a few recommendations of what to do with these files in the long run in a systemd world:

Just drop them without replacement. If they are fully redundant (like the local/UTC RTC setting) this is should be a relatively easy way out (well, ignoring the need for compatibility). If systemd natively supports an equivalent option in the unit files there is no need to duplicate these settings in sysconfig files. For a list of execution options you may set for a service check out the respective man pages: systemd.exec(5) and systemd.service(5). If your setting simply adds another layer where a service can be disabled, remove it to keep things simple. There's no need to have multiple ways to disable a service.
Find a better place for them. For configuration of the system locale or system timezone we hope to gently push distributions into the right direction, for more details see previous episode of this series.
Turn these settings into native settings of the daemon. If necessary add support for reading native configuration files to the daemon. Thankfully, most of the stuff we run on Linux is Free Software, so this can relatively easily be done.

Of course, there's one very good reason for supporting these files for a bit longer: compatibility for upgrades. But that's is really the only one I could come up with. It's reason enough to keep compatibility for a while, but I think it is a good idea to phase out usage of these files at least in new packages.

If compatibility is important, then systemd will still allow you to read these configuration files even if you otherwise use native systemd unit files. If your sysconfig file only knows simple options EnvironmentFile=-/etc/sysconfig/foobar (See systemd.exec(5) for more information about this option.) may be used to import the settings into the environment and use them to put together command lines. If you need a programming language to make sense of these settings, then use a programming language like shell. For example, place an short shell script in /usr/lib/<your package>/ which reads these files for compatibility, and then exec's the actual daemon binary. Then spawn this script instead of the actual daemon binary with ExecStart= in the unit file.

And this is all for now. Thank you very much for your interest.

posted at: 00:34 | path: /projects | permanent link to this entry | 17 comments

Posted by Joe Tennies at Mon Jul 18 02:18:28 2011
While I certainly wouldn't mind seeing these files "go away", I don't think the current landscape really allows for it completely. The problem is that you seem to be under the assumption that everyone is using systemd is false. Ubuntu uses Upstart by default, Solaris uses Service Management Facility, Darwin and OSX use launchd, and the BSDs probably use several other ones. Most things I'd expect to need an init script are very low level daemons and are going to try to be cross-platform (a la SQL databases, web servers, module configuration). The things you point out, could probably be removed from these files though.

PS: Thanks, I'm switching to Fedora from Ubuntu and didn't know where something like /etc/default went =)

Posted by Your mom at Mon Jul 18 07:05:50 2011
One problem with your suggestion of copying files from /lib/systemd/system to /etc/systemd is that you assume that the unit file shipped with a package will not change on upgrade.. If it does change, then the sysadmin may not notice it and may not get whatever improvement comes with the unit file of the new version.

It would probably be better to add a place where you can put unit files that are merged by systemd to the content of the default ones.

Posted by Michael at Mon Jul 18 12:28:10 2011
Removing the files are clearly a good idea ( and one that would for sure solve some headaches on the configuration tool side ), but I am a little bit concerned over what to do to solve the service without configuration file.

Should we 1) push some form of configuration file to upstream ( they may not want, as this can be more complex to manage )
2) push the configuration under the form of arguments to the service ?

I do not have good solution to propose yet.

Posted by bochecha at Mon Jul 18 12:50:56 2011
@Michael:

What is the difference between:

1. an initscript which will first source the following sysconfig file:
OPTIONS=-a foo -b bar -f
and which will then run:
/usr/sbin/service $OPTIONS

2. the following unit file:
[... snip ...]
Environment=OPTIONS=-a foo -b bar -f
[... snip ...]
ExecStart=/usr/sbin/service $OPTIONS

In both cases an admin can configure the options that get passed to the daemon, and that works for services without configuration files.

I, for one, am welcoming the effort to restore sanity that systemd is steering.

Posted by Lennart at Mon Jul 18 14:37:06 2011
Joe: shipping unit files upstream does not necessarily mean the other distros can't continue what they want to do. Also, sysconfig files generally are not something the distros share, so I am not sure what you are trying to say anyway.

Your: you can use ".include" in unit files you edit to include the original version. That way you can import the original version and simply extend/override a few fields of it.

Michael: the daemons should be patched upstream to gain proper configuration files.

Posted by Miloslav Trmac at Mon Jul 18 16:05:40 2011
The historical context and rationale need correcting.

First, don't look at /etc/sysconfig as "additional sysv script configuration": it is really "configuration for distribution-specific software" - and the things it usually contains are things like host name, network configuration, that are not a responsibility of a particular "upstream" daemon. /etc/sysconfig/* hosts configuration for software that was written for the specific distribution, be that rc.sysinit, its network scripts, or something else. The "sysconfig" subdirectory simply created a namespace for the distribution-specific configuration that was separate from, and thus didn't conflict with, the other users of /etc/.

In RHL, "sysconfig" has been primarily used by initscripts; in Suse, it was (is?) managed by Yast. Yast is in fact a typical example of sysconfig use: the configuration options are "related" to what upstream software does, but their semantics, level of abstraction, and the software that actually interprets these options
are completely different from the upstream options.

Using /etc/sysconfig/* to configure SysV initscripts is consistent with the above, but it is really a minor use that has been much rarer.

Second, a typical /etc/sysconfig/* file is, at least in RH-derived distributions, not an arbitrary shell script! These files were edited and have been used by various tools that are not shells, see the shvar.[ch] library copied in various RH tools. Placing arbitrary shell into these files has never been supported by initscripts, the primary RH user of /etc/sysconfig. Therefore, the rationale of speeding boot by avoiding shell doesn't apply in most situations.

Ultimately, a configuration file can only be removed when the configuration options stored there are is no longer needed or implemented.

- It is very rarely the case that the option is no longer needed. Software typically grows more, not less, capable over time; options are usually no longer needed only because they apply to hardware or protocols that users no longer care about.

- If the option is still needed, but the implementation was completely removed, users will be angry - with, from their point of view, a good reason.

- If the option is still needed, still implemented, but it is now configured elsewhere, users deserve a better migration path than "figure out why the thing suddenly stopped working when I didn't do anything". And typically it saves everyone time (both the software user and the programmer) if the configuration file is left in its place, even if that place is not ideal. There is some logic in moving e.g. the hostname configuration - after all the software that used to set the host name (rc.sysinit) was replaced (by systemd). But this really doesn't apply to options that are not implemented by systemd itself. "$daemon is still at the same minor version as it was in the previous release of the distribution, why is it now configured elsewhere?"

Posted by Horst H. von Brand at Mon Jul 18 16:11:47 2011
Lennart, your criticism that /etc/sysconfig/foo requires starting a new shell is false. The file is sourced by the shell running the init script.

BTW, in Fedora you can also set the display manager for X in /etc/sysconfig/desktop.

I do share Your Mom's concern that just shoving configuration into copies of the shipped unit files (and which override them!) will end us in the same mess that sysconfig/default are being used to kludge around.

Posted by Shawn at Mon Jul 18 17:49:33 2011
I would like systemd to have a fallback /sbin/init. Too many times will system in single user mode break. Take the syslog pipe failing to start.

Although oddly, systemd now ignores it starting up after a few reboot attempts.

Is it possible to have systemd mark a failed process as disabled until the administrator can fix it either during single user mode or another runlevel?

Posted by Your Name: at Tue Jul 19 05:13:53 2011
So you find "/etc/sysconfig" redundant "on the language level", yet you suggest naming most files under /etc/ with a ".conf" suffix? Does Not Compute.

Posted by Phil at Tue Jul 19 12:00:41 2011
(This comment might appear twice, I got "rejected" the first time, so I'm rewriting it.)

Would it be possible or practical to pass in the configuration in the same way as sockets are passed in on activation? It seems error prone to put similar file reading/parsing code into lots of tiny daemons.

Having systemd manage this would also make it easy to have multiple instances of services when that might be appropriate, since there would be no hard coded config locations, so multiple configs would be very easy.

Posted by Rob at Tue Jul 19 15:31:51 2011
Bit OT, but it would be nice if you could add cmake section to http://0pointer.de/public/systemd-man/daemon.html
Right now there's only autoconf part.

Posted by Anonymous at Tue Jul 19 23:19:41 2011
@Your Name: .conf serves a useful function in one specific case: allowing /etc/foo.d directories to distinguish between configuration files and random garbage such as backup files or package manager temporary files.

Posted by Josh Triplett at Tue Jul 19 23:43:10 2011
A quick survey of /etc/default on my Debian system:

/etc/default/alsa contains a list of modules to unload before suspending with apmd. Completely dead.

/etc/default/apache2 contains configuration for htcacheclean, which should become proper configuration for that program.

/etc/default/avahi-daemon contains configuration for the mechanism that detects and works around crazy local DNS also using .local. This should just go away, or if people complain too bitterly about their broken local networks then it should go upstream into avahi.

/etc/default/bluetooth contains an enable/disable toggle for the daemon (redundant with need-based activation) and configuration for hid2hci (should ideally just go away in favor of autodetection).

/etc/default/bootlogd contains an enable/disable toggle; redundant, along with bootlogd itself.

/etc/default/cacerts contains configuration for the Java certificate store. That should almost certainly not exist, and if it should then it should go in some java-specific configuration file.

/etc/default/cron contains extra options for cron (should go in /etc/cron.conf or something) and configuration options for a random cronjob related to lost+found (should move to the fsck logic instead, not cron).

/etc/default/cryptdisks contains information that should live in /etc/fstab, or possibly in /etc/crypttab if that still needs to exist.

/etc/default/cups contains a boolean determining whether to load the "lp" module; should get replaced with autodetection and/or /etc/load-modules.d.

/etc/default/dbus contains extra parameters for dbus-daemon. To quote it: "it allows you to perform common modifications to the behavior of the dbus daemon startup without editing the init script (and thus getting prompted by dpkg on upgrades). We all love dpkg prompts." Ridiculous. (I actually want those dpkg prompts if I've changed configuration, and I'll like it better when those configuration files will rarely need to change since they don't contain piles of buggy scripting.)

/etc/default/debsums contains configuration for a cron job periodically running debsums. Should live in /etc proper, or better yet just get configured by enabling/disabling the periodic job.

/etc/default/devpts contains the GID and mode for /dev/pts devices. Should live in a real configuration file, and better yet use a group name rather than a number.

/etc/default/grub contains information used by update-grub to build a configuration file. This seems redundant both with the more general /etc/grub.d mechanism and with editing the configuration file itself (if not autogenerating it from /etc/grub.d). And these days GRUB has enough scripting support to go find OSes and kernels in its own configuration file at boot time without needing update-grub. (That and someday I'd like to replace GRUB itself with a minimal Linux kernel and kexec. ;) )

/etc/default/halt configures whether "halt" implies poweroff. It should always imply poweroff unless you specifically say otherwise; if you want some other behavior, use a shell alias. :)

/etc/default/kdm.d/ contains configuration for KDM, which should just live in a KDM configuration file and/or /etc/kdm.d.

/etc/default/keyboard contains keyboard layout information, which systemd has already standardized.

/etc/default/locale contains the system default locale.

/etc/default/nss contains strange environment variables configuring NSS, which should probably live in /etc/nsswitch.conf and/or die.

/etc/default/ntp contains command-line options for ntp.

/etc/default/rcS contains configuration options for various init scripts, such as whether to wipe /tmp, whether to spawn sulogin, whether to obey nologin, whether to use UTC, whether to spew verbose messages, whether to actually fix things when fscking (as though anyone would do otherwise), and whether to use a tmpfs for /var/run and /var/lock. All redundant AFAICT.

/etc/default/rsync contains configuration for the rsync daemon, which should live in a proper configuration file and/or the invocation of rsync.

/etc/default/rsyslog contains command-line options.

/etc/default/tmpfs contains the size of the tmpfs filesystems to mount at various points.

/etc/default/useradd contains configuration for adduser and useradd, which ought to live in a proper configuration file, or in some cases (such as the path to /etc/skel) go away.

Posted by nameless reject at Thu Jul 21 06:17:19 2011
@Josh: The reason /etc/defaults initially appeared in Debian is because of the "/etc/default/dbus" issue you mentioned. Specifically, at least early on, config files (or initscripts) would often change but the only change would be a comment, or a commented-out new feature. To avoid needless prompting, local system configuration would go into /etc/default.

I haven't played with systemd yet, but it seems that if each unit configuration came with a ".include /etc/local/$0" type directive the goal of not prompting for trivial changes would be met. Whether or not this is useful or dangerous can be left to the reader ...

Posted by Karellen at Thu Jul 21 13:36:08 2011
@nameless reject: Or, distro packages could not ship /etc/systemd/[servicefile]. Rather, as suggested in the article, they'd ship /lib/systemd/[servicefile]. If you wanted to make local changes to this, instead of copying the servicefile to /etc/systemd, you could create your own /etc/system/[servicefile] with the first line being ".include /lib/systemd/[servicefile]"

Posted by SlackUser at Mon Aug 1 05:46:34 2011
http://www.linuxquestions.org/questions/slackware-14/slackware-and-systemd-885228/

Looks like Slackware is not even considering to switch to systemd :\

Posted by DaboD at Wed Aug 17 01:21:54 2011
Can you explain how to use systemd as session manager?

Leave a Comment:

It should be obvious but in case it isn't: the opinions reflected here are my own. They are not the views of my employer, or Ronald McDonald, or anyone else.

Please note that I take the liberty to delete any comments posted here that I deem inappropriate, off-topic, or insulting. And I excercise this liberty quite agressively. So yes, if you comment here, I might censor you. If you don't want to be censored your are welcome to comment on your own blog instead.

レナート Wunschkonzert, Ponyhof und Abenteuerspielplatz ﻟﻴﻨﺎﺭﺕ

Mon, 18 Jul 2011

systemd for Administrators, Part IX

On /etc/sysconfig and /etc/default