{"id":3134,"date":"2018-05-07T15:45:30","date_gmt":"2018-05-07T20:45:30","guid":{"rendered":"http:\/\/lisa.rushworth.us\/?p=3134"},"modified":"2018-05-09T15:56:29","modified_gmt":"2018-05-09T20:56:29","slug":"systemd-a-k-a-where-did-my-log-files-go","status":"publish","type":"post","link":"https:\/\/www.rushworth.us\/lisa\/?p=3134","title":{"rendered":"Systemd (a.k.a. where did my log files go!?!?!)"},"content":{"rendered":"

A systemd Primer For sysvinit Users<\/strong><\/p>\n

Background:<\/em><\/strong><\/p>\n

Starting in Fedora 15 and RHEL 7, systemd replaces sysvinit. This is a touchy subject among Unix folks \u2013 some people think it\u2019s a great change, others think Linux has been ruined forever. Our personal opinions of the shift doesn\u2019t matter: vendors are implementing it, WIN Linux servers use it, so we need to know it. Basically, throw \u201csystemd violates the minimalist, modular philosophy at the core of Unix development\u201d on the \u201cbut emacs is so awesome, why are we using vim\u201d and \u201cBETA outperforms VHS any day of the week\u201d pile.<\/p>\n

Quick terminology \u2013 services are now called units. You\u2019ll see that word a lot. A unit is configured in a \u201cunit file\u201d. Additionally, \u201crun levels\u201d (0-6) have been replaced with the concept of \u201ctargets\u201d that have friendly names.<\/p>\n

What\u2019s the difference?<\/em><\/strong><\/p>\n

Sysvinit wasn\u2019t designed to know about your system, it was designed to run scripts on<\/em> your system. Sysvinit essentially runs scripts, whereas systemd is a service manager. Systemd knows<\/em> about the system. One place this becomes apparent \u2013 if you manually run the run line from a sysvinit script then check the service status, it will show running because the binary has a PID. If you do the same with systemd, it will say the service is down. This is like Windows \u2013 if you have a Docker service that runs \u201c”C:\\Program Files\\Docker\\Docker\\com.docker.service”\u201d set to run manually, and use start-run to run the exact same string \u2026 the service will not show as running.<\/p>\n

Systemd manages a lot of different unit types. As application owners, we\u2019ll use \u2018service\u2019 units. \u2018Mount\u2019 or \u2018automount\u2019 type units manage mountpoints. Socket and device unit types manage sockets (which have associated service unit files using<\/em> the socket) and devices. Because systemd manages sockets, inetd\/xinetd have been obsoleted.<\/p>\n

Sysvinit scripts could run user-defined commands. If the init script for myapplication has a section called \u201cbob\u201d, you can run \u201cservice myapplication bob\u201d and it will do whatever the \u2018bob\u2019 part of the script says to do. Systemd has a fixed list of directives \u2013 start, stop, restart, reload, status, enable, disable, is-enabled, list-unit-files, list-dependencies, daemon-reload. You cannot just make a new one.<\/p>\n

Systemd may also require a system reboot for more than just kernel patches. This is really different, and I expect there will be a learning curve as to what requires a reboot.<\/p>\n

Log files have “vanished”. If you are using a default installation, you won\u2019t find \/var\/log\/messages. You can use \u201cjournalctl -f\u201d to tail the equivalent of the messages file. The systemd log files are stored in binary format \u2013 potentially corruptible, which is another aspect of the change Unix-types don\u2019t care for.<\/p>\n

What does systemd give me?<\/em><\/strong><\/p>\n

Systemd doesn\u2019t just start\/stop a service when run levels change. A unit can be started because it is configured to start on the runlevel (just like sysvinit scripts), if another service requires it, if the service abends, or if dbus triggers it. \u201cIf another service requires it\u201d \u2013 that\u2019s a dependency chain. Instead of defining an order<\/em> and hoping everything you need was loaded by the time the init script ran, systemd allows you to include an \u201cAfter\u201d directive \u2013 units started before<\/em> the current unit or \u201cBefore\u201d \u2013 units that will not be started until the current unit starts. Additional directives for \u201cRequires\u201d \u2013 units which must be activated to activate the current unit and \u201cWants\u201d \u2013 units that will be started in parallel with the current unit but<\/em> failing to start these units will not fail the current unit.<\/p>\n

A directive, \u201cConflicts\u201d, allows systemd to identify other units that cannot coexist with the current unit. Conflicting units will be stopped to allow the current unit to start. In addition to the base command starting in the unit file (ExecStart), there are pre (ExecStartPre) and post (ExecStartPost) operations that are run before\/after the base command. These could be related to the service itself but do not have to be. You could run a mail command line to alert an admin every time the unit starts or stops cleanly.<\/p>\n

Another nice feature of systemd is user-level services \u2013 using systemctl –user will control unit files located in user-specific directories like \/usr\/lib\/systemd\/user\/ and ~\/.config\/system\/user\/<\/p>\n

Using systemd<\/em><\/strong>: (Warning: this is going to get odd)<\/em><\/p>\n

You use systemctl to control units, and you use journalctl to view the binary blobs that have replaced log files. Use the man pages or your favourite search engine if you want details. The general syntax for systemclt is \u201csystemctl operation unit.type\u201d<\/em> \u2013 e.g. \u201csystemctl restart sendmail\u201d would restart sendmail.<\/p>\n

Chkconfig has been completely supplanted. Use \u201csystemctl enable unit.type\u201d and \u201csystemctl disable unit.type\u201d to control if a service auto-starts. Instead of using chkconfig \u2013list, you can query the startup state of an individual unit. Use systemctl –is-enabled unit.type<\/p>\n

There\u2019s a service shell script that replaces <\/em>\u2018service\u2019 that you used with sysvinit systems. It turns the old \u201cservice something-or-other action\u201d into \u201csystemctl action name.service\u201d so it still works.<\/p>\n

Here\u2019s the odd part \u2013 it is quite easy to define a permitted sudo operation that allows a non-root user to control sysvinit services. Allow \u201cservice sendmail\u201d and the user can run \u201cservice sendmail start\u201d, \u201cservice sendmail stop\u201d, \u201cservice sendmail status\u201d, \u201cservice sendmail RandomStuffITossedIntoTheFile\u201d. Because the service name and directive are swapped around in systemctl, we would have to enumerate each individual directive that should be permitted. More secure, because RandomStuffITossedIntoTheFile should not make the cut. But we haven\u2019t done this yet. So<\/em><\/strong> until we go through and enumerate the reasonable actions (Are there directives beyond start\/stop\/status that we should be running? Do we have any business enabling and disabling our services?), submit the access request, confirm it\u2019s all functioning as expected, and remove the \u201csudo service\u201d access \u2026 continue using \u201csudo service something-or-other action\u201d. We will advise you when the systemctl sudo access has been granted so we can start using the \u201cnew way\u201d to control services on RHEL7 systems.<\/p>\n

Unlike init scripts, changes to systemd unit files are not immediately activated on the system. Running “systemctl daemon-reload” makes systemd aware of the config change.<\/p>\n

Using journalctl:<\/em><\/strong><\/p>\n

Our Unix team has implemented rsyslogd to output log data to the expected files. This means you can more or less ignore journalctl \u2013 tail\/grep the log file as usual. I don\u2019t foresee this changing in the near to mid term, but if you use cloud-hosted sandbox servers (i.e. boxes that don\u2019t have the Unix group\u2019s standard config) \u2026 journalctl is what happened to all the log files you cannot find.<\/p>\n

To view logs specific to an individual unit, use journalctl -u unit.type. Additionally \u201csystemctl unit.type status\u201d will display the last handful of log lines from the unit.<\/p>\n","protected":false},"excerpt":{"rendered":"

A systemd Primer For sysvinit Users Background: Starting in Fedora 15 and RHEL 7, systemd replaces sysvinit. This is a touchy subject among Unix folks \u2013 some people think it\u2019s a great change, others think Linux has been ruined forever. Our personal opinions of the shift doesn\u2019t matter: vendors are implementing it, WIN Linux servers …<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[30],"tags":[294,586,585],"class_list":["post-3134","post","type-post","status-publish","format-standard","hentry","category-system-administration","tag-linux","tag-systemd","tag-sysvinit"],"_links":{"self":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/3134","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3134"}],"version-history":[{"count":4,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/3134\/revisions"}],"predecessor-version":[{"id":3138,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/3134\/revisions\/3138"}],"wp:attachment":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3134"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3134"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3134"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}