Systemd (a.k.a. where did my log files go!?!?!)

Avatar
by Lisa

A systemd Primer For sysvinit Users

Background:

Starting in Fedora 15 and RHEL 7, systemd replaces sysvinit. This is a touchy subject among Unix folks – some people think it’s a great change, others think Linux has been ruined forever. Our personal opinions of the shift doesn’t matter: vendors are implementing it, WIN Linux servers use it, so we need to know it. Basically, throw “systemd violates the minimalist, modular philosophy at the core of Unix development” on the “but emacs is so awesome, why are we using vim” and “BETA outperforms VHS any day of the week” pile.

Quick terminology – services are now called units. You’ll see that word a lot. A unit is configured in a “unit file”. Additionally, “run levels” (0-6) have been replaced with the concept of “targets” that have friendly names.

What’s the difference?

Sysvinit wasn’t designed to know about your system, it was designed to run scripts on your system. Sysvinit essentially runs scripts, whereas systemd is a service manager. Systemd knows about the system. One place this becomes apparent – if you manually run the run line from a sysvinit script then check the service status, it will show running because the binary has a PID. If you do the same with systemd, it will say the service is down. This is like Windows – if you have a Docker service that runs “”C:\Program Files\Docker\Docker\com.docker.service”” set to run manually, and use start-run to run the exact same string … the service will not show as running.

Systemd manages a lot of different unit types. As application owners, we’ll use ‘service’ units. ‘Mount’ or ‘automount’ type units manage mountpoints. Socket and device unit types manage sockets (which have associated service unit files using the socket) and devices. Because systemd manages sockets, inetd/xinetd have been obsoleted.

Sysvinit scripts could run user-defined commands. If the init script for myapplication has a section called “bob”, you can run “service myapplication bob” and it will do whatever the ‘bob’ part of the script says to do. Systemd has a fixed list of directives – start, stop, restart, reload, status, enable, disable, is-enabled, list-unit-files, list-dependencies, daemon-reload. You cannot just make a new one.

Systemd may also require a system reboot for more than just kernel patches. This is really different, and I expect there will be a learning curve as to what requires a reboot.

Log files have “vanished”. If you are using a default installation, you won’t find /var/log/messages. You can use “journalctl -f” to tail the equivalent of the messages file. The systemd log files are stored in binary format – potentially corruptible, which is another aspect of the change Unix-types don’t care for.

What does systemd give me?

Systemd doesn’t just start/stop a service when run levels change. A unit can be started because it is configured to start on the runlevel (just like sysvinit scripts), if another service requires it, if the service abends, or if dbus triggers it. “If another service requires it” – that’s a dependency chain. Instead of defining an order and hoping everything you need was loaded by the time the init script ran, systemd allows you to include an “After” directive – units started before the current unit or “Before” – units that will not be started until the current unit starts. Additional directives for “Requires” – units which must be activated to activate the current unit and “Wants” – units that will be started in parallel with the current unit but failing to start these units will not fail the current unit.

A directive, “Conflicts”, allows systemd to identify other units that cannot coexist with the current unit. Conflicting units will be stopped to allow the current unit to start. In addition to the base command starting in the unit file (ExecStart), there are pre (ExecStartPre) and post (ExecStartPost) operations that are run before/after the base command. These could be related to the service itself but do not have to be. You could run a mail command line to alert an admin every time the unit starts or stops cleanly.

Another nice feature of systemd is user-level services – using systemctl –user will control unit files located in user-specific directories like /usr/lib/systemd/user/ and ~/.config/system/user/

Using systemd: (Warning: this is going to get odd)

You use systemctl to control units, and you use journalctl to view the binary blobs that have replaced log files. Use the man pages or your favourite search engine if you want details. The general syntax for systemclt is “systemctl operation unit.type” – e.g. “systemctl restart sendmail” would restart sendmail.

Chkconfig has been completely supplanted. Use “systemctl enable unit.type” and “systemctl disable unit.type” to control if a service auto-starts. Instead of using chkconfig –list, you can query the startup state of an individual unit. Use systemctl –is-enabled unit.type

There’s a service shell script that replaces ‘service’ that you used with sysvinit systems. It turns the old “service something-or-other action” into “systemctl action name.service” so it still works.

Here’s the odd part – it is quite easy to define a permitted sudo operation that allows a non-root user to control sysvinit services. Allow “service sendmail” and the user can run “service sendmail start”, “service sendmail stop”, “service sendmail status”, “service sendmail RandomStuffITossedIntoTheFile”. Because the service name and directive are swapped around in systemctl, we would have to enumerate each individual directive that should be permitted. More secure, because RandomStuffITossedIntoTheFile should not make the cut. But we haven’t done this yet. So until we go through and enumerate the reasonable actions (Are there directives beyond start/stop/status that we should be running? Do we have any business enabling and disabling our services?), submit the access request, confirm it’s all functioning as expected, and remove the “sudo service” access … continue using “sudo service something-or-other action”. We will advise you when the systemctl sudo access has been granted so we can start using the “new way” to control services on RHEL7 systems.

Unlike init scripts, changes to systemd unit files are not immediately activated on the system. Running “systemctl daemon-reload” makes systemd aware of the config change.

Using journalctl:

Our Unix team has implemented rsyslogd to output log data to the expected files. This means you can more or less ignore journalctl – tail/grep the log file as usual. I don’t foresee this changing in the near to mid term, but if you use cloud-hosted sandbox servers (i.e. boxes that don’t have the Unix group’s standard config) … journalctl is what happened to all the log files you cannot find.

To view logs specific to an individual unit, use journalctl -u unit.type. Additionally “systemctl unit.type status” will display the last handful of log lines from the unit.

Load Balance and Failover Sendmail Mailertable Relays

Avatar
by Lisa

A coworker asked me today how to get the mailertable relays to load balance instead of fail over. Trick is to think beyond sendmail. The square brackets around hosts tell sendmail not to check for an MX record (you’re generally using an A record, so this saves a tiny little bit of time … not to mention *if* there is an MX record there, it creates a whole heap-o confusion). *But* the MX lookup is right useful when setting up load balanced or failover relay targets.

Single host relay in the mailertable looks like this:
yourdomain.gTLD      relay:[somehost.mydomain.gTLD]

If you want to fail over between relays (that is try #1, if it is unavailable try #2, and so on), you can stay within the mailertable and use:
yourdomain.gTLD      relay:[somehost.mydomain.gTLD]:[someotherhost.mydomain.gTLD]

Or even try direct delivery and fail back to a smart host:
yourdomain.gTLD      relay:%1:smart-host

But none of this evenly distributes traffic across multiple servers. The trick to load balancing within the mailertable is to create equal weight MX records in your domain to be used as the relay.

In ISC Bind, this looks like:
yourdomainmailrouting.mydomain.gTLD     IN MX 10 somehost.mydomain.gTLD.
yourdomainmailrouting.mydomain.gTLD     IN MX 10 somehost.mydomain.gTLD.

Once you have created the DNS records, simply use the MX record hostname in your mailertable:

yourdomain.gTLD      relay:yourdomainmailrouting.mydomain.gTLD

By leaving out the square brackets, sendmail will resolve an MX record for ‘yourdomainmailrouting.mydomian.gTLD’, find the equal weight MX records, and do the normal sendmail thing to use both.

Retainers And Loans

Avatar
by Lisa

Giuliani’s assertion on Hannity’s show: When I heard Cohen’s retainer of thirty-five thousand, when he was doing no work for the President, I said, ‘That’s how he’s repaying it, with a little profit and a little margin for paying taxes, for Michael.’

I don’t care to dig into the nuances of loan repayment, but I’ve paid lawyers retainer fees for doing no work. A retainer is money paid to ensure you have a lawyer on-hand to provide legal advice and services if you need them. Should your requirements exceed that which the retainer contract permits, you get billed extra. But I’ve never seen a retainer contract that had stipulations in case the client ended up *not* needing services during the period. If you pay 35k for a one-year contract that provides up to 40 hours per month on assignments determined by the client … well, zero is “up to 40” and you’ve paid 35k for no work. You paid for the willingness to provide work and for the work should it be needed. Which is why people who do not routinely require legal assistance don’t tend to keep a lawyer on retainer: no point dropping a couple grand a month every month you don’t need a lawyer. Wait until you are, say, writing your will and hire one for the specific task.

Reality and the Law, An Ongoing Saga

Avatar
by Lisa

Sayeth Trump

Some legal infractions are straight-forward. Speeding — there is empirical evidence that the vehicle which you were driving was moving at 63 miles per hour. The posted speed limit for the road, again empirical evidence, is 45. The line of questioning in this case may be “Were you speeding?”. It’s a lot quicker than asking what speed you were travelling, what the speed limit is on the road, and if your speed exceeded that limit.

Many infractions are not this distinct. Driving too fast for road conditions — that’s a matter of opinion. In fact, a decent argument could be made that someone involved in *most* traffic accidents was driving too fast for road conditions. I had a friend wipe out his motorbike on highway gravel. He was abraded but fine. A cop drove by as he was righting his motorbike, and stopped to help. Eighteen year old kid with a grudge against pretty much everyone mouthed off to the cop sufficiently to be cited for driving too fast for road conditions. Because gravel? That’s a road condition.

Collusion and obstruction of justice both fall into the “not clear cut” category. An unemployed guy notices a business district has a problem with vandalism and offers to patrol the street from 8p-6a for ten dollars an hour because they seem to have a vandal problem. That’s not extortion or racketeering — that’s someone who needs work offering to provide a service someone else needs. An unemployed guy starts vandalizing the business district, then offers to patrol the street from 8p-6a for ten dollars an hour because they seem to have a vandal problem … that’s a protection racket. The prosecution may not directly ask “are you running a protection racket?”. They could delve into how the guy noticed the vandalism problem on multiple occasions, ask questions show how desperate he was for money, ask the guy where he was during the vandalism. That line of questioning doesn’t mean the prosecution doesn’t think he’s running a protection racket. It means they’re asking questions that address all of what differentiates the perfectly innocent first scenario from the criminal second scenario.

In the NYTimes list of questions Mueller is said to have for Trump, there’s a whole section titled “Campaign Coordination With Russia”. Surely not Mueller’s title, but how can anyone reading this list say there are “no questions on collusion”? Because the word doesn’t literally appear!?!

Technicalities of Legalities

Avatar
by Lisa

Anyone else glad the head of the Executive Office of the United States has such a firm grasp on the law?

Obstruction of justice is corrupt interference in the proceedings or people serving at a proceeding from doing his duty. Nowhere in the US Code does it say “assuming, of course, the proceeding leads to a conviction”. If you are found guilty of a crime you didn’t commit, sentenced to five years in jail, escape jail, are subsequently found to be innocent and your initial conviction vacated … you can still be charged with escaping jail and sentenced to jail time for the offense. Now you might get time served, or a reduced sentence … but you still committed the crime of escaping the jail when incarcerated. Same deal-e-o here. If I didn’t commit a crime but was being investigated, and tried to influence witnesses or stop police from investigating the non-crime … that is a crime.

18 U.S.C. § 1503: “Whoever corruptly, or by threats or force, or by any threatening letter or communication, endeavors to influence, intimidate, or impede any grand or petit juror, or officer in or of any court of the United States, or officer who may be serving at any examination or other proceeding before any United States magistrate judge or other committing magistrate, in the discharge of his duty, or injures any such grand or petit juror in his person or property on account of any verdict or indictment assented to by him, or on account of his being or having been such juror, or injures any such officer, magistrate judge, or other committing magistrate in his person or property on account of the performance of his official duties, or corruptly or by threats or force, or by any threatening letter or communication, influences, obstructs, or impedes, or endeavors to influence, obstruct, or impede, the due administration of justice, shall be punished as provided in subsection (b). If the offense under this section occurs in connection with a trial of a criminal case, and the act in violation of this section involves the threat of physical force or physical force, the maximum term of imprisonment which may be imposed for the offense shall be the higher of that otherwise provided by law or the maximum term that could have been imposed for any offense charged in such case.”

Political Pragmatism

Avatar
by Lisa

This is a difficult one for me — totally un-democratic, but I can also see the point. The general election isn’t about picking a guy to represent the Democrats of the Sixth Congressional District of Colorado, it’s about picking a guy to represent *all* of the Sixth District. I’d generally prefer to be represented by a less progressive Democrat than a less conservative Republican.

 
The salient questions are *does* the candidate need to appeal to some percentage of swing voters (i.e. what is the electorate split)? And do those swing voters really care about that which the Democratic Party sees in the candidate they support? The DCCC is making what they believe is a pragmatic choice. Without proving reality bifurcates at infinite junctures and visiting alternative timelines … really no way to *know* if they are right or not.
 
In a district where 80% of the electorate are Democrats, the primary can figure out which guy those people want. But in a district that’s, say, 45% Democrat / 45% Republican / 10% swing voters … winning the general election requires nominating someone who appeals to that 10%. Coffman won in 2016 by 8% (30k votes). This *particular* district seems like one where the Democratic candidate needs to appeal to those who voted Republican for the last decade. Clinton won the district, so there’s some empirical evidence to support a belief that enough voters in the district *can* be swayed.
 
What I see wrong with it is not being up front about the pragmatic reasons for supporting the individual and allowing voters in the District to decide if *they* want to nominate a less progressive candidate.

Reality Check – The VA

Avatar
by Lisa

Alternative Fact: “We can talk about experience but the VA, when you think about 13 million people, you could take the head of the biggest hospital corporation of the world and it’s peanuts compared to the VA. So nobody has experience” — Trump on Fox & Friends this morning.

Real Fact: The VA does not have thirteen million employees, they’ve just just under 400k. By their own documentation, they have nine million enrolled veterans. Unless this number does not include dependents who *quality* to receive services *and* there are an additional four million qualified dependents … thirteen million is another Trump-ed number. Even if they’ve got thirteen million people enrolled in their health plan, the number of patient *visits* (i.e. one guy comes in every week, that’s fifty patient visits a year), a standard metric within the health care industry, is more useful (and, honestly, impressive sounding). They had 95 million outpatient visits and 700k inpatient admissions in 2015.

Now that’s a lot of employees , but Amazon has more. Amazon also has something like 300 million active customers. So it’s not like anyone anywhere is this size. But OK, he’s limiting it to hospital corporations.

Hospital Corporation of America has like 200 thousand employees and handles twenty seven million patient visits a year. Less, sure, but how many employees and patient visits does the White House doctor handle? It’s not like Trump went with the Cleveland Clinic guy who oversees fifty thousand employees and seven million patient visits and defends the choice saying anyone’s experience is going to need to scale when joining the VA.

The Truth Is Out There

Avatar
by Lisa

Hey, they *track* where aircraft go. Darn deep-state retroactively hacking into FAA data archived on third party sites to make Trump look bad! Turns out Trump was technically honest in telling Comey that he didn’t sleep in Moscow after the pagent – had the plane leave that night! No one asked about the night or two previous to the padgent.

I’ve pondered Trump’s ability to lie all.the.time without consequence – and it seems (to me) to hinge on the difference between “I outsmarted you” lying and “we’re all in on this one” lying. Will lying to an FBI agent be deemed OK because he’s part of this deep state out to get Trump? Will lying to the public to cover his own posterior be deemed OK because ‘the media’ (liberals, East coast elites) are out to get Trump? Will “they’re out to get me, so I had to lie” be a new class of falsehood approved by his supporters?

Data Privacy

Avatar
by Lisa

Facebook is getting a lot of attention for the information it gathers and how well it secures personal data you provide. We should look just as intently at other companies. Some provide services to individuals in exchange for advertising data, and some provide advertising targeting services without offering anything to the individuals being tracked.

LinkedIn — Maybe because “professional” information about oneself does not feel as private as that which is shared on Facebook, LinkedIn gets overlooked a bit. The companies I’ve worked for and titles I’ve held almost seem like public records. You can download a copy of “your data” (like Facebook, this is not apt to contain meta-data they’ve gathered regarding you – just data you have submitted to the site). In your settings, use the privacy tab and scroll down to “How LinkedIn uses your data” – the first selection is to download your data.

Nothing stunning – a list of contacts, my various employers and titles. But LinkedIn is trying to slurp in my entire contact list, maintain a web of people who know people, and allow advertisers to target users. There’s a whole tab apart from your privacy settings to control how your data is used for advertising purposes. “Advertisers” seem to be corporate hiring agents and recruiters, so this marketing is not always mentally classified as “advertising”.

LinkedIn also has a setting which allows you to opt-out (mine was on, and I’ve never opted in so I assume it is an opt-out deal) of having some of your data made available to third parties for policy and academic research.

And remember that Facebook Pixel? LinkedIn wants to track information about “websites you’ve visited” and “information you’ve shared with businesses” to show you more relevant jobs and ads.

Beyond the data feeling less private, having high-paying jobs that need my exact skill set and tend to hire people with my browsing history … well, that feels like a score compared to Facebook’s ad trying to coerce me once again to buy a pair of roller skates I already decided wouldn’t work for my daughter. Even if you’re not actively interested in changing jobs, it is nice to feel wanted. But that’s a nice veneer to data hording, analysis, and target marketing. They’ve even got a peculiar setting under the “Communications” tab that wants to use algorithms to analyze your messages to formulate suggested replies. This too seems to be an opt-out setting.

Google — no one uses Google+ (pity, that) but Google amasses information from searches, e-mails, Hangouts, Android phones. You can request an archive of your data through https://takeout.google.com — it takes a long time for the archive to be built, and it was an incredible amount of data. A few +1s from mis-clicks that there is no immediately obvious way to delete. “Bookmarks” that all appear to be map locations. A calendar that apparently was syncing with my home server back in 2009 since that’s the create date on all of the items. A whole folder for Chrome with 75 meg of browsing history and another meg of bookmarks (a meg of text is a *lot* of data, but I *love* that my bookmarks sync between devices). A handful of contacts that I assume my husband created in our shared account. The totality of every conversation I’ve ever had in Hangouts. Some Google Keep notes that I also assume are my husband’s from our shared account. My entire GMail mailbox, which is an obvious data source. The very tiny set of profile data I actually shared with Google.

Hell, Google has years worth of location data that I guess comes from my phone (it’s got fairly accurate lat/long coordinates, so GPS is the likely source). Following Google’s directions to delete the data didn’t work either (on the map, hit the hamburger menu then scroll ALL THE WAY DOWN to the ‘history’ selection”. Google both claims to have no history data for me and has 423 places on my timeline. Sooo, yeah, that would be history data. I finally managed to delete the stuff through my phone. There is a “Google Settings” app. Select “Location” from it, then “Google Location History”. There is a “Manage Activities” selection (use Google Maps to open it). Confirm you don’t want to use location history because, of course, it asks you to turn it on. Then use the hamburger menu button and select “Settings”. Waaay down at the bottom, there’s an option to delete all history or a date range of history. A couple of warnings later, the timeline map shows no data.

Then there are the photos. Gig after gig of photos. I had an Android phone that went into a reboot loop. I spent a few days wiping and reloading my phone, then failed back to an old phone. One of those iterations, evidently, slurped up all of the photos on my SD card because companies *want* your data. So the initial phone setup pushes you to backup your data, sync up your media, and generally upload ‘stuff’. One erroneous click and they’ve got metadata they’ll be able to keep forever. And there’s no readily apparent way to delete everything at once either. I’ve spent days on the web site deleting a couple hundred photos at a time. Not fun. Click the first picture, scroll down a bit, hold shift and click another picture. If you’re lucky, you didn’t select more than whatever the limit is (guessing 500) and you’ll get “389 Selected” in the upper left hand corner. At which point, you can click the delete and remove that chunk of photos. If you are not lucky, you get “2 Selected” and have to try again.

Ceasing data collection is much easier than removing data they’ve already grabbed. From your account settings, elect to “Manage your Google activity”. Then go into “Go To Activity Controls” and turn off (well, pause) whatever you want to turn off.

And I assume any bucket into which they’ve placed you based on previously gathered information will be retained even if you’ve deleted the underlying data.

 

Sendmail In CHROOT Jail

Avatar
by Lisa

Running our sendmail mail relay in a chroot jail, ‘make’ does not update sendmail config files with changes. While I’m certain there’s a way to sort that, it’s a lot easier to go back to the old-school way of updating sendmail.cf and sendmail’s hash files.

Modifying Sendmail Configuration (sendmail.mc) on Servers with CHROOT Jailed Sendmail

  1. SSH to server using your ID
  2. Change to the sendmail service account (e.g. sudo /bin/su – sendmail)
  3. Change directory to the jailed sendmail /etc/mail locatio (e.g. cd /smt00p20/sendmail/etc/mail)
  4. vi sendmail.mc
  5. Make requisite changes and save file
  6. m4 sendmail.mc > sendmail.cf
  7. Under your ID, restart sendmail using “sudo systemctl stop sendmail stop;sudo systemctl start sendmail”
  8. Validate changes

Modifying Sendmail Data Files on Servers with CHROOT Jailed Sendmail

  1. SSH to server using your ID
  2. Change to the sendmail service account (e.g. sudo /bin/su – sendmail)
  3. Change directory to the jailed sendmail /etc/mail locatio (e.g. cd /smt00p20/sendmail/etc/mail)
  4. vi filetoedit
  5. Make requisite changes and save file
  6. makemap hash ./filetoedit.db < ./filetoedit
  7. Under your ID, restart sendmail using  “sudo systemctl stop sendmail stop;sudo systemctl start sendmail”
  8. Validate changes

Where filetoedit is the name of the data file. For example, run “makemap hash ./access.db < ./access” to update the changes to the access file into access.db