Category: System Administration

Network Manager GUI

You can use nmcli to configure network interfaces controlled by NetworkManager. But, honestly, I don’t see any advantage to learning the cryptic CLI instead of using the cryptic config file stuff I’ve already learned. And yet … I need my server to have /etc/resolv.conf populated when it reboots. So I figured out how to launch the KDE system settings (assuming you’ve got the X display redirected to your host) — systemsettings5

Mine takes a few minutes to render, during which time the window is black and a handful of errors are written out to the console. But it got there eventually, and I was able to edit the network interface.

Mosquitto 1.6.8 With Websockets Becomes Unresponsive

When I installed software on our new server, I got the “latest and greatest”. And have found that my mosquitto server hangs after about 20 minutes. No errors. Even strace doesn’t show anything beyond it not doing anything. I am able to use the command line utilities to confirm that the service isn’t there even though it looks like it is working.

Subscribe to the subtree of a topic:

mosquitto_sub -h mqtt.example.com -t testtopic/#

Publish a message to a topic:

mosquitto_pub -h mqtt.example.com -t testtopic/data/lisa -m "Test4"

To test WebSockets, I’ve put together a Python script that subscribes to a topic. Changing the commented lines switches between the WebSocket reverse proxy, the old and new MQTT servers via WebSockets, and the old and new MQTT servers directly in an attempt to isolate what is wrong.

import sys
import paho.mqtt.client as mqtt

def on_connect(mqttc, obj, flags, rc):
    print(f"rc: {str(rc)}")

def on_message(mqttc, obj, msg):
    print(f"{datetime.datetime.now()}: {msg.topic} {str(msg.qos)} {str(msg.payload)}")

def on_publish(mqttc, obj, mid):
    print(f"mid: {str(mid)}")

def on_subscribe(mqttc, obj, mid, granted_qos):
    print(f"Subscribed at {datetime.datetime.now()}: {str(mid)} {str(granted_qos)}")
def on_log(mqttc, obj, level, string):
    print(string)
# Client uses websockets
mqttc = mqtt.Client(transport='websockets', client_id="ljrTestingPythonScript", clean_session=False)
# Client uses MQTT directly
#mqttc = mqtt.Client(client_id="ljrTestingPythonScript", clean_session=False)

mqttc.username_pw_set("whateveruser", "wh@t3v3rP@s5w0rd")
mqttc.on_message = on_message
mqttc.on_connect = on_connect
mqttc.on_publish = on_publish
mqttc.on_subscribe = on_subscribe

#mqttc.connect("oldmosquitto.example.com", 80, 60)
mqttc.connect("newmosquitto.example.com", 80, 60)
#mqttc.connect("newmqtt.example.com", 1883, 60)
#mqttc.connect("newmqtt.example.com", 9001, 60)
#mqttc.connect("oldmqtt.example.com", 1883, 60)
#mqttc.connect("oldmqtt.example.com", 9001, 60)

mqttc.subscribe("owntracks/#", 0)

mqttc.loop_forever()

None of this helped, unfortunately. The reverse proxy couldn’t communicate with the MQTT server because it was unresponsive. Attempting to communicate with the server directly fails too.

I see a few issues in the Mosquitto repository that are similar — and the current discussion indicates that libwebsockets 3.2.0 introduced some incompatibility that they’ve addressed in Mosquitto 1.6.7 … since my problem relates to WebSockets, I wanted to try running the iteration before whatever changed.

git clone --branch v3.1.0 https://libwebsockets.org/repo/libwebsockets
cd libwebsockets
mkdir build
cd build
cmake ..
make
make install

git clone --branch v1.6.2 https://github.com/eclipse/mosquitto.git
cd mosquitto
vi config.mk # Change WITH_WEBSOCKETS:=no to :=yes
make
make install

ln -s /usr/local/lib/libwebsockets.so.14 /lib64/
ln -s /usr/local/lib/libmosquitto.so.1 /usr/lib64/libmosquitto.so.1

/usr/local/sbin/mosquitto -v -c /etc/mosquitto/mosquitto.conf

We’ll see if this is more reliable. It’s been up for 30 minutes … which is longer than the 1.6.8 iteration managed to run.

MariaDB Strangeness

I had to remove and reinstall MariaDB to upgrade from Fedora 29. I assumed Fedora 31 would have a more recent version, so I just installed the database and attempted to start it. Nope.

In journalctl

Dec 28 23:37:49 fedora123 mysql-prepare-db-dir[3977]: Database MariaDB is probably initialized in /var/lib/mysql already, nothing is done.
Dec 28 23:37:49 fedora123 mysql-prepare-db-dir[3977]: If this is not the case, make sure the /var/lib/mysql is empty before running mysql-prepare-db-dir.
Dec 28 23:37:49 fedora123 mysqld[4013]: 2019-12-28 23:37:49 0 [Note] /usr/libexec/mysqld (mysqld 10.3.20-MariaDB) starting as process 4013 …
Dec 28 23:37:49 fedora123 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE

In mariadb.log

2019-12-28 23:23:53 0 [ERROR] Fatal error: Can’t open and lock privilege tables: ‘mysql.user’ is not of type ‘TABLE’

Turns out the Fedora repo has 10.3, but the MariaDB repo that I’d used with Fedora 29 had MariaDB 10.4. After enabling the MariaDB repo and installing 10.4, the service without error.

QNAP – SMB1

We’ve got an old NAS, and after the most recent Windows Update … I lost access to it. Now the NAS is really just a backup, so I never really use the NAS. I don’t generally waste my time fixing things I don’t need, but Scott wasn’t able to use the NAS from Fedora 31 either. So I tried to map the drive again and got told the NAS could only use SMB1 but Windows required 2 or 3.

I ran smb2status and, yeah, the max protocol is SMB 1.0. Simply running smb3enable or smb21enable configures Samba for SMB3 or SMB2.1 and restarts the services. We can use the QNAP Samba share from both Windows 10 and Fedora 31 now.

The end of password changes?

I knew Microsoft was publishing recommendations against forced password expiry, but it was still surprising to see this banner in my Azure admin portal. It would be nice if their message was clearer on the nuances here — especially that enabling MFA (preferably not SMS-based MFA that is just asking for someone important’s number to get hijacked) is an important component of this recommendation.

In one of my first jobs, I was a sys admin for call-center systems. As such, I interacted with a lot of the call center management and staff … and, when you know someone in IT, you ping them when the proper support route isn’t as responsive as you’d like. Which is to say I did a good bit of end-user support as well. The number of people whose password was written on a post-it note under the keyboard astonished me. This particular call center didn’t have floating seating, but two or three people would share a cube because they worked different schedules. If I’ve got to come up with a new thing I need to remember every 90 days … well, that’s how you end up with Winter19, Spring20, Summer20, Autumn20 or Maggie12, Maggie13, Maggie14 passwords. That then get posted under the keyboard so I can remember that I’m up to “14” now. Couple that with the overhead of supporting password resets for those who didn’t write it down and happened to forget the password. I’d been a proponent of long password expiry coupled with increased complexity requirements. Maybe !Maggie-19? is good for all of 2019. It’s nice to see a major IT vendor starting to realize the real-world impact of IT policies.

Screen Mouse Scrolling Sends Arrows

I’m finding all sorts of things we researched and sorted on the old server without documenting 🙁 Today’s discovery is that screen uses alternate screens where the mouse wheel sends arrow keys. Which means you end up going through your history instead of scrolling up a few lines. Fix? Either create a .screenrc in the user’s account or globally change /etc/screenrc (which is what we do):

 

[root@server-replacement ~]# diff /etc/screenrc /root/oldserver-screenrc
216a217,218
>
> termcapinfo xterm* ti@:te@

Zoneminder, time zones, and php-fpm

I’m in the process of installing Zoneminder on our new server. It was a fairly straightforward process — stop Zoneminder on the old server, dump the SQL database, fix the DEFINER values since I’m using a central database server instead of a server on localhost, install Zoneminder, copy the config file, set up the database user, pull in the SQL file, and start it all up.

Visiting the website, I get “ZoneMinder is not installed properly: php’s date.timezone is not set to a valid timezone”. I’d forgotten to set the timezone in php.ini. Added ‘date.timezone = “America/New_York”‘, restarted httpd and Zoneminder. And got the same error.

<?php
error_reporting(E_ALL);
var_dump(ini_get('date.timezone'),date_default_timezone_get());
?>

It’s not set. This isn’t a funky Zoneminder thing — this is a PHP problem. I realized that PHP now runs as its own service. Restarting httpd is insufficient. Restarted php-ftm and the time zone I’d set in php.ini showed up. This is a case where a reboot would have sorted it … but good to remember that, when changing PHP settings, the php service needs to be restarted.

Samba and SELinux

I had a horrendous time trying to get the Samba share on our new server working. It worked insomuchas I could map a drive to the share … but I couldn’t actually see any files. Increasing the log level (smb.conf)

log level = 10 passdb:5 auth:5

showed that, yeah, I was getting a lot of access denied errors.

[2019/12/14 23:04:53.249959, 10, pid=17854, effective(0, 0), real(0, 0)] ../../source3/smbd/open.c:5438(create_file_unixpath)
create_file_unixpath: NT_STATUS_ACCESS_DENIED
[2019/12/14 23:04:53.249982, 10, pid=17854, effective(0, 0), real(0, 0)] ../../source3/smbd/open.c:5716(create_file_default)
create_file: NT_STATUS_ACCESS_DENIED
[2019/12/14 23:04:53.250012, 3, pid=17854, effective(0, 0), real(0, 0), class=smb2] ../../source3/smbd/smb2_server.c:3254(smbd_smb2_request_error_ex)
smbd_smb2_request_error_ex: smbd_smb2_request_error_ex: idx[1] status[NT_STATUS_ACCESS_DENIED] || at ../../source3/smbd/smb2_create.c:296
[2019/12/14 23:04:53.250038, 10, pid=17854, effective(0, 0), real(0, 0), class=smb2] ../../source3/smbd/smb2_server.c:3142(smbd_smb2_request_done_ex)
smbd_smb2_request_done_ex: idx[1] status[NT_STATUS_ACCESS_DENIED] body[8] dyn[yes:1] at ../../source3/smbd/smb2_server.c:3304

Many, many iterations of samba configs later, I wondered if SELinux was causing a problem. Temporarily disabling SELinux allowed files to be seen in the mapped drive … so that was the problem. I needed to tweak the SELinux settings to allow Samba to actually share files.

semanage fcontext -a -t samba_share_t "/data(/.*)?"

And

setsebool -P samba_export_all_rw=1

Apache — Switching to PHP-FPM

A few system updates ago, PHP fell over completely because of some multi-processing module. The quick fix was to change the multi-processing module and avoid having to figure out what changed and how to use php-fpm. Part of moving my VM’s to the new server, though, is cleaning up anything I’ve patched together as a quick fix. And, supposedly, php-fpm is a lot faster than the old-school Apache handler. Switching was a lot less involved than I had expected.

Install php-fpm:

dnf install php-fpm

Edit 00-mpm.conf

My quick fix was to switch to a non-default multi-processing module. That change is reverted to re-enable the ‘event’ module

vim /etc/httpd/conf.modules.d/00-mpm.conf

Configure Apache PHP Module

Verify the socket name used in /etc/php-fpm.d/ — Fedora is configured from /etc/php-fpm.d/www.conf with a socket at /var/run/php-fpm/www.sock

cp /etc/httpd/conf.modules.d/15-php.conf /etc/httpd/conf.modules.d/15-php.conf.orig
vi /etc/httpd/conf.modules.d/15-php.conf

# Handle files with .php extension using PHP interpreter

# Proxy declaration
<Proxy "unix:/var/run/php-fpm/www.sock|fcgi://php-fpm">
    	ProxySet disablereuse=off
</Proxy>

# Redirect to the proxy
<FilesMatch \.php$>
	SetHandler proxy:fcgi://php-fpm
</FilesMatch>

#
# Allow php to handle Multiviews
#
AddType text/html .php

#
# Add index.php to the list of files that will be served as directory
# indexes.
#
DirectoryIndex index.php

Enable php-fpm to auto-start, start php-fpm, and restart Apache

systemctl enable php-fpm
systemctl start php-fpm
systemctl restart httpd

Voila — phpinfo() confirms that I am using FPM/FastCGI

We’ll see if this actually does anything to improve performance!