I frequently use du -sh to determine what is using all the space on my Linux box … but mounting a data volume makes that simple command really suck (also, the mount point isn’t the problem for my local storage!). And all the /proc errors are just annoying — so pass in exclusions!
1 | du -sh --exclude='/mnt/*' --exclude='/proc/*' /* |
Scott has been trying to set up KRDP recently, and continued to get a lot of strange errors attempting to start the server. Through the GUI, it would fall over. From the command line, it output a lot of text. But they all seemed to indicate something couldn’t load. The log file had shared libraries (although ldd said all dependencies were met). The command line said things were found but could not run.
Had him run netstat to see if something else was bound to the port … and it was, but instead of printing the pid and binary name, it said off … which was a new one to me. Fortunately, lsof didshow us what was listening on the port. Stopped xrdp and, voila, krdp starts and runs.
[lisa@fedora01 ~/]# netstat -nap | grep 3389 tcp 0 0 0.0.0.0:3389 0.0.0.0:* LISTEN off... [lisa@fedora01 ~/]# lsof -i TCP:3389 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME xrdp 1855 root 13u IPv4 39470 0t0 TCP *:ms-wbt-server (LISTEN)
In retrospect, it does tell you what the problem is. ‘Unable to listen for connections on QHostAddress(“”) 0’ means “unable to bind to ip:port
Jan 29 06:49:14 fedora01 systemd[10239]: Started plasma-krdp_server.service - KRDP Server.
Jan 29 06:49:16 fedora01 krdpserver[11054]: libEGL warning: egl: failed to create dri2 screen
Jan 29 06:49:16 fedora01 krdpserver[11054]: libEGL warning: egl: failed to create dri2 screen
Jan 29 06:49:17 fedora01 krdpserver[11054]: org.kde.krdp: Unable to listen for connections on QHostAddress("") 0
Jan 29 06:49:17 fedora01 krdpserver[11054]: qt.dbus.integration: QDBusConnection: error: could not send message to service "org.freedesktop.portal.Desktop" path "/org/freedesktop/portal/desktop" interface "org.freedesktop.portal.Re moteDesktop" member "NotifyKeyboardKeycode": Marshalling failed: Invalid object path passed in arguments
Jan 29 06:49:17 fedora01 krdpserver[11054]: qt.dbus.integration: QDBusConnection: error: could not send message to service "org.freedesktop.portal.Desktop" path "/org/freedesktop/portal/desktop" interface "org.freedesktop.portal.Re moteDesktop" member "NotifyKeyboardKeycode": Marshalling failed: Invalid object path passed in arguments
Jan 29 06:49:17 fedora01 krdpserver[11054]: qt.dbus.integration: QDBusConnection: error: could not send message to service "org.freedesktop.portal.Desktop" path "/org/freedesktop/portal/desktop" interface "org.freedesktop.portal.Re moteDesktop" member "NotifyKeyboardKeycode": Marshalling failed: Invalid object path passed in arguments
Jan 29 06:49:17 fedora01 krdpserver[11054]: qt.dbus.integration: QDBusConnection: error: could not send message to service "org.freedesktop.portal.Desktop" path "/org/freedesktop/portal/desktop" interface "org.freedesktop.portal.Re moteDesktop" member "NotifyKeyboardKeycode": Marshalling failed: Invalid object path passed in arguments
Jan 29 06:49:17 fedora01 krdpserver[11054]: qt.dbus.integration: QDBusConnection: error: could not send message to service "org.freedesktop.portal.Desktop" path "/org/freedesktop/portal/desktop" interface "org.freedesktop.portal.Re moteDesktop" member "NotifyKeyboardKeycode": Marshalling failed: Invalid object path passed in arguments
Jan 29 06:49:17 fedora01 krdpserver[11054]: qt.dbus.integration: QDBusConnection: error: could not send message to service "org.freedesktop.portal.Desktop" path "/org/freedesktop/portal/desktop" interface "org.freedesktop.portal.Re moteDesktop" member "NotifyKeyboardKeycode": Marshalling failed: Invalid object path passed in arguments
Jan 29 06:49:17 fedora01 krdpserver[11054]: qt.dbus.integration: QDBusConnection: error: could not send message to service "org.freedesktop.portal.Desktop" path "/org/freedesktop/portal/desktop" interface "org.freedesktop.portal.Re moteDesktop" member "NotifyKeyboardKeycode": Marshalling failed: Invalid object path passed in arguments
Jan 29 06:49:17 fedora01 krdpserver[11054]: qt.dbus.integration: QDBusConnection: error: could not send message to service "org.freedesktop.portal.Desktop" path "/org/freedesktop/portal/desktop" interface "org.freedesktop.portal.Re moteDesktop" member "NotifyKeyboardKeycode": Marshalling failed: Invalid object path passed in arguments
Jan 29 06:49:17 fedora01 krdpserver[11054]: qt.dbus.integration: QDBusConnection: error: could not send message to service "org.freedesktop.portal.Desktop" path "" interface "org.freedesktop.portal.Session" member "Close": Object p ath cannot be empty
Jan 29 06:49:17 fedora01 systemd[10239]: plasma-krdp_server.service: Main process exited, code=exited, status=255/EXCEPTION
Jan 29 06:49:17 fedora01 systemd[10239]: plasma-krdp_server.service: Failed with result 'exit-code'.
adsaf
Ever since we upgraded to Fedora 41, we have been having horrible problems with our Exchange server. It will drop off the network for half an hour at a time. I cannot even ping the VM from the physical server. Some network captures show there’s no response to the ARP request.
Evidently, the VM configuration contains a machine type that doesn’t automatically update. We are using PC-Q35 as the chipset … and 4.1 was the version when we built our VMs. This version, however has been deprecated. Which you can see by asking virsh what capabilities it has:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 | 2025-01-02 23:17:26 [lisa@linux01 /var/log/libvirt/qemu/]# virsh capabilities | grep pc-q35 <machine maxCpus='288' deprecated='yes'>pc-q35-5.2</machine> <machine maxCpus='288' deprecated='yes'>pc-q35-4.2</machine> <machine maxCpus='255' deprecated='yes'>pc-q35-2.7</machine> <machine maxCpus='4096'>pc-q35-9.1</machine> <machine canonical='pc-q35-9.1' maxCpus='4096'>q35</machine> <machine maxCpus='288'>pc-q35-7.1</machine> <machine maxCpus='1024'>pc-q35-8.1</machine> <machine maxCpus='288' deprecated='yes'>pc-q35-6.1</machine> <machine maxCpus='255' deprecated='yes'>pc-q35-2.4</machine> <machine maxCpus='288' deprecated='yes'>pc-q35-2.10</machine> <machine maxCpus='288' deprecated='yes'>pc-q35-5.1</machine> <machine maxCpus='288' deprecated='yes'>pc-q35-2.9</machine> <machine maxCpus='288' deprecated='yes'>pc-q35-3.1</machine> <machine maxCpus='288' deprecated='yes'>pc-q35-4.1</machine> <machine maxCpus='255' deprecated='yes'>pc-q35-2.6</machine> <machine maxCpus='4096'>pc-q35-9.0</machine> <machine maxCpus='288' deprecated='yes'>pc-q35-2.12</machine> <machine maxCpus='288'>pc-q35-7.0</machine> <machine maxCpus='288'>pc-q35-8.0</machine> <machine maxCpus='288' deprecated='yes'>pc-q35-6.0</machine> <machine maxCpus='288' deprecated='yes'>pc-q35-4.0.1</machine> <machine maxCpus='288' deprecated='yes'>pc-q35-5.0</machine> <machine maxCpus='288' deprecated='yes'>pc-q35-2.8</machine> <machine maxCpus='288' deprecated='yes'>pc-q35-3.0</machine> <machine maxCpus='288'>pc-q35-7.2</machine> <machine maxCpus='288' deprecated='yes'>pc-q35-4.0</machine> <machine maxCpus='1024'>pc-q35-8.2</machine> <machine maxCpus='288'>pc-q35-6.2</machine> <machine maxCpus='255' deprecated='yes'>pc-q35-2.5</machine> <machine maxCpus='288' deprecated='yes'>pc-q35-2.11</machine> <machine maxCpus='288' deprecated='yes'>pc-q35-5.2</machine> <machine maxCpus='288' deprecated='yes'>pc-q35-4.2</machine> <machine maxCpus='255' deprecated='yes'>pc-q35-2.7</machine> <machine maxCpus='4096'>pc-q35-9.1</machine> <machine canonical='pc-q35-9.1' maxCpus='4096'>q35</machine> <machine maxCpus='288'>pc-q35-7.1</machine> <machine maxCpus='1024'>pc-q35-8.1</machine> <machine maxCpus='288' deprecated='yes'>pc-q35-6.1</machine> <machine maxCpus='255' deprecated='yes'>pc-q35-2.4</machine> <machine maxCpus='288' deprecated='yes'>pc-q35-2.10</machine> <machine maxCpus='288' deprecated='yes'>pc-q35-5.1</machine> <machine maxCpus='288' deprecated='yes'>pc-q35-2.9</machine> <machine maxCpus='288' deprecated='yes'>pc-q35-3.1</machine> <machine maxCpus='288' deprecated='yes'>pc-q35-4.1</machine> <machine maxCpus='255' deprecated='yes'>pc-q35-2.6</machine> <machine maxCpus='4096'>pc-q35-9.0</machine> <machine maxCpus='288' deprecated='yes'>pc-q35-2.12</machine> <machine maxCpus='288'>pc-q35-7.0</machine> <machine maxCpus='288'>pc-q35-8.0</machine> <machine maxCpus='288' deprecated='yes'>pc-q35-6.0</machine> <machine maxCpus='288' deprecated='yes'>pc-q35-4.0.1</machine> <machine maxCpus='288' deprecated='yes'>pc-q35-5.0</machine> <machine maxCpus='288' deprecated='yes'>pc-q35-2.8</machine> <machine maxCpus='288' deprecated='yes'>pc-q35-3.0</machine> <machine maxCpus='288'>pc-q35-7.2</machine> <machine maxCpus='288' deprecated='yes'>pc-q35-4.0</machine> <machine maxCpus='1024'>pc-q35-8.2</machine> <machine maxCpus='288'>pc-q35-6.2</machine> <machine maxCpus='255' deprecated='yes'>pc-q35-2.5</machine> <machine maxCpus='288' deprecated='yes'>pc-q35-2.11</machine> |
Or filtering out the deprecated ones …
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | 2025-01-02 23:16:50 [lisa@linux01 /var/log/libvirt/qemu/]# virsh capabilities | grep pc-q35 | grep -v "deprecated='yes'" <machine maxCpus='4096'>pc-q35-9.1</machine> <machine canonical='pc-q35-9.1' maxCpus='4096'>q35</machine> <machine maxCpus='288'>pc-q35-7.1</machine> <machine maxCpus='1024'>pc-q35-8.1</machine> <machine maxCpus='4096'>pc-q35-9.0</machine> <machine maxCpus='288'>pc-q35-7.0</machine> <machine maxCpus='288'>pc-q35-8.0</machine> <machine maxCpus='288'>pc-q35-7.2</machine> <machine maxCpus='1024'>pc-q35-8.2</machine> <machine maxCpus='288'>pc-q35-6.2</machine> <machine maxCpus='4096'>pc-q35-9.1</machine> <machine canonical='pc-q35-9.1' maxCpus='4096'>q35</machine> <machine maxCpus='288'>pc-q35-7.1</machine> <machine maxCpus='1024'>pc-q35-8.1</machine> <machine maxCpus='4096'>pc-q35-9.0</machine> <machine maxCpus='288'>pc-q35-7.0</machine> <machine maxCpus='288'>pc-q35-8.0</machine> <machine maxCpus='288'>pc-q35-7.2</machine> <machine maxCpus='1024'>pc-q35-8.2</machine> <machine maxCpus='288'>pc-q35-6.2</machine> |
So I shut down my Exchange server again (again, again), used “virsh edit “exchange01”, changed
1 2 3 4 | <os> <type arch='x86_64' machine='pc-q35-4.1'>hvm</type> <boot dev='hd'/></os> |
to
1 2 3 | <os> <type arch='x86_64' machine='pc-q35-7.1'>hvm</type></os> |
And started my VM. It took about an hour to boot. It absolutely hogged the disk physical server’s resources. It was the top listing in iotop -o
But then … all of the VMs dropped off of iotop. My attempt to log into the server via the console was logged in and waiting for me. My web mail, which had failed to load all day, was in my e-mail. And messages that had been queued for delivery had all come through.
The load on our physical server dropped from 30 to 1. Everything became responsive. And Exchange has been online for a good thirty minutes now.
We upgraded all of our internal servers to Fedora 41 after a power outage yesterday — had a number of issues to resolve (the liblockdev legacy config reverted so OpenHAB no longer could use USB serial devices, the physical server was swapping 11GB of data even though it had 81GB of memory free, and our Gerbera installation requires some libspdlog.so.1.12 which was updated to version 1.14 with the Fedora upgrade.
The last issue was more challenging to figure out because evidently DNF is now DNF5 and instead of throwing an error like “hey, new version dude! Use the new syntax” when you use an old command to list what is installed … it just says “No matching packages to list”. Like there are no packages installed? Since I’m using bash, openssh, etc … that’s not true.
Luckily, the new syntax works just fine. dnf repoquery –installed
Also:
dnf5 repoquery –available
dnf5 repoquery –userinstalled
Quick script to list interface info for the VMs
1 2 3 4 5 6 | #!/bin/bashfor vm in $(virsh list --name); do echo "Interfaces for VM: $vm" virsh domiflist "$vm" echo "--------------------------------"done |
I lost access to all of my Linux servers at work. And, unlike the normal report where nothing changed but xyz is now failing, I knew exactly what happened. A new access request had been approved about ten minutes previously. Looking at my ID, for some reason adding a new group membership changed account gid number to that new group. Except … that shouldn’t have actually dropped my access. If I needed the group to be my primary ID, I should have been able to use newgrp to switch contexts. Instead, I got prompted for a group password (which, yes, is a thing. No, no one uses it).
The hosts were set up to authenticate to AD using LDAP, and very successfully let me log in (or not, if I mistyped my password). They, however, would only see me as a member of my primary group. Well, today, I finally got a back door with sufficient access to poke around.
Turns out I was right — something was improperly configured so groups were not being read from the directory but rather implied from the gid value. I added the configuration parameter ldap_schema to instruct the server to use member instead of memberUid for memberships. I used rfc2307bis as that’s the value I was familiar with. I expect “AD” could be used as well, but figured we were well beyond AD 2008r2 and didn’t really want to dig farther into the nuanced differences between the two settings.
ldap_schema (string)
Specifies the Schema Type in use on the target LDAP server. Depending on the selected schema, the default attribute names retrieved from the servers may vary. The way that some attributes are handled may also differ.
Four schema types are currently supported:
The main difference between these schema types is how group memberships are recorded in the server. With rfc2307, group members are listed by name in the memberUid attribute. With rfc2307bis and IPA, group members are listed by DN and stored in the member attribute. The AD schema type sets the attributes to correspond with Active Directory 2008r2 values.
When sending configuration files to other people for reference, I like to redact any credential-type information … endpoints that allow you to post data without creds, auth configs, etc. Sometimes I replace the string with REDACTED and sometimes I just drop the line completely.
Make a copy of the config files elsewhere, then run sed
1 2 3 4 5 | # Retain parameter but replace value with REDACTED# Remove line from configsed -i '/authorization: Basic/d' *.yaml |
We upgraded Anya’s laptop to Fedora 40, and Skype has evidently moved from an installable RPM to a snap package. Which didn’t work with the firewall rules we built earlier in the year (video and audio calls would not connect); and, worse, nothing logs out. Looks like the netfilter kernel logging isn’t enabled
Enabled the logging:
echo 1 | sudo tee /proc/sys/net/netfilter/nf_log_all_netns
And, voila, we’ve got log records from nftables. And now Skype works … so I don’t know what to add. Sigh!
Column is an interesting command – it will turn delimited text into, well, columns. Simply tell it you want a table (-t) and indicate what separator to use (-s). Optionally, you can add table column headers
1 2 3 4 5 6 | [lisa@linux01 ~/]# cat /etc/group | column -t -s :root x 0 root,lisabin x 1daemon x 2...passim x 987 |
Alternately, you can use -J to get JSON-formatted output. Here you need the –table-columns as a comma delimited list of column names:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | [lisa@linux01 ~/]# cat /etc/group | column -J -s : --table-columns "group,password,gid,members"{ "table": [ { "group": "root", "password": "x", "gid": "0", "members": "root,lisa" },{ "group": "bin", "password": "x", "gid": "1", "members": null },{ "group": "daemon", "password": "x", "gid": "2", "members": null },{ "group": "passim", "password": "x", "gid": "987", "members": null } ]} |
Which can then be parsed with jq
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | [lisa@linux01 ~/]# cat /etc/group | column -J -s : --table-columns "group,password,gid,members" | jq '[.table[] | {group: .group, members: .members}]'[ { "group": "root", "members": "root,lisa" }, { "group": "bin", "members": null }, { "group": "daemon", "members": null }, { "group": "passim", "members": null }] |
This process presumes you have generated a signing key (/root/signing/MOK.priv and /root/signing/MOK.der) that has been registered for signing modules.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 | ################################################################################## Install from Repo and Sign Modules################################################################################yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpmyum install kernel-devel# Install kmod version of ZSyum install https://zfsonlinux.org/epel/zfs-release-2-3$(rpm --eval "%{dist}").noarch.rpmdnf config-manager --disable zfsdnf config-manager --enable zfs-kmodyum install zfs# And autoloadecho zfs >/etc/modules-load.d/zfs.conf# Use rpm -ql to list out the kernel modules that this version of ZFS uses -- 2.1.x has quite a few of them, and they each need to be signed# Sign zfs.ko and spl.ko in current kernel/usr/src/kernels/$(uname -r)/scripts/sign-file sha256 /root/signing/MOK.priv /root/signing/MOK.der /lib/modules/$(uname -r)/weak-updates/zfs/zfs/zfs.ko/usr/src/kernels/$(uname -r)/scripts/sign-file sha256 /root/signing/MOK.priv /root/signing/MOK.der /lib/modules/$(uname -r)/weak-updates/zfs/spl/spl.ko# And sign the bunch of other ko files in the n-1 kernel rev (these are symlinked from the current kernel)/usr/src/kernels/$(uname -r)/scripts/sign-file sha256 /root/signing/MOK.priv /root/signing/MOK.der /lib/modules/4.18.0-513.18.1.el8_9.x86_64/extra/zfs/avl/zavl.ko/usr/src/kernels/$(uname -r)/scripts/sign-file sha256 /root/signing/MOK.priv /root/signing/MOK.der /lib/modules/4.18.0-513.18.1.el8_9.x86_64/extra/zfs/icp/icp.ko/usr/src/kernels/$(uname -r)/scripts/sign-file sha256 /root/signing/MOK.priv /root/signing/MOK.der /lib/modules/4.18.0-513.18.1.el8_9.x86_64/extra/zfs/lua/zlua.ko/usr/src/kernels/$(uname -r)/scripts/sign-file sha256 /root/signing/MOK.priv /root/signing/MOK.der /lib/modules/4.18.0-513.18.1.el8_9.x86_64/extra/zfs/nvpair/znvpair.ko/usr/src/kernels/$(uname -r)/scripts/sign-file sha256 /root/signing/MOK.priv /root/signing/MOK.der /lib/modules/4.18.0-513.18.1.el8_9.x86_64/extra/zfs/unicode/zunicode.ko/usr/src/kernels/$(uname -r)/scripts/sign-file sha256 /root/signing/MOK.priv /root/signing/MOK.der /lib/modules/4.18.0-513.18.1.el8_9.x86_64/extra/zfs/common/zcommon.ko/usr/src/kernels/$(uname -r)/scripts/sign-file sha256 /root/signing/MOK.priv /root/signing/MOK.der /lib/modules/4.18.0-513.18.1.el8_9.x86_64/extra/zfs/zstd/zzstd.ko# Verify they are signed nowmodinfo -F signer /usr/lib/modules/$(uname -r)/weak-updates/zfs/zfs/zfs.komodinfo -F signer /usr/lib/modules/$(uname -r)/weak-updates/zfs/spl/spl.komodinfo -F signer /lib/modules/4.18.0-513.18.1.el8_9.x86_64/extra/zfs/avl/zavl.komodinfo -F signer /lib/modules/4.18.0-513.18.1.el8_9.x86_64/extra/zfs/icp/icp.komodinfo -F signer /lib/modules/4.18.0-513.18.1.el8_9.x86_64/extra/zfs/lua/zlua.komodinfo -F signer /lib/modules/4.18.0-513.18.1.el8_9.x86_64/extra/zfs/nvpair/znvpair.komodinfo -F signer /lib/modules/4.18.0-513.18.1.el8_9.x86_64/extra/zfs/unicode/zunicode.komodinfo -F signer /lib/modules/4.18.0-513.18.1.el8_9.x86_64/extra/zfs/zcommon/zcommon.komodinfo -F signer /lib/modules/4.18.0-513.18.1.el8_9.x86_64/extra/zfs/zstd/zzstd.ko# Rebootinit 6# And we've got ZFS, so create the poolzpool create pgpool sdczfs create zpool/zdatazfs set compression=lz4 zpool/zdatazfs get compressratio zpool/zdatazfs set mountpoint=/zpool/zdata zpool/zdata |
What happens if you only sign zfs.ko? All sorts of errors that look like there’s some sort of other problem — zfs will not load. It will tell you the required key is not available
1 | May 22 23:42:44 sandboxserver systemd-modules-load[492]: Failed to insert 'zfs': Required key not available |
Using insmod to try to manually load it will tell you there are dozens of unknown symbols:
1 2 3 4 5 | May 22 23:23:23 sandboxserver kernel: zfs: Unknown symbol ddi_strtoll (err 0)May 22 23:23:23 sandboxserver kernel: zfs: Unknown symbol spl_vmem_alloc (err 0)May 22 23:23:23 sandboxserver kernel: zfs: Unknown symbol taskq_empty_ent (err 0)May 22 23:23:23 sandboxserver kernel: zfs: Unknown symbol zone_get_hostid (err 0)May 22 23:23:23 sandboxserver kernel: zfs: Unknown symbol tsd_set (err 0) |
But the real problem is that there are unsigned modules so … there are unknown symbols. But not because something is incompatible. Just because the module providing that symbol will not load.