Category: System Administration

Daily Updates

My prediction that weekly updates are going to be a thing of the past WAY more quickly than the monthly patch and reboot became insufficient: I switched to daily updates at home. Had at least 40 updates each night since! This host runs sendmail and Apache HTTPD on a Linux OS – not a million different applications or anything.

​[lisa@fedora5 ~]# dnf history list
ID Command line              Date and time       Action(s) Altered
67 dnf update                2026-09-18 02:39:22                53
66 dnf update                2026-09-17 02:21:58                44
65 dnf update                2026-09-16 02:44:46                41
64 dnf update                2026-09-15 02:23:13                46

 

SPIRE Setup Documentation

Overview

This setup deploys SPIRE as follows:

  • SPIRE Server on <SPIRE_SERVER_HOST>
  • SPIRE Agent on <SPIRE_AGENT_HOST>
  • Trust domain: <TRUST_DOMAIN>
  • Server/agent communication port: <SERVER_PORT>/tcp

How it works

SPIRE provides machine and workload identity.

The SPIRE Server on <SPIRE_SERVER_HOST> is the trust authority for the trust domain <TRUST_DOMAIN>.

The SPIRE Agent on <SPIRE_AGENT_HOST> attests to the server using x509pop with an X.509 certificate issued by an enterprise/internal CA.

Applications on <SPIRE_AGENT_HOST> do not talk directly to the SPIRE Server. They talk to the local SPIRE Agent over the local workload API socket <AGENT_SOCKET_PATH>.

The agent returns an X.509-SVID representing the workload identity.

Installation Instructions

  1. Install SPIRE binaries

Run on both hosts:

mkdir -p /opt/spire
cd /tmp
wget https://github.com/spiffe/spire/releases/download/v1.15.2/spire-1.15.2-linux-amd64-musl.tar.gz
tar zxf spire-1.15.2-linux-amd64-musl.tar.gz
cp -r spire-1.15.2/. /opt/spire/

On the SPIRE Server host:

ln -sf /opt/spire/bin/spire-server /usr/bin/spire-server

On the SPIRE Agent host:

ln -sf /opt/spire/bin/spire-agent /usr/bin/spire-agent

  1. Configure SPIRE Server

Create directories:

mkdir -p /opt/spire/conf
mkdir -p /opt/spire/data/server
mkdir -p /opt/spire/conf/x509pop

Place the CA bundle at:

/opt/spire/conf/x509pop/bundle.pem

Contents:

—–BEGIN CERTIFICATE—–
<REDACTED CA CERTIFICATE>
—–END CERTIFICATE—–
—–BEGIN CERTIFICATE—–
<REDACTED CA CERTIFICATE>
—–END CERTIFICATE—–
—–BEGIN CERTIFICATE—–
<REDACTED CA CERTIFICATE>
—–END CERTIFICATE—–

Create /opt/spire/conf/server.conf:

server {
bind_address = “0.0.0.0”
bind_port = “<SERVER_PORT>”
trust_domain = “<TRUST_DOMAIN>”
data_dir = “/opt/spire/data/server”
log_level = “INFO”
}

plugins {
DataStore “sql” {
plugin_data {
database_type = “sqlite3”
connection_string = “/opt/spire/data/server/datastore.sqlite3”
}
}

NodeAttestor “x509pop” {
plugin_data {
ca_bundle_path = “/opt/spire/conf/x509pop/bundle.pem”
}
}

KeyManager “memory” {
plugin_data {}
}
}

health_checks {
listener_enabled = true
bind_address = “127.0.0.1”
bind_port = “<HEALTH_PORT>”
}

Create /etc/systemd/system/spire-server.service:

[Unit]
Description=SPIRE Server
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/opt/spire/bin/spire-server run -config /opt/spire/conf/server.conf
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Start server:

systemctl daemon-reload
systemctl enable –now spire-server
systemctl status spire-server –no-pager

Validate server:

spire-server healthcheck

  1. Configure SPIRE Agent

Create directories:

mkdir -p /opt/spire/conf
mkdir -p /opt/spire/data/agent
mkdir -p /opt/spire/sockets
mkdir -p /opt/spire/conf/x509pop

Issue a certificate through your enterprise PKI platform. Download as OpenSSL format and split CRT/KEY files. Copy the node certificate to:

/opt/spire/conf/x509pop/agent.crt

Copy the node private key to:

/opt/spire/conf/x509pop/agent.key

The private key must be unencrypted PEM.

Set permissions:

chmod 700 /opt/spire/conf/x509pop
chmod 600 /opt/spire/conf/x509pop/agent.key
chmod 644 /opt/spire/conf/x509pop/agent.crt

Create /opt/spire/conf/agent.conf:

agent {
data_dir = “/opt/spire/data/agent”
log_level = “INFO”
trust_domain = “<TRUST_DOMAIN>”
server_address = “<SPIRE_SERVER_HOST>”
server_port = “<SERVER_PORT>”
socket_path = “<AGENT_SOCKET_PATH>”
insecure_bootstrap = true
}

plugins {
KeyManager “disk” {
plugin_data {
directory = “/opt/spire/data/agent”
}
}

WorkloadAttestor “unix” {
plugin_data {}
}

NodeAttestor “x509pop” {
plugin_data {
private_key_path = “/opt/spire/conf/x509pop/agent.key”
certificate_path = “/opt/spire/conf/x509pop/agent.crt”
}
}
}

Create /etc/systemd/system/spire-agent.service:

[Unit]
Description=SPIRE Agent
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/opt/spire/bin/spire-agent run -config /opt/spire/conf/agent.conf
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Start agent:

systemctl daemon-reload
systemctl enable –now spire-agent
systemctl status spire-agent –no-pager

  1. Validate x509pop agent attestation

On the SPIRE Server host:

spire-server agent list

Expected result:

  • Agent attestation type is x509pop
  • Can re-attest is true
  • Parent ID format resembles:

spiffe://<TRUST_DOMAIN>/spire/agent/x509pop/<AGENT_HASH>

  1. Create workload registration entry

Use the current x509pop agent SPIFFE ID from spire-server agent list.

On the SPIRE Server host:

spire-server entry create \
-spiffeID spiffe://<TRUST_DOMAIN>/workload/<WORKLOAD_NAME> \
-parentID spiffe://<TRUST_DOMAIN>/spire/agent/x509pop/<AGENT_HASH> \
-selector unix:uid:0

This authorizes a root-owned process on the SPIRE Agent host.

  1. Fetch workload identity on the SPIRE Agent host

/opt/spire/bin/spire-agent api fetch x509 -socketPath <AGENT_SOCKET_PATH>

Expected SPIFFE ID:

spiffe://<TRUST_DOMAIN>/workload/<WORKLOAD_NAME>

  1. Write certs to disk for testing

Create destination directory:

mkdir -p /etc/spire/svid/test
chmod 700 /etc/spire/svid/test

Write files:

/opt/spire/bin/spire-agent api fetch x509 \
-socketPath <AGENT_SOCKET_PATH> \
-write /etc/spire/svid/test

Inspect output:

ls -l /etc/spire/svid/test

openssl x509 -in /etc/spire/svid/test/svid.0.pem -text -noout

  1. Reboot persistence validation

Reboot the SPIRE Agent host.

After reboot, validate:

systemctl status spire-agent –no-pager

ls -l <AGENT_SOCKET_PATH>

/opt/spire/bin/spire-agent api fetch x509 -socketPath <AGENT_SOCKET_PATH>

Expected behavior:

  • spire-agent starts automatically
  • Workload API socket exists
  • X.509-SVID fetch succeeds
  1. Operational notes
  • Current architecture: <SPIRE_SERVER_HOST> = SPIRE Server; <SPIRE_AGENT_HOST> = SPIRE Agent
  • Current trust domain: <TRUST_DOMAIN>
  • Current server/agent path: <SPIRE_AGENT_HOST> to <SPIRE_SERVER_HOST> on TCP <SERVER_PORT>
  • Current Workload API socket: <AGENT_SOCKET_PATH>
  • Current example workload selector: unix:uid:0
  • Current example workload identity: spiffe://<TRUST_DOMAIN>/workload/<WORKLOAD_NAME>

JWT

Create JWT registration on the SPIRE Server host:

spire-server entry create \
-spiffeID spiffe://<TRUST_DOMAIN>/workload/<JWT_WORKLOAD_NAME> \
-parentID spiffe://<TRUST_DOMAIN>/spire/agent/x509pop/<AGENT_HASH> \
-selector unix:uid:0

Fetch JWT-SVID from the SPIRE Agent host:

/opt/spire/bin/spire-agent api fetch jwt \
-socketPath <AGENT_SOCKET_PATH> \
-audience <JWT_AUDIENCE> \
-spiffeID spiffe://<TRUST_DOMAIN>/workload/<JWT_WORKLOAD_NAME>

Example output:

token(spiffe://<TRUST_DOMAIN>/workload/<JWT_WORKLOAD_NAME>):
<REDACTED JWT-SVID>

bundle(spiffe://<TRUST_DOMAIN>):
<REDACTED JWKS BUNDLE>

SPIRE OIDC Discovery Provider

On the SPIRE Server host:

mkdir -p /opt/spire-extras
cd /tmp
wget https://github.com/spiffe/spire/releases/download/v1.15.2/spire-extras-1.15.2-linux-amd64-musl.tar.gz
tar zxf spire-extras-1.15.2-linux-amd64-musl.tar.gz
cp -r spire-extras-1.15.2/ /opt/spire-extras/

ln -sf /opt/spire-extras/bin/oidc-discovery-provider /usr/bin/oidc-discovery-provider
mkdir -p /opt/spire-extras/conf/oidc-discovery-provider

Create certificate and key files:

/opt/spire-extras/conf/oidc-discovery-provider/tls.crt
/opt/spire-extras/conf/oidc-discovery-provider/tls.key

Set permissions:

chmod 644 /opt/spire-extras/conf/oidc-discovery-provider/tls.crt
chmod 600 /opt/spire-extras/conf/oidc-discovery-provider/tls.key

Create /opt/spire-extras/conf/oidc-discovery-provider/oidc-discovery-provider.conf:

log_level = “INFO”

domains = [“<OIDC_DISCOVERY_DOMAIN>”]

server_api {
address = “unix://<SPIRE_SERVER_API_SOCKET>”
}

serving_cert_file {
cert_file_path = “/opt/spire-extras/conf/oidc-discovery-provider/tls.crt”
key_file_path = “/opt/spire-extras/conf/oidc-discovery-provider/tls.key”
}

Create /etc/systemd/system/spire-oidc-discovery-provider.service:

[Unit]
Description=SPIRE OIDC Discovery Provider
After=network-online.target spire-server.service
Wants=network-online.target

[Service]
Type=simple
ExecStart=/opt/spire-extras/bin/oidc-discovery-provider -config /opt/spire-extras/conf/oidc-discovery-provider/oidc-discovery-provider.conf
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Ping Integration

PingFederate Integration Note for SPIRE JWT Validation

Purpose

Configure PingFederate to trust and validate JWTs issued from the SPIRE environment.

SPIRE issuer details

Use these values:

  • Issuer: https://<OIDC_DISCOVERY_DOMAIN>
  • OIDC discovery URL: https://<OIDC_DISCOVERY_DOMAIN>/.well-known/openid-configuration
  • JWKS URL: https://<OIDC_DISCOVERY_DOMAIN>/keys

Trust model

PingFederate should validate JWT signatures using the JWKS published by the SPIRE OIDC Discovery Provider.

Ping does not need to call the SPIRE server directly for every token validation. It should use the discovery/JWKS metadata from the OIDC Discovery Provider.

Expected JWT characteristics

Issuer

Ping should require:

iss = https://<OIDC_DISCOVERY_DOMAIN>

Audience

Recommended audience value:

<PING_AUDIENCE>

Clients requesting JWT-SVIDs from SPIRE should request them with this audience.

Subject

The workload identity will be in:

sub

Example:

spiffe://<TRUST_DOMAIN>/workload/<WORKLOAD_NAME>

This is the primary identity claim Ping should use to identify the calling workload.

Recommended validation rules in Ping

Validate:

  • JWT signature against SPIRE JWKS
  • iss matches https://<OIDC_DISCOVERY_DOMAIN>
  • aud contains <PING_AUDIENCE>
  • token is within validity window (exp, iat)
  • sub is an allowed SPIFFE ID or matches allowed policy rules

Example workload identity currently in use

Current example SPIFFE ID:

spiffe://<TRUST_DOMAIN>/workload/<WORKLOAD_NAME>

Client-side JWT retrieval model

A workload on the SPIRE Agent host should obtain its JWT from the local SPIRE agent, not from the SPIRE server directly.

Local agent socket:

<AGENT_SOCKET_PATH>

Example operational flow

  1. Workload on the SPIRE Agent host requests a JWT-SVID from the local SPIRE agent.
  2. JWT-SVID is issued with:
  • issuer = https://<OIDC_DISCOVERY_DOMAIN>
  • audience = <PING_AUDIENCE>
  • subject = workload SPIFFE ID
  1. Workload presents JWT to PingFederate.
  2. PingFederate validates the JWT using SPIRE OIDC discovery/JWKS.
  3. PingFederate maps the SPIFFE workload identity to access policy, token issuance, or downstream application authorization.

Suggested placeholder legend

| Placeholder | Meaning |
|—|—|

| <SPIRE_SERVER_HOST> | SPIRE server hostname |

| <SPIRE_AGENT_HOST> | SPIRE agent hostname |

| <TRUST_DOMAIN> | SPIRE trust domain |

| <SERVER_PORT> | SPIRE server listener port |

| <HEALTH_PORT> | Health check port |

| <AGENT_SOCKET_PATH> | Local SPIRE Agent workload API socket |

| <AGENT_HASH> | x509pop parent/agent hash |

| <WORKLOAD_NAME> | Example X.509 workload name |

| <JWT_WORKLOAD_NAME> | Example JWT workload name |

| <JWT_AUDIENCE> | JWT audience used by client |

| <PING_AUDIENCE> | Audience PingFederate validates |

| <OIDC_DISCOVERY_DOMAIN> | Public/abstracted OIDC issuer hostname |

| <SPIRE_SERVER_API_SOCKET> | SPIRE server private API socket |

 

Windows Defender Auto-Updates Fail

Windows Defender is supposed to automatically update itself. But I periodically experience these updates getting ‘stuck’. To force an update from the CLI:

cd “C:\ProgramData\Microsoft\Windows Defender\Platform”
dir
# REM cd into the latest iteration
cd 4.18.26060.3008-0
MpCmdRun.exe -removedefinitions -dynamicsignatures
MpCmdRun.exe -SignatureUpdate

  • -removedefinitions -dynamicsignatures removes only the dynamic security intelligence/signature content already present on the box. In practice, this is a rollback/flush step. It is commonly used when you want to clear bad/poisoned/cached dynamic detections.
  • -SignatureUpdate does the fetch/install step: it contacts the configured update source and downloads the current antimalware definitions/security intelligence.

If all else fails, download latest from https://www.microsoft.com/en-us/wdsi/defenderupdates and manually install it

Fedora: Disk Cleanup

Useful commands for disk cleanup …

Subfolder size excluding specific folders

[root@FVP05 ~]# du -shx --exclude=/mnt --exclude=/proc /*
0 /afs
0 /bin
312M /boot
2.1M /ca
8.0K /cacert.pem
4.0K /careq.pem
0 /certs
0 /crl
0 /dev
48M /etc
37M /home
0 /lib
0 /lib64
0 /media
0 /newcerts
0 /openhab
954M /opt
4.0K /private
457M /root
1.2M /run
120K /SampleCode
0 /sbin
0 /srv
0 /swapfile
0 /sys
227M /tmp
8.2G /usr
14G /var

List the largest n RPM installs

[root@FVP05 ~]# rpm -qa –qf ‘%{SIZE}\t%{NAME}\n’ | sort -nr | head -10

488208897 azure-cli
248737290 java-latest-openjdk-headless
238259971 glibc-all-langpacks
182326049 mesa-vulkan-drivers
147275895 llvm-libs
127750222 gcc
106627293 kernel-core
106091788 kernel-core
105915244 nvidia-gpu-firmware
105470864 kernel-modules

And clean up the journal logs

journalctl --vacuum-size=100M

 

Flatpak – Escaping the Sandbox

We’ve been running Cura from a flatpak because the rpm distributed version was out of date. The big drawback, though, is that this flatpak could not see the files from network mounts. The mount is fine – in fstab, same user account can interact with those files in other applications. Just not this flatpak thing.

Turns out that’s normal – flatpaks operate in a sort of sandbox. You just have to tell it to let an individual flatpak access the location where the network mounts are. In this case, the /mnt path:

flatpack override com.ultimaker.cura --filesystem=/mnt

MiniPlasma Mitigation

Microsoft Windows Cloud Files Mini Filter Driver Elevation of Privilege Vulnerability (alternately referred to as MiniPlasma) is one of those findings that is a little confusing. The vulnerability appears tied to an issue Microsoft originally addressed in 2020, which raises the obvious question: if it was already patched, why is it relevant again now?

The original fix may have closed a specific exploitation path without fully eliminating the underlying bug class. The remediation may have been incomplete. Or a later code change may have reintroduced a previously fixed condition. Without updated vendor detail, it is hard to say exactly which of those happened; but the operational conclusion is the same: a system can still be exposed today even if the original 2020 patch was installed.

For environments looking for a practical mitigation, the good news is that many servers do not need Windows Cloud Files functionality at all. The vulnerable component is the Cloud Files Mini Filter Driver (CldFlt), which supports placeholder and hydration behavior used by features such as OneDrive Files On-Demand and other CfAPI-based integrations.

That makes CldFlt a viable mitigation target. If the filter is loaded but not attached to any volumes, there is a good chance it can be safely unloaded and disabled. This does not remove the driver from disk, but it does remove the active kernel attack surface associated with the running minifilter. Since this is an elevation-of-privilege issue, that distinction matters: the goal is not to claim the file no longer exists, but to prevent the vulnerable driver from being active in the system.

The following process checks whether any volumes are associated with CldFlt, temporarily unloads the filter, verifies that the system continues functioning normally, and then disables the driver persistently.

REM Check instances on CldFlt — if 0, proceed with testing disablement
fltmc filters

REM Example output:
REM Filter Name Num Instances Altitude Frame
REM —————————— ————- ———— —–
REM bindflt 0 409800 0
REM MsSecFlt 9 385600 0
REM CSAgent 9 321410.78870 0
REM storqosflt 0 244000 0
REM wcifs 0 189900 0
REM CldFlt 0 180451 0
REM FileCrypt 0 141100 0
REM UnionFS 0 130850 0
REM npsvctrig 1 46000 0
REM Wof 1 40700 0

REM Unload cldflt
fltmc unload cldflt

REM Verify stopped
sc query cldflt

REM Verify no longer in filters list
fltmc filters

REM Verify applications and expected file operations still work normally
REM If everything looks good, disable persistent startup
sc config cldflt start= disabled

Fedora 43 to 44 upgrade: dnf5 plugin load failure after upgrade

After completing an in-place upgrade from Fedora 43 to Fedora 44, dnf5 failed to run with this error:

[lisa@fedora05 ~]# dnf5 update
Cannot load dnf5 plugin: /usr/lib64/dnf5/plugins/automatic_cmd_plugin.so
Cannot load shared library “/usr/lib64/dnf5/plugins/automatic_cmd_plugin.so”: libdnf5-cli.so.2: cannot open shared object file: No such file or directory

What happened

The Fedora 44 upgrade completed, and the installed dnf5 packages were all current Fedora 44 versions. However, there was a leftover plugin file still sitting in /usr/lib64/dnf5/plugins/automatic_cmd_plugin.so.

That file was not owned by any RPM package and had been built against an older library, libdnf5-cli.so.2.

But Fedora 44 had /usr/lib64/libdnf5-cli.so.3.

dnf5 was trying to load a stale plugin from before the upgrade.

How I verified it

These commands showed the problem:

ls -l /usr/lib64/libdnf5-cli.so*
rpm -qf /usr/lib64/dnf5/plugins/automatic_cmd_plugin.so
ldd /usr/lib64/dnf5/plugins/automatic_cmd_plugin.so

Results:

  • libdnf5-cli.so.3 existed
  • automatic_cmd_plugin.so was not owned by any package
  • ldd showed it was looking for libdnf5-cli.so.2

Fix

Remove the orphaned plugin file:

rm /usr/lib64/dnf5/plugins/automatic_cmd_plugin.so
ldconfig
dnf5 update

After deleting the stale plugin, dnf5 worked normally again.

Root cause

This appears to be a leftover orphaned dnf5 plugin from before the major version upgrade. Even though the main dnf5 and libdnf5 packages were updated correctly, dnf5 still tried to load the old .so file it found in the plugins directory.