Category: System Administration

Venafi Cert Issuance Fails after Windows 2022 Upgrade

Certificate Issuance Fails

After requesting a certificate, the request immediately fails with the error:

Failed to post CSR with error: Unknown certificate profile type.

I think it is just a coincidence, but wanted to document the scenario in case it comes up again. The application makes web calls to a vendor API to issue certs. The API calls, after the upgrade, were failing.

In this scenario, a call was being made to {base_url}/api/ssl/v1/types, the connection failed. Since the list of valid certificate profiles could not be retrieved, the request failed saying the certificate profile was unknown.

GET https://hard.cert-manager.com/api/ssl/v1/types?organizationId=####

Looking at a debug trace, the following flow was observed:

  • Authentication headers sent: login=<REDACTED>, password=<REDACTED>, customerUri=<REDACTED>
  • Transport-level failure (no HTTP status returned on the failing attempt)
    • Symptoms: “Decrypt failed with error 0X90317” followed by “The underlying connection was closed: The connection was closed unexpectedly.”
    • Context: Revocation checks reported “revocation server was offline,” then the client proceeded; long idle/keep-alive reuse likely contributed to the close.

 

Connection reuse vs server keep-alive: Apache is advertising Keep-Alive: timeout=3. The .NET client is reusing long-idle TLS connections via the proxy; by the time it sends application data, the server/proxy has already closed the session, leading to “underlying connection was closed” errors.

Revocation checks through the proxy: The .NET trace shows “revocation server was offline” before proceeding. That extra handshake work plus proxy blocking CRL/OCSP can increase latency and contribute to idle reuse issues.

.NET SChannel quirks: Older HttpWebRequest/ServicePoint behaviors (Expect100-Continue, connection pooling) can interact poorly with short keep-alive servers/proxies.

Luckily, this is a .NET application, and you can create custom configuration files for .NET apps. In the file with the binary, look for a text file named BinaryName.exe.config

If none exists, create one. The following disables the proxy:

<?xml version=”1.0″ encoding=”utf-8″?>
<configuration>
<system.net>
<!– Turn off use of the system proxy for this app –>
<defaultProxy enabled=”true”>
<proxy usesystemdefault=”false” />
</defaultProxy>
</system.net>
</configuration>

 

Client Connections to HTTPS IIS Site Fail After Upgrade to Windows Server 2022

Client connections to the HTTPS IIS site failed with the following error:

Secure Connection Failed

An error occurred during a connection to certmgr-dev.uniti.com.

PR_CONNECT_RESET_ERROR

Error code: PR_CONNECT_RESET_ERROR

The page you are trying to view cannot be shown because the authenticity of the received data could not be verified. Please contact the website owners to inform them of this problem.

 

The IIS site was set to “accept” client certificates.

  • Client Certificates = Accept means IIS/HTTP.sys will try to retrieve a client certificate only if the app touches Request.ClientCertificate (or a module that maps/validates client certs). That retrieval is done via TLS renegotiation in TLS 1.2.
  • On Server 2022, browsers prefer TLS 1.3. TLS 1.3 does not support the old renegotiation used to fetch a client cert mid‑request. When your app/module at “/” accesses the client cert, IIS attempts renegotiation, fails, and the connection is reset.

Setting Client Certificates to “Ignore” prevents IIS from attempting to renegotiate, so the site loads. This obviously isn’t a solution if you want to use client certificates to authenticate … but we’re authenticating through Ping, so don’t actually need the client certs.

Apache OIDC Authentication to PingFederate (or PingID) Using OIDC

This is kind of a silly update to my attempt to document using mod_auth_openidc in Apache. At the time, I didn’t know who set up the PingFederate side of the connection, so I just used Google as the authentication provider. Five years later, I am one of the people setting up the connections and can finally finish the other side. So here is an update — now using PingFederate as the OIDC/OAUTH provider.

OAUTH Client Setup – Apache

First, make sure mod_auth_openidc is installed

In your Apache config, you can add authentication to the entire site or just specific paths under the site. In this example, we are creating an authenticated sub-directory at /authtest

In the virtual host, I am adding an alias for the protected path as /authtest, configuring the directory, and configuring the location to require valid-user using openid-connect. I am then configuring the OIDC connection.

The OIDCClientID and OIDCClientSecret will be provided to you after the connection is set up in PingID. Just put placeholders in until the real values are known.

The OIDCRedirectURI needed to be a path under the protected directory for me – the Apache module handles the callback. Provide this path on the OIDC connection request.

The OIDCCryptoPassphrase just needs to be a long pseudo-random string. It can include special characters.

# Serve /authtest from local filesystem
Alias /authtest "/var/www/vhtml/sandbox/authtest/"

<Directory "/var/www/vhtml/sandbox/authtest">
Options -Indexes +FollowSymLinks
AllowOverride None
Require all granted
</Directory>

# mod_auth_openidc configuration for Ping (PingFederate/PingID)
# The firewall will need to be configured to allow web server to communicate with this host
# Prod: https://login.windstream.com/.well-known/openid-configuration
# UAT: https://login-test.windstream.com/.well-known/openid-configuration
# Dev: https://login-dev.windstream.com/.well-known/openid-configuration
OIDCProviderMetadataURL https://login-dev.windstream.com/.well-known/openid-configuration

# The ID and secret will be provided to you
OIDCClientID ddda3c59-7f2e-4f55-a269-b920c89f40d5
OIDCClientSecret p78…Q2kxB

# Redirect/callback URI – provide this in the request form for the callback URL
OIDCRedirectURI https://www.rushworth.us/authtest/callback

# Session/cookie settings – you make up the OIDCCryptoPassphrase
OIDCCryptoPassphrase "…T9y"
OIDCCookiePath /authtest
OIDCSessionInactivityTimeout 3600
OIDCSessionMaxDuration 28800

# Scopes and client auth
OIDCScope "openid profile email"
OIDCRemoteUserClaim preferred_username
OIDCProviderTokenEndpointAuth client_secret_basic

# If Ping's TLS cert at https://localhost:9031 isn't trusted by the OS CA store,
# install the proper CA chain, or temporarily disable validation (not recommended long-term):
# OIDCSSLValidateServer Off

# Protect the URL path with OIDC
<Location /authtest>
AuthType openid-connect
Require valid-user
OIDCUnAuthAction auth
</Location>

Sample web code for the “protected” page if you want to use the user’s ID. The user’s email is found at $_server[‘OIDC_CLAIM_email’]

[lisa@fedora conf.d]# cat /var/www/vhtml/sandbox/authtest/index.php
<?php
     if( isset($_SERVER['OIDC_CLAIM_iss']) && $_SERVER['OIDC_CLAIM_iss'] == "https://login-dev.windstream.com"){
          echo "I trust you are " . $_SERVER['OIDC_CLAIM_username'] . "\n";
     }

else{
     print "Not authenticated ... \n";
     print "<UL>\n";
     foreach($_SERVER as $key_name => $key_value) {
          print "<LI>" . $key_name . " = " . $key_value . "\n";
     }
     
     print "</UL>\n";
}
?>

Results on the web page – user will be directed to PingID to authenticate, and you will verify that login-dev.windstream.com (or login.windstream.com in production) has authenticated them as the OIDC_CLAIM_username value:

OAUTH Client Setup – PingID

Client auth, add redirect URLs

 

DNF5 History *UNDO*?!?

This is cool – I’ve only tested with a package I didn’t need and didn’t matter if it got mucked up. No idea if there’s an undo for, say, kernels. Or if undo on an update would roll back to the previous version. That’s the sort of testing to do on a sandbox that you don’t want running 30 minutes from now!

[root@fedora log]# dnf5 install bvi
Updating and loading repositories:
Repositories loaded.
Package Arch Version Repository Size
Installing:
bvi x86_64 1.5.0-1.fc43 fedora 157.1 KiB

Transaction Summary:
Installing: 1 package

Total size of inbound packages is 83 KiB. Need to download 83 KiB.
After this operation, 157 KiB extra will be used (install 157 KiB, remove 0 B).
Is this ok [y/N]: y
[1/1] bvi-0:1.5.0-1.fc43.x86_64 100% | 351.6 KiB/s | 83.0 KiB | 00m00s
———————————————————————————————————————————————————————————————————————————————
[1/1] Total 100% | 164.9 KiB/s | 83.0 KiB | 00m01s
Running transaction
[1/3] Verify package files 100% | 142.0 B/s | 1.0 B | 00m00s
[2/3] Prepare transaction 100% | 2.0 B/s | 1.0 B | 00m00s
[3/3] Installing bvi-0:1.5.0-1.fc43.x86_64 100% | 243.9 KiB/s | 160.0 KiB | 00m01s
Complete!
[root@fedora log]# dnf5 history list
ID Command line Date and time Action(s) Altered
29 dnf install bvi 2026-02-06 04:50:43 1
28 dnf remove kernel-core-6.18.3-200.fc43.x86_64 kernel-modules-6.18.3-200.fc43.x86_64 kernel-modules-extra-6.18.3-200.fc43.x86_64 2026-01-14 05:07:22 4
27 dnf update 2026-01-14 05:01:58 70
26 dnf remove kernel-core-6.17.9-300.fc43.x86_64 kernel-modules-6.17.9-300.fc43.x86_64 kernel-modules-extra-6.17.9-300.fc43.x86_64 2026-01-14 05:01:28 4
25 dnf update 2026-01-09 19:19:48 42
24 dnf update 2026-01-08 19:57:05 618
23 dnf5 remove kernel-core-6.11.10-300.fc41.x86_64 kernel-modules-6.11.10-300.fc41.x86_64 kernel-modules-core-6.11.10-300.fc41.x86_64 2026-01-08 19:53:42 3
22 dnf remove kernel-6.11.10-300.fc41 2025-12-26 02:06:47 1
21 dnf install speedtest-cli 2025-12-19 23:06:12 1
20 yum install chromedriver 2025-12-08 00:20:24 3
19 dnf system-upgrade download –releasever=43 2025-12-06 18:40:14 3792
18 dnf upgrade –refresh 2025-12-06 08:22:19 1422
17 dnf install -y cloud-utils-growpart 2025-12-06 08:16:00 1
16 dnf install xmlsec1 xmlsec1-openssl 2025-10-27 20:44:39 1
15 yum install xmlsec1 2025-10-27 20:42:29 1
14 dnf install mod_md 2025-07-03 19:22:36 1
13 yum install npm 2025-06-23 20:05:13 5
12 dnf update 2025-02-23 19:56:36 482
11 dnf update 2025-01-31 16:01:33 626
10 dnf5 install dnf5-plugin-automatic 2025-01-31 15:59:29 9
9 yum install xxd 2025-01-15 21:38:38 1
8 dnf install mosquitto 2025-01-04 23:40:52 3
7 dnf update 2025-01-03 04:51:41 14
6 dnf update 2025-01-01 00:19:22 358
5 dnf update 2024-12-06 04:30:32 18
4 dnf update 2024-12-06 04:14:41 404
3 dnf update 2024-11-22 17:21:43 65
2 dnf update 2024-11-18 18:33:18 116
1 dnf update 2024-11-14 18:58:09 54

[root@fedora log]# dnf5 history undo 29
Updating and loading repositories:
Repositories loaded.
Package Arch Version Repository Size
Removing:
bvi x86_64 1.5.0-1.fc43 fedora 157.1 KiB

Transaction Summary:
Removing: 1 package

After this operation, 157 KiB will be freed (install 0 B, remove 157 KiB).
Is this ok [y/N]: y
Running transaction
[1/2] Prepare transaction 100% | 4.0 B/s | 1.0 B | 00m00s
[2/2] Removing bvi-0:1.5.0-1.fc43.x86_64 100% | 35.0 B/s | 20.0 B | 00m01s
Complete!

Did you know … Powershell can create Visio diagrams!?!

I had to create a number of Visio diagrams for a new project. Since Blender has a Python API, I wondered if I could do something similar with Visio. There does appear to be an VSDX library for Python, I also found that Powershell can just control the Visio instance on my laptop.

This is a demo creating a diagram for a simple web server with a database back end. You can, however, use any stencils and make more complicated diagrams. The lines aren’t great — part of my Visio diagramming process is moving things around to optimize placement to avoid overlapping and confusing lines. The programmatic approach doesn’t do that, but it gets everything in the diagram. You can then move them as needed.

# Sample Visio diagram: Firewall -> Load Balancer -> Web Servers -> Database
# Auto-discovers stencils
# Works on Windows PowerShell 5.x

$ErrorActionPreference = "Stop"

# Output
$docName = "WebApp-LB-Firewall-DB.vsdx"
$outPath = Join-Path $HOME "Documents\$docName"

# Start Visio
$visio = New-Object -ComObject Visio.Application
$visio.Visible = $true

# New document/page
$doc = $visio.Documents.Add("")
$page = $visio.ActivePage
$page.Name = "Architecture"
$page.PageSheet.CellsU("PageWidth").ResultIU  = 22.0
$page.PageSheet.CellsU("PageHeight").ResultIU = 14.0

# -------------------------------
# Stencil discovery and loading
# -------------------------------

$searchRoots = @(
    "$env:PROGRAMFILES\Microsoft Office\root\Office16\Visio Content",
    "$env:PROGRAMFILES\Microsoft Office\root\Office16\Visio Content\1033",
    "$env:ProgramFiles(x86)\Microsoft Office\root\Office16\Visio Content",
    "$env:ProgramFiles(x86)\Microsoft Office\root\Office16\Visio Content\1033",
    "$env:PROGRAMFILES\Microsoft Office\root\Office15\Visio Content",
    "$env:ProgramFiles(x86)\Microsoft Office\root\Office15\Visio Content",
    "$env:PROGRAMFILES\Microsoft",
    "$env:ProgramFiles(x86)\Microsoft",
    "$env:PROGRAMFILES",
    "$env:ProgramFiles(x86)"
) | Where-Object { Test-Path $_ }

# Keywords to select useful stencils (filename match, case-insensitive)
$stencilKeywords = @("network","server","compute","computer","azure","cloud","firewall","security","database","sql","load","balancer","web","iis")

function Find-StencilFiles {
    param([string[]]$roots, [string[]]$keywords)
    $results = @()
    foreach ($root in $roots) {
        try {
            Get-ChildItem -Path $root -Filter *.vssx -Recurse -ErrorAction SilentlyContinue | ForEach-Object {
                $fname = $_.Name.ToLower()
                foreach ($kw in $keywords) {
                    if ($fname -match $kw) { $results += $_.FullName; break }
                }
            }
        } catch { }
    }
    $results | Select-Object -Unique
}

function Load-Stencils {
    param([string[]]$files)
    $loaded = @()
    foreach ($file in $files) {
        try {
            Write-Host "Loading stencil: $file"
            $loaded += $visio.Documents.OpenEx($file, 64) # read-only
        } catch {
            Write-Warning "Could not load stencil: $file"
        }
    }
    foreach ($docX in $visio.Documents) {
        if ($docX.FullName -ne $doc.FullName) { $loaded += $docX }
    }
    $loaded | Sort-Object FullName -Unique
}

$files = Find-StencilFiles -roots $searchRoots -keywords $stencilKeywords
$stencils = Load-Stencils -files $files

if (!$stencils -or $stencils.Count -eq 0) {
    Write-Warning "No stencil files loaded automatically. Fallback rectangles will be used."
} else {
    Write-Host "`nLoaded stencils:" -ForegroundColor Cyan
    foreach ($s in $stencils) { Write-Host " - $($s.FullName)" }
}

# -------------------------------
# Master selection helpers
# -------------------------------

function List-Masters {
    foreach ($st in $stencils) {
        Write-Host ("Stencil/Doc: {0}" -f $st.Name) -ForegroundColor Cyan
        foreach ($m in $st.Masters) {
            Write-Host ("  - {0} (NameU: {1})" -f $m.Name, $m.NameU)
        }
    }
}

function Get-MasterByPattern([string[]]$patterns) {
    foreach ($st in $stencils) {
        foreach ($m in $st.Masters) {
            foreach ($p in $patterns) {
                if ($m.NameU -match $p -or $m.Name -match $p) {
                    Write-Host ("Selected master '{0}' from '{1}' for pattern '{2}'" -f $m.Name, $st.Name, $p) -ForegroundColor Green
                    return $m
                }
            }
        }
    }
    return $null
}

# Drop master centered at x,y; keep default size; label it
function Add-Device([double]$x,[double]$y,[string]$label,[string[]]$patterns,[double]$fontSize=10) {
    $m = Get-MasterByPattern $patterns
    if ($null -eq $m) {
        Write-Warning ("No master matched patterns: {0}. Using fallback rectangle." -f ($patterns -join ", "))
        $w = 2.0; $h = 1.2
        $shape = $page.DrawRectangle($x - ($w/2), $y - ($h/2), $x + ($w/2), $y + ($h/2))
    } else {
        $shape = $page.Drop($m, $x, $y)
    }
    $shape.Text = $label
    $shape.CellsU("Char.Size").FormulaU = "$fontSize pt"
    return $shape
}

# Simple transparent containers (thin gray outline; sent behind shapes)
function Add-Container([double]$x,[double]$y,[double]$w,[double]$h,[string]$text) {
    $shape = $page.DrawRectangle($x, $y, $x + $w, $y + $h)
    $shape.CellsU("LineColor").FormulaU = "RGB(180,180,180)"
    $shape.CellsU("LineWeight").FormulaU = "1 pt"
    $shape.CellsU("FillForegnd").FormulaU = "RGB(255,255,255)"
    $shape.CellsU("FillForegndTrans").ResultIU = 1.0
    $shape.Text = $text
    $shape.CellsU("Char.Size").FormulaU = "12 pt"
    try { $shape.SendToBack() } catch {}
    return $shape
}

# Connector
function Connect($fromShape,$toShape,[string]$text="") {
    $conn = $page.Drop($visio.Application.ConnectorToolDataObject, 0, 0)
    $conn.CellsU("LineColor").FormulaU = "RGB(60,60,60)"
    $conn.CellsU("LineWeight").FormulaU = "0.75 pt"
    $fromShape.AutoConnect($toShape, 0, $conn)
    if ($text) { $conn.Text = $text }
    return $conn
}

# -------------------------------
# Diagram content
# -------------------------------

# Title
$title = $page.DrawRectangle(1.0, 13.4, 21.0, 13.9)
$title.Text = "Web App Architecture: Firewall -> Load Balancer -> Web Servers -> Database"
$title.CellsU("Char.Size").FormulaU = "14 pt"

# Patterns for official icons (broad to match common stencils)
$patFirewall    = @("Firewall|Security|Shield|Azure.*Firewall")
$patLoadBalancer= @("Load.*Balancer|Application.*Gateway|LB|Azure.*Load.*Balancer")
$patWebServer   = @("Web.*Server|IIS|Server(?! Rack)|Computer|Windows.*Server")
$patDatabase    = @("Database|SQL|Azure.*SQL|DB|Cylinder")

# Containers (optional zones)
$dmz     = Add-Container 1.0 10.8 20.0 2.0 "DMZ (Edge/Ingress)"
$webtier = Add-Container 4.0 6.8 14.0 3.2 "Web Tier"
$dbtier  = Add-Container 8.0 3.5 10.0 2.8 "Database Tier"
$clients = Add-Container 1.0 1.0 6.0 2.2 "Clients"

# Devices (kept at native size; spaced widely)
# Edge/Ingress
$fw      = Add-Device 3.0 11.8 "Firewall" $patFirewall 10
$lb      = Add-Device 8.0 11.8 "Load Balancer" $patLoadBalancer 10

# Web servers (pair)
$web1    = Add-Device 9.5 8.0 "Web Server 1\nIIS" $patWebServer 10
$web2    = Add-Device 13.5 8.0 "Web Server 2\nIIS" $patWebServer 10

# Database
$db      = Add-Device 13.0 4.6 "Database\nSQL" $patDatabase 10

# Clients
$client1 = Add-Device 2.0 1.8 "Client\nPC" @("Desktop|PC|Computer|Laptop") 10
$client2 = Add-Device 5.0 1.8 "Client\nServer" @("Server(?! Rack)|Windows.*Server|Computer") 10

# Connectors (flow: clients -> firewall -> LB -> web servers -> database)
Connect $client1 $fw "HTTPS"
Connect $client2 $fw "HTTPS"
Connect $fw $lb "Allow: 443"
Connect $lb $web1 "HTTP/HTTPS"
Connect $lb $web2 "HTTP/HTTPS"
Connect $web1 $db "SQL (1433/Encrypted)"
Connect $web2 $db "SQL (1433/Encrypted)"

# Save
$doc.SaveAs($outPath)
Write-Host "Saved Visio to: $outPath"

Expanding a qcow2-backed system disk (host + guest)

Expanding a qcow2-backed system disk (host + guest) — guest volume is lvm and xfs file system

HOST (resize qcow2)

  1. Optional backup:
    cp –reflink=auto /vms/fedora02.qcow2 /vms/fedora02.qcow2.bak
  2. Offline resize (VM stopped):
    qemu-img resize /vms/fedora02.qcow2 +5G
    # Start the VM after resizing.

GUEST (grow partition, PV, LV, filesystem)

  1. Confirm the disk shows the larger size:
    lsblk -o NAME,SIZE,TYPE,MOUNTPOINT
    #If needed:
    #partprobe /dev/sda
  2. Grow the LVM partition (sda2) to the end of the disk:
    dnf install -y cloud-utils-growpart
    growpart /dev/sda 2
    partprobe /dev/sda
  3. Resize the LVM PV and extend the root LV:
    pvresize /dev/sda2
    lvextend -l +100%FREE /dev/fedora/root
  4. Grow the filesystem:
    xfs_growfs /
  5. Verify:
    lsblk -o NAME,SIZE,TYPE,MOUNTPOINT
    df -h /

Exchange SMTP – Sender Reputation DB

Our Exchange server was refusing mail

451 4.7.0 Temporary server error. Please try again later. PRX5

Attempts to send mail would connect, send data, and then hang for a few seconds before returning the tempfail error.

Looks like there’s “sender reputation” data stored at .\Exchange Server\V15\TransportRoles\data\SenderReputation that is used. Since I’m not actually doing filtering on the Exchange server, stopping the transport services, moving the files out of the folder, and then re-starting the services rebuilt the data and allowed mail to send again.

Linux: Getting Drive Serial Number

[lisa@FVD01 /mnt/lisa/]# smartctl -i /dev/sdc
smartctl 7.5 2025-04-30 r5714 [x86_64-linux-6.15.7-200.fc42.x86_64] (local build)
Copyright (C) 2002-25, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Western Digital Red (CMR)
Device Model: WDC WD40EFRX-68N32N0
Serial Number: WD-WCC7K4HY5TKD
LU WWN Device Id: 5 0014ee 2b9a3d0c5
Firmware Version: 82.00A82
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Form Factor: 3.5 inches
Device is: In smartctl database 7.5/5706
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Tue Dec 2 17:24:27 2025 EST
SMART support is: Available – device has SMART capability.
SMART support is: Enabled

2025-12-02 17:24:27 [root@FPP01 /mnt/MythAndZoneminder/]# smartctl -i /dev/sda
smartctl 7.5 2025-04-30 r5714 [x86_64-linux-6.15.7-200.fc42.x86_64] (local build)
Copyright (C) 2002-25, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Western Digital Red (CMR)
Device Model: WDC WD40EFRX-68N32N0
Serial Number: WD-WCC7K7JZSZ0E
LU WWN Device Id: 5 0014ee 264576d5e
Firmware Version: 82.00A82
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Form Factor: 3.5 inches
Device is: In smartctl database 7.5/5706
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Tue Dec 2 17:24:38 2025 EST
SMART support is: Available – device has SMART capability.
SMART support is: Enabled