Category: Technology

Venafi Cert Issuance Fails after Windows 2022 Upgrade

Certificate Issuance Fails

After requesting a certificate, the request immediately fails with the error:

Failed to post CSR with error: Unknown certificate profile type.

I think it is just a coincidence, but wanted to document the scenario in case it comes up again. The application makes web calls to a vendor API to issue certs. The API calls, after the upgrade, were failing.

In this scenario, a call was being made to {base_url}/api/ssl/v1/types, the connection failed. Since the list of valid certificate profiles could not be retrieved, the request failed saying the certificate profile was unknown.

GET https://hard.cert-manager.com/api/ssl/v1/types?organizationId=####

Looking at a debug trace, the following flow was observed:

  • Authentication headers sent: login=<REDACTED>, password=<REDACTED>, customerUri=<REDACTED>
  • Transport-level failure (no HTTP status returned on the failing attempt)
    • Symptoms: “Decrypt failed with error 0X90317” followed by “The underlying connection was closed: The connection was closed unexpectedly.”
    • Context: Revocation checks reported “revocation server was offline,” then the client proceeded; long idle/keep-alive reuse likely contributed to the close.

 

Connection reuse vs server keep-alive: Apache is advertising Keep-Alive: timeout=3. The .NET client is reusing long-idle TLS connections via the proxy; by the time it sends application data, the server/proxy has already closed the session, leading to “underlying connection was closed” errors.

Revocation checks through the proxy: The .NET trace shows “revocation server was offline” before proceeding. That extra handshake work plus proxy blocking CRL/OCSP can increase latency and contribute to idle reuse issues.

.NET SChannel quirks: Older HttpWebRequest/ServicePoint behaviors (Expect100-Continue, connection pooling) can interact poorly with short keep-alive servers/proxies.

Luckily, this is a .NET application, and you can create custom configuration files for .NET apps. In the file with the binary, look for a text file named BinaryName.exe.config

If none exists, create one. The following disables the proxy:

<?xml version=”1.0″ encoding=”utf-8″?>
<configuration>
<system.net>
<!– Turn off use of the system proxy for this app –>
<defaultProxy enabled=”true”>
<proxy usesystemdefault=”false” />
</defaultProxy>
</system.net>
</configuration>

 

Client Connections to HTTPS IIS Site Fail After Upgrade to Windows Server 2022

Client connections to the HTTPS IIS site failed with the following error:

Secure Connection Failed

An error occurred during a connection to certmgr-dev.uniti.com.

PR_CONNECT_RESET_ERROR

Error code: PR_CONNECT_RESET_ERROR

The page you are trying to view cannot be shown because the authenticity of the received data could not be verified. Please contact the website owners to inform them of this problem.

 

The IIS site was set to “accept” client certificates.

  • Client Certificates = Accept means IIS/HTTP.sys will try to retrieve a client certificate only if the app touches Request.ClientCertificate (or a module that maps/validates client certs). That retrieval is done via TLS renegotiation in TLS 1.2.
  • On Server 2022, browsers prefer TLS 1.3. TLS 1.3 does not support the old renegotiation used to fetch a client cert mid‑request. When your app/module at “/” accesses the client cert, IIS attempts renegotiation, fails, and the connection is reset.

Setting Client Certificates to “Ignore” prevents IIS from attempting to renegotiate, so the site loads. This obviously isn’t a solution if you want to use client certificates to authenticate … but we’re authenticating through Ping, so don’t actually need the client certs.

Apache OIDC Authentication to PingFederate (or PingID) Using OIDC

This is kind of a silly update to my attempt to document using mod_auth_openidc in Apache. At the time, I didn’t know who set up the PingFederate side of the connection, so I just used Google as the authentication provider. Five years later, I am one of the people setting up the connections and can finally finish the other side. So here is an update — now using PingFederate as the OIDC/OAUTH provider.

OAUTH Client Setup – Apache

First, make sure mod_auth_openidc is installed

In your Apache config, you can add authentication to the entire site or just specific paths under the site. In this example, we are creating an authenticated sub-directory at /authtest

In the virtual host, I am adding an alias for the protected path as /authtest, configuring the directory, and configuring the location to require valid-user using openid-connect. I am then configuring the OIDC connection.

The OIDCClientID and OIDCClientSecret will be provided to you after the connection is set up in PingID. Just put placeholders in until the real values are known.

The OIDCRedirectURI needed to be a path under the protected directory for me – the Apache module handles the callback. Provide this path on the OIDC connection request.

The OIDCCryptoPassphrase just needs to be a long pseudo-random string. It can include special characters.

# Serve /authtest from local filesystem
Alias /authtest "/var/www/vhtml/sandbox/authtest/"

<Directory "/var/www/vhtml/sandbox/authtest">
Options -Indexes +FollowSymLinks
AllowOverride None
Require all granted
</Directory>

# mod_auth_openidc configuration for Ping (PingFederate/PingID)
# The firewall will need to be configured to allow web server to communicate with this host
OIDCProviderMetadataURL https://login-dev.windstream.com/.well-known/openid-configuration

# The ID and secret will be provided to you
OIDCClientID d5d53555-7525-4555-a565-b525c59545d5
OIDCClientSecret p78…Q2kxB

# Redirect/callback URI – provide this in the request form for the callback URL
OIDCRedirectURI https://www.rushworth.us/authtest/callback

# Session/cookie settings – you make up the OIDCCryptoPassphrase
OIDCCryptoPassphrase "…T9y"
OIDCCookiePath /authtest
OIDCSessionInactivityTimeout 3600
OIDCSessionMaxDuration 28800

# Scopes and client auth
OIDCScope "openid profile email"
OIDCRemoteUserClaim preferred_username
OIDCProviderTokenEndpointAuth client_secret_basic

# If Ping's TLS cert at https://localhost:9031 isn't trusted by the OS CA store,
# install the proper CA chain, or temporarily disable validation (not recommended long-term):
# OIDCSSLValidateServer Off

# Protect the URL path with OIDC
<Location /authtest>
AuthType openid-connect
Require valid-user
OIDCUnAuthAction auth
</Location>

Sample web code for the “protected” page if you want to use the user’s ID. The user’s email is found at $_server[‘OIDC_CLAIM_email’]

[lisa@fedora conf.d]# cat /var/www/vhtml/sandbox/authtest/index.php
<?php
     if( isset($_SERVER['OIDC_CLAIM_iss']) && $_SERVER['OIDC_CLAIM_iss'] == "https://login-dev.windstream.com"){
          echo "I trust you are " . $_SERVER['OIDC_CLAIM_username'] . "\n";
     }

else{
     print "Not authenticated ... \n";
     print "<UL>\n";
     foreach($_SERVER as $key_name => $key_value) {
          print "<LI>" . $key_name . " = " . $key_value . "\n";
     }
     
     print "</UL>\n";
}
?>

Results on the web page – user will be directed to PingID to authenticate, and you will verify that login-dev.windstream.com (or login.windstream.com in production) has authenticated them as the OIDC_CLAIM_username value:

OAUTH Client Setup – PingID

Client auth, add redirect URLs

 

Viewing *Real* Certificate Chain

Browsers implement AIA which “helps” by repairing the certificate chain and forming a trust even without proper server configuration. Which is great for user experience, but causes a lot of challenges to people troubleshooting SSL connection failures from devices, old equipment, etc. It’s fine when I try it from my laptop!

This python script reports on the real certificate chain being served from an endpoint. Self-signed certificates will show as untrusted

And public certs will show the chain and show as trusted

Code:

import ssl
import socket
import datetime
import select

# Third-party modules (install: pip install pyopenssl cryptography)
try:
    from OpenSSL import SSL, crypto
    from cryptography import x509
    from cryptography.hazmat.primitives import hashes
except ImportError as e:
    raise SystemExit(
        "Missing required modules. Please install:\n"
        "  pip install pyopenssl cryptography\n"
        f"Import error: {e}"
    )

def prompt(text, default=None):
    s = input(text).strip()
    return s if s else default

def check_trust(hostname: str, port: int, timeout=6.0):
    """
    Attempt a TLS connection using system trust store and hostname verification.
    Returns (trusted: bool, message: str).
    """
    try:
        ctx = ssl.create_default_context()
        with socket.create_connection((hostname, port), timeout=timeout) as sock:
            with ctx.wrap_socket(sock, server_hostname=hostname) as ssock:
                # Minimal HTTP GET to ensure we fully complete the handshake
                req = f"GET / HTTP/1.1\r\nHost: {hostname}\r\nConnection: close\r\n\r\n"
                ssock.sendall(req.encode("utf-8"))
                _ = ssock.recv(1)
        return True, "TRUSTED (system trust store)"
    except ssl.SSLCertVerificationError as e:
        return False, f"NOT TRUSTED (certificate verification error): {e}"
    except ssl.SSLError as e:
        return False, f"NOT TRUSTED (SSL error): {e}"
    except Exception as e:
        return False, f"Error connecting: {e}"

def _aware_utc(dt: datetime.datetime) -> datetime.datetime:
    """
    Ensure a datetime is timezone-aware in UTC. cryptography returns naive UTC datetimes.
    """
    if dt.tzinfo is None:
        return dt.replace(tzinfo=datetime.timezone.utc)
    return dt.astimezone(datetime.timezone.utc)

def _cert_to_info(cert: x509.Certificate):
    subj = cert.subject.rfc4514_string()
    issr = cert.issuer.rfc4514_string()
    fp_sha1 = cert.fingerprint(hashes.SHA1()).hex().upper()
    nb = _aware_utc(cert.not_valid_before_utc)
    na = _aware_utc(cert.not_valid_after_utc)
    now = datetime.datetime.now(datetime.timezone.utc)
    delta_days = max(0, (na - now).days)
    return {
        "subject": subj,
        "issuer": issr,
        "sha1": fp_sha1,
        "not_before": nb,
        "not_after": na,
        "days_to_expiry": delta_days
    }

def _do_handshake_blocking(conn: SSL.Connection, sock: socket.socket, timeout: float):
    """
    Drive the TLS handshake, handling WantRead/WantWrite by waiting with select.
    """
    deadline = datetime.datetime.now() + datetime.timedelta(seconds=timeout)
    while True:
        try:
            conn.do_handshake()
            return
        except SSL.WantReadError:
            remaining = (deadline - datetime.datetime.now()).total_seconds()
            if remaining <= 0:
                raise TimeoutError("TLS handshake timed out (WantRead)")
            r, _, _ = select.select([sock], [], [], remaining)
            if not r:
                raise TimeoutError("TLS handshake timed out (WantRead)")
            continue
        except SSL.WantWriteError:
            remaining = (deadline - datetime.datetime.now()).total_seconds()
            if remaining <= 0:
                raise TimeoutError("TLS handshake timed out (WantWrite)")
            _, w, _ = select.select([], [sock], [], remaining)
            if not w:
                raise TimeoutError("TLS handshake timed out (WantWrite)")
            continue

def fetch_presented_chain(hostname: str, port: int, timeout: float = 12.0):
    """
    Capture the presented certificate chain using pyOpenSSL.
    Returns (chain: list of {subject, issuer, sha1, not_before, not_after, days_to_expiry}, error: str or None).
    """
    # TCP connect
    try:
        sock = socket.create_connection((hostname, port), timeout=timeout)
        sock.settimeout(timeout)
    except Exception as e:
        return [], f"Error connecting: {e}"

    try:
        # TLS client context
        ctx = SSL.Context(SSL.TLS_CLIENT_METHOD)

        # Compatibility tweaks:
        # - Lower OpenSSL security level
        try:
            ctx.set_cipher_list(b"DEFAULT:@SECLEVEL=1")
        except Exception:
            pass

        # - Disable TLS 1.3 and set minimum TLS 1.2
        try:
            ctx.set_options(SSL.OP_NO_TLSv1_3)
        except Exception:
            pass
        try:
            # Ensure TLSv1.2+ (pyOpenSSL exposes set_min_proto_version on some builds)
            if hasattr(ctx, "set_min_proto_version"):
                ctx.set_min_proto_version(SSL.TLS1_2_VERSION)
        except Exception:
            pass

        # - Set ALPN to http/1.1 (some paths work better when ALPN is present)
        try:
            ctx.set_alpn_protos([b"http/1.1"])
        except Exception:
            pass

        conn = SSL.Connection(ctx, sock)
        conn.set_tlsext_host_name(hostname.encode("utf-8"))
        conn.set_connect_state()

        # Blocking mode (best effort)
        try:
            conn.setblocking(True)
        except Exception:
            pass

        # Drive handshake
        _do_handshake_blocking(conn, sock, timeout=timeout)

        # Retrieve chain (some servers only expose leaf)
        chain = conn.get_peer_cert_chain()
        infos = []
        if chain:
            for c in chain:
                der = crypto.dump_certificate(crypto.FILETYPE_ASN1, c)
                cert = x509.load_der_x509_certificate(der)
                infos.append(_cert_to_info(cert))
        else:
            peer = conn.get_peer_certificate()
            if peer is not None:
                der = crypto.dump_certificate(crypto.FILETYPE_ASN1, peer)
                cert = x509.load_der_x509_certificate(der)
                infos.append(_cert_to_info(cert))

        # Cleanup
        try:
            conn.shutdown()
        except Exception:
            pass
        finally:
            try:
                conn.close()
            except Exception:
                pass
            try:
                sock.close()
            except Exception:
                pass

        if not infos:
            return [], "No certificates captured (server did not present a chain and peer cert unavailable)"
        return infos, None

    except Exception as e:
        try:
            sock.close()
        except Exception:
            pass
        etype = type(e).__name__
        emsg = str(e) or "no message"
        return [], f"TLS handshake or chain retrieval error: {etype}: {emsg}"

def main():
    hostname = prompt("Enter hostname to test (e.g., example.domain.com): ")
    if not hostname:
        print("Hostname is required.")
        return
    port_str = prompt("Enter port [default 443]: ", "443")
    try:
        port = int(port_str)
    except ValueError:
        print("Invalid port.")
        return

    print(f"\nTesting TLS chain for {hostname}:{port} ...")
    chain, err = fetch_presented_chain(hostname, port)
    print("\nPresented chain:")
    if err:
        print(f"  [ERROR] {err}")
    elif not chain:
        print("  [No certificates captured]")
    else:
        for i, ci in enumerate(chain, 1):
            print(f"  [{i}] Subject: {ci['subject']}")
            print(f"       Issuer:  {ci['issuer']}")
            print(f"       SHA1:    {ci['sha1']}")
            nb_val = ci.get("not_before")
            na_val = ci.get("not_after")
            nb_str = nb_val.isoformat() if isinstance(nb_val, datetime.datetime) else str(nb_val)
            na_str = na_val.isoformat() if isinstance(na_val, datetime.datetime) else str(na_val)
            print(f"       Not Before: {nb_str}")
            print(f"       Not After:  {na_str}")
            dte = ci.get("days_to_expiry")
            if dte is not None:
                print(f"       Expires In: {dte} days")

    trusted, msg = check_trust(hostname, port)
    print(f"\nTrust result: {'TRUSTED' if trusted else 'NOT TRUSTED'} - {msg}")

if __name__ == "__main__":
    main()

Blender – Box Trim Not Supported in Dynamic Topology Mode

I hand-sculpted bases for my chess set before realizing that I wanted them to be very identical. So I wanted to “cut” the hand-sculpted base off of the figure and replace it with a programmatic one. Except I kept getting this error using box trim — Not supported in dynamic topology mode.

Switch to a drawing brush and uncheck Dynamic Topology box

Then switch back to box trim and it actually trims

And then, of course, you need to remember to turn dynamic topography back on for the drawing tools to draw.