Category: System Administration

Fedora 41 — A LOT of Swapping

With 81 GB available, there’s no good reason to be using 11GB of swap!

2024-12-30 09:11:16 [root@FPP01 /var/named/chroot/etc/]# free -h
total        used        free      shared  buff/cache   available
Mem:           125Gi        44Gi        58Gi       1.8Mi        23Gi        81Gi
Swap:           11Gi        11Gi       121Mi

Memory Stats

2024-12-30 09:11:22 [root@FPP01 /var/named/chroot/etc/]# vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st gu
0  1 12458476 43626964 710656 42012692 1354 1895 51616  8431 16514    6  1  2 88  7  0  3

How to see what is using swap

2024-12-30 09:11:45 [root@FPP01 /var/named/chroot/etc/]# smem -rs swap
PID User     Command                         Swap      USS      PSS      RSS
2903 qemu     /usr/bin/qemu-system-x86_64  4821840  3638976  3643669  3675164
3579 qemu     /usr/bin/qemu-system-x86_64  2282316  6237632  6242508  6275260
3418 qemu     /usr/bin/qemu-system-x86_64  2182844  2063528  2068041  2098292
3331 qemu     /usr/bin/qemu-system-x86_64  1398728  7078176  7082951  7115368
3940 qemu     /usr/bin/qemu-system-x86_64  1020944  4258144  4262757  4294080
3622 qemu     /usr/bin/qemu-system-x86_64   525272  7942284  7947159  7979876
25088 qemu     /usr/bin/qemu-system-x86_64   160456  8298900  8305130  8342252
2563 root     /usr/bin/python3 -Es /usr/s    11696     1332     4050    10872
2174 squid    (squid-1) --kid squid-1 --f     6944     4312     5200    10832
1329 root     /sbin/mount.ntfs-3g /dev/sd     5444    29636    29642    30224
24593 root     /usr/sbin/smbd --foreground     4940    16004    19394    31712
2686 root     /usr/sbin/libvirtd --timeou     4172    28704    30096    37964
2159 root     /usr/sbin/squid --foregroun     3340      152      763     4532
5454 root     /usr/sbin/smbd --foreground     3180      212      496     3552
2134 root     /usr/sbin/smbd --foreground     3008      208      598     4368
2157 root     /usr/sbin/smbd --foreground     2992      136      245     1504
2156 root     /usr/sbin/smbd --foreground     2912      212      304     1648
17963 root     /usr/sbin/smbd --foreground     2880      480     1603     8964
1631 named    /usr/sbin/named -u named -c     2820    60696    60896    63892
1424 polkitd  /usr/lib/polkit-1/polkitd -     2700      704      913     3864
4271 root     /usr/sbin/smbd --foreground     2644     1996     3106     8408
1 root     /usr/lib/systemd/systemd --     2220     4680     5826     9512
2766 root     /usr/sbin/virtlogd              1972      112      873     4548
30736 root     /usr/sbin/smbd --foreground     1864      884     3861    15756
31077 root     /usr/sbin/smbd --foreground     1844     1044     4189    16368
2453 root     /usr/lib/systemd/systemd --     1656      824     1707     4588
1446 root     /usr/sbin/NetworkManager --     1636     4748     5593    10348
1413 dbus     dbus-broker --log 4 --contr     1288      964     1072     2000
21904 root     sshd-session: root@pts/9        1028      644     1287     5412
1402 dbus     /usr/bin/dbus-broker-launch      968      456      571     1872
21900 root     sshd-session: root [priv]        848      488     1911     8588

DNF – Listing Contents of Package Prior to Installation

Voila! Well, install dnf-utils and then

[lisa@linux03 lisa]# repoquery –list gerbera
Last metadata expiration check: 0:00:48 ago on Mon 27 Dec 2024 12:04:05 PM EST.
/etc/gerbera
/etc/gerbera/config.xml
/etc/gerbera/gerbera.db
/etc/gerbera/gerbera.html
/etc/logrotate.d
/etc/logrotate.d/gerbera
/usr/bin/gerbera
/usr/lib/.build-id
/usr/lib/.build-id/8e
/usr/lib/.build-id/8e/cba8f3a7f9db93d01a462f31a8270f1c8ff975
/usr/lib/systemd/system/gerbera.service
/usr/lib/sysusers.d/gerbera.conf
/usr/share/doc/gerbera
/usr/share/doc/gerbera/AUTHORS
/usr/share/doc/gerbera/CONTRIBUTING.md
/usr/share/doc/gerbera/ChangeLog.md
/usr/share/licenses/gerbera
/usr/share/licenses/gerbera/LICENSE.md
/usr/share/man/man1/gerbera.1.gz
/var/log/gerbera

Getting K8s Secrets

A single line command to retrieve the the secrets from a namespace and decode the values:

k8shost:~ # kubectl get secret ca-secret -n mynamespace -o json | jq -r '.data | to_entries[] | "\(.key): \(.value | @base64d)"'
ACCESS_SECRET: X7aB-52p-p2y
API_USER: PM_USER
BASE_URL: https://apiserver.example.com/api/
COMPONENT_ID: 955_18
CPU_MEM_ID: 955_17
INTERFACE_ID: 955_16
INVENTORY_ID: 955_5
RAW_ID: 955_19

Wireshark Capture of Source and Dest Pairs

I was trying to use nethogs with a -t switch to see what is causing the large quantity of traffic that gets bucketed as “unknown TCP”. But the display jumped around a lot – I think because they’re attempting to increment the sums at the top of the “page” rather than just stream information to STDOUT. Figured I could more readily see what I wanted to see using Wireshark. Or, more accurately, tshark.

tshark -i any -f "not port 22" -Y "tcp or udp" -T fields -e ip.src -e tcp.srcport -e udp.srcport -e ip.dst -e tcp.dstport -e udp.dstport | tee /path/tonetcap.cap

Yields:

10.5.5.90 56572 10.5.5.91 3306
10.5.5.90 56572 10.5.5.91 3306
10.5.5.91 3306 10.5.5.90 56572
10.5.5.91 3306 10.5.5.90 56572
10.5.5.90 56572 10.5.5.91 3306
10.5.5.90 56572 10.5.5.91 3306
10.5.5.90 56572 10.5.5.91 3306
10.5.5.75 38552 10.5.5.85 443
10.5.5.75 38552 10.5.5.85 443
10.5.5.75 443 40.97.205.53 12160
10.5.5.75 443 40.97.205.53 12160
10.5.5.75 443 40.97.205.53 12160
10.5.5.75 443 40.97.205.53 12160
10.5.5.75 443 40.97.205.53 12160
10.5.5.75 443 40.97.205.53 12160
10.5.5.61 51389 255.255.255.255 6667
10.5.5.61 51389 255.255.255.255 6667
10.5.5.61 51389 255.255.255.255 6667
10.5.5.61 51389 255.255.255.255 6667
10.5.5.61 51389 255.255.255.255 6667

Listing Unit Files

I usually know what the name of the unit file for a service is … but sometimes you just need to ask what’s there. Or search for one that isn’t showing up with the expected name.

linux1505:~ # systemctl list-unit-files | grep zfs
zfs-import-cache.service                   enabled
zfs-import-scan.service                    disabled
zfs-import.service                         masked
zfs-load-key.service                       masked
zfs-mount.service                          enabled
zfs-scrub@.service                         static
zfs-share.service                          enabled
zfs-volume-wait.service                    enabled
zfs-zed.service                            enabled
zfs-import.target                          enabled
zfs-volumes.target                         disabled
zfs.target                                 enabled
zfs-scrub-monthly@.timer                   disabled
zfs-scrub-weekly@.timer                    disabled

SSSD LDAP Schema

I lost access to all of my Linux servers at work. And, unlike the normal report where nothing changed but xyz is now failing, I knew exactly what happened. A new access request had been approved about ten minutes previously. Looking at my ID, for some reason adding a new group membership changed account gid number to that new group. Except … that shouldn’t have actually dropped my access. If I needed the group to be my primary ID, I should have been able to use newgrp to switch contexts. Instead, I got prompted for a group password (which, yes, is a thing. No, no one uses it).

The hosts were set up to authenticate to AD using LDAP, and very successfully let me log in (or not, if I mistyped my password). They, however, would only see me as a member of my primary group. Well, today, I finally got a back door with sufficient access to poke around.

Turns out I was right — something was improperly configured so groups were not being read from the directory but rather implied from the gid value. I added the configuration parameter ldap_schema to instruct the server to use member instead of memberUid for memberships. I used rfc2307bis as that’s the value I was familiar with. I expect “AD” could be used as well, but figured we were well beyond AD 2008r2 and didn’t really want to dig farther into the nuanced differences between the two settings.

From https://linux.die.net/man/5/sssd-ldap

ldap_schema (string)

Specifies the Schema Type in use on the target LDAP server. Depending on the selected schema, the default attribute names retrieved from the servers may vary. The way that some attributes are handled may also differ.

Four schema types are currently supported:

  • rfc2307
  • rfc2307bis
  • IPA
  • AD

The main difference between these schema types is how group memberships are recorded in the server. With rfc2307, group members are listed by name in the memberUid attribute. With rfc2307bis and IPA, group members are listed by DN and stored in the member attribute. The AD schema type sets the attributes to correspond with Active Directory 2008r2 values.

 

Sumo Logic: Running Queries via API

This is my base script for using the Sumo Logic API to query logs and analyze data. This particular script finds hosts sending syslog data successfully through our firewall, looks who owns the netblock (they weren’t all internal!), and checks our configuration management database (cmdb) to see if we have a host registered with the destination IP address of the syslog traffic.

import requests
from requests.auth import HTTPBasicAuth
import time
from collections import defaultdict
import cx_Oracle
import pandas as pd
import ipaddress
from datetime import datetime
from ipwhois import IPWhois
from ipwhois.exceptions import IPDefinedError

# Import credentials from a config file
from config import access_id, access_key, oracle_username, oracle_password

# Initialize Oracle Client
cx_Oracle.init_oracle_client(lib_dir=r"C:\Oracle\instantclient_21_15")
oracle_dsn = cx_Oracle.makedsn('cmdb_db.example.com', 1521, service_name='cmdb_db.example.com')

# Function to query Oracle database
def query_oracle_cmdb(strIPAddress):
    with cx_Oracle.connect(user=oracle_username, password=oracle_password, dsn=oracle_dsn) as connection:
        cursor = connection.cursor()
        query = """
            SELECT HOSTNAME, FRIENDLYNAME, STATUS, COLLECTIONTIME, RETIREDBYDISPLAYNAME, 
                    RETIREDDATETIME, SERVERAPPSUPPORTTEAM, SERVERENVIRONMENT
            FROM NBIREPORT.CHERWELL_CMDBDATA_FULL
            WHERE IPADDRESS = :ipaddy
        """
        cursor.execute(query, [strIPAddress])
        result = cursor.fetchone()
        cursor.close()
        return result if result else ("",) * 8

# Function to determine IP ownership
def get_ip_ownership(ip):
    # Define internal IP ranges
    internal_networks = [
        ipaddress.IPv4Network("10.0.0.0/8"),
        ipaddress.IPv4Network("172.16.0.0/12"),
        ipaddress.IPv4Network("192.168.0.0/16")
    ]
    
    # Check if the IP is internal
    ip_obj = ipaddress.IPv4Address(ip)
    if any(ip_obj in network for network in internal_networks):
        return "INTERNAL"
    
    # For external IPs, use ipwhois to get ownership info
    try:
        obj = IPWhois(ip)
        result = obj.lookup_rdap(depth=1)
        ownership = result['network']['name']
    except IPDefinedError:
        ownership = "Reserved IP"
    except Exception as e:
        print(f"Error looking up IP {ip}: {e}")
        ownership = "UNKNOWN"
    
    return ownership

# Base URL for Sumo Logic API
base_url = 'https://api.sumologic.com/api/v1'

# Define the search query
search_query = '''
(dpt=514)
AND _sourcecategory = "observe/perimeter/firewall/logs"
| where !(act = "deny")
| where !(act = "timeout")
| where !(act = "ip-conn")
| where (proto=17 or proto=6)
| count dst, act
'''

# Function to create and manage search jobs
def run_search_job(start_time, end_time):
    search_job_data = {
        'query': search_query,
        'from': start_time,
        'to': end_time,
        'timeZone': 'UTC'
    }

    # Create a search job
    search_job_url = f'{base_url}/search/jobs'
    response = requests.post(
        search_job_url,
        auth=HTTPBasicAuth(access_id, access_key),
        json=search_job_data
    )

    if response.status_code != 202:
        print('Error starting search job:', response.status_code, response.text)
        return None

    # Get the search job ID
    job_id = response.json()['id']
    print('Search Job ID:', job_id)

    # Poll for the search job status
    job_status_url = f'{search_job_url}/{job_id}'
    while True:
        response = requests.get(job_status_url, auth=HTTPBasicAuth(access_id, access_key))
        status = response.json().get('state', None)
        print('Search Job Status:', status)
        if status in ['DONE GATHERING RESULTS', 'CANCELLED', 'FAILED']:
            break
        time.sleep(5)  # Reasonable delay to prevent overwhelming the server

    return job_id if status == 'DONE GATHERING RESULTS' else None

# Function to retrieve results of a search job
def retrieve_results(job_id):
    dst_counts = defaultdict(int)
    results_url = f'{base_url}/search/jobs/{job_id}/messages'
    offset = 0
    limit = 1000

    while True:
        params = {'offset': offset, 'limit': limit}
        try:
            response = requests.get(results_url, auth=HTTPBasicAuth(access_id, access_key), params=params, timeout=30)
            if response.status_code == 200:
                results = response.json()
                messages = results.get('messages', [])
                
                for message in messages:
                    message_map = message['map']
                    dst = message_map.get('dst')
                    if dst:
                        dst_counts[dst] += 1
                
                if len(messages) < limit:
                    break

                offset += limit
            else:
                print('Error retrieving results:', response.status_code, response.text)
                break
        except requests.exceptions.RequestException as e:
            print(f'Error during request: {e}')
            time.sleep(5)
            continue

    return dst_counts

# Main execution
if __name__ == "__main__":
    # Prompt for the start date
    start_date_input = input("Enter the start date (YYYY-MM-DD): ")
    try:
        start_time = datetime.strptime(start_date_input, "%Y-%m-%d").strftime("%Y-%m-%dT00:00:00")
    except ValueError:
        print("Invalid date format. Please enter the date in YYYY-MM-DD format.")
        exit()

    # Use today's date as the end date
    end_time = datetime.now().strftime("%Y-%m-%dT00:00:00")

    # Create a search job
    job_id = run_search_job(start_time, end_time)
    if job_id:
        # Retrieve and process results
        dst_counts = retrieve_results(job_id)

        # Prepare data for Excel
        data_for_excel = []

        print("\nDestination IP Counts and Oracle Data:")
        for dst, count in dst_counts.items():
            oracle_data = query_oracle_cmdb(dst)
            ownership = get_ip_ownership(dst)
            # Use only Oracle data columns
            combined_data = (dst, count, ownership) + oracle_data
            data_for_excel.append(combined_data)
            print(combined_data)

        # Create a DataFrame and write to Excel
        df = pd.DataFrame(data_for_excel, columns=[
            "IP Address", "Occurrence Count", "Ownership",
            "CMDB_Hostname", "CMDB_Friendly Name", "CMDB_Status", "CMDB_Collection Time", 
            "CMDB_Retired By", "CMDB_Retired Date", "CMDB_Support Team", "CMDB_Environment"
        ])

        # Generate the filename with current date and time
        timestamp = datetime.now().strftime("%Y%m%d-%H%M")
        output_file = f"{timestamp}-sumo_oracle_data.xlsx"
        df.to_excel(output_file, index=False)
        print(f"\nData written to {output_file}")
    else:
        print('Search job did not complete successfully.')

AD passwordLastSet Times

I’m doing “stuff” in AD again, and have again come across Microsoft’s wild “nanoseconds elapsed since 1601” reference time. AKA “Windows file time”. In previous experience, I was just looking to calculate deltas (how long since that password was set) so figuring out now, subtracting then, and converting nanoseconds elapsed into something a little less specific (days, for example) was fine. Today, though, I need to display a human readable date and time in Excel. Excel, which has its own peculiar way of storing date time values. Fortunately, I happened across a formula that works

=((C2-116444736000000000)/864000000000)+DATE(1970,1,1)

Voila!

Quick sed For Sanitizing Config Files

When sending configuration files to other people for reference, I like to redact any credential-type information … endpoints that allow you to post data without creds, auth configs, etc. Sometimes I replace the string with REDACTED and sometimes I just drop the line completely.

Make a copy of the config files elsewhere, then run sed


# Retain parameter but replace value with REDACTED
sed -i 's|http_post_url: "https://.*"|post_url: "REDACTED"|' *.yaml

# Remove line from config
sed -i '/authorization: Basic/d' *.yaml