Month: December 2024

Sumo Logic: Validating Collector Data Sources via API

This script is an example of using the Sumo Logic API to retrieve collector details. This particular script looks for Linux servers and validates that each collector has the desired log sources defined. Those that do not contain all desired sources are denoted for farther investigation.

import requests
from requests.auth import HTTPBasicAuth
import pandas as pd
from config import access_id, access_key  # Import your credentials from config.py

# Base URL for Sumo Logic API
base_url = 'https://api.sumologic.com/api/v1'

def get_all_collectors():
    """Retrieve all collectors with pagination support."""
    collectors = []
    limit = 1000  # Adjust as needed; check API docs for max limit
    offset = 0

    while True:
        url = f'{base_url}/collectors?limit={limit}&offset={offset}'
        response = requests.get(url, auth=HTTPBasicAuth(access_id, access_key))
        if response.status_code == 200:
            result = response.json()
            collectors.extend(result.get('collectors', []))
            if len(result.get('collectors', [])) < limit:
                break  # Exit the loop if we received fewer than the limit, meaning it's the last page
            offset += limit
        else:
            print('Error fetching collectors:', response.status_code, response.text)
            break

    return collectors

def get_sources(collector_id):
    """Retrieve sources for a specific collector."""
    url = f'{base_url}/collectors/{collector_id}/sources'
    response = requests.get(url, auth=HTTPBasicAuth(access_id, access_key))
    if response.status_code == 200:
        sources = response.json().get('sources', [])
        # print(f"Log Sources for collector {collector_id}: {sources}")
        return sources
    else:
        print(f'Error fetching sources for collector {collector_id}:', response.status_code, response.text)
        return []

def check_required_logs(sources):
    """Check if the required logs are present in the sources."""
    required_logs = {
        '_security_events': False,
        '_linux_system_events': False,
        'cron_logs': False,
        'dnf_rpm_logs': False
    }

    for source in sources:
        if source['sourceType'] == 'LocalFile':
            name = source.get('name', '')
            for key in required_logs.keys():
                if name.endswith(key):
                    required_logs[key] = True

    # Determine missing logs
    missing_logs = {log: "MISSING" if not present else "" for log, present in required_logs.items()}
    return missing_logs

# Main execution
if __name__ == "__main__":
    collectors = get_all_collectors()
    report_data = []

    for collector in collectors:
        # Check if the collector's osName is 'Linux'
        if collector.get('osName') == 'Linux':
            collector_id = collector['id']
            collector_name = collector['name']
            print(f"Checking Linux Collector: ID: {collector_id}, Name: {collector_name}")

            sources = get_sources(collector_id)
            missing_logs = check_required_logs(sources)
            if any(missing_logs.values()):
                report_entry = {
                    "Collector Name": collector_name,
                    "_security_events": missing_logs['_security_events'],
                    "_linux_system_events": missing_logs['_linux_system_events'],
                    "cron_logs": missing_logs['cron_logs'],
                    "dnf_rpm_logs": missing_logs['dnf_rpm_logs']
                }
                # print(f"Missing logs for collector {collector_name}: {report_entry}")
                report_data.append(report_entry)

    # Create a DataFrame and write to Excel
    df = pd.DataFrame(report_data, columns=[
        "Collector Name", "_security_events", "_linux_system_events", "cron_logs", "dnf_rpm_logs"
    ])

    # Generate the filename with current date and time
    if not df.empty:
        timestamp = pd.Timestamp.now().strftime("%Y%m%d-%H%M")
        output_file = f"{timestamp}-missing_logs_report.xlsx"
        df.to_excel(output_file, index=False)
        print(f"\nData written to {output_file}")
    else:
        print("\nAll collectors have the required logs.")

Fedora 41 – Using DNF to List Installed Packages

We upgraded all of our internal servers to Fedora 41 after a power outage yesterday — had a number of issues to resolve (the liblockdev legacy config reverted so OpenHAB no longer could use USB serial devices, the physical server was swapping 11GB of data even though it had 81GB of memory free, and our Gerbera installation requires some libspdlog.so.1.12 which was updated to version 1.14 with the Fedora upgrade.

The last issue was more challenging to figure out because evidently DNF is now DNF5 and instead of throwing an error like “hey, new version dude! Use the new syntax” when you use an old command to list what is installed … it just says “No matching packages to list”. Like there are no packages installed? Since I’m using bash, openssh, etc … that’s not true.

Luckily, the new syntax works just fine. dnf repoquery –installed

Also:

dnf5 repoquery –available
dnf5 repoquery –userinstalled

Fedora 41 — A LOT of Swapping

With 81 GB available, there’s no good reason to be using 11GB of swap!

2024-12-30 09:11:16 [root@FPP01 /var/named/chroot/etc/]# free -h
total        used        free      shared  buff/cache   available
Mem:           125Gi        44Gi        58Gi       1.8Mi        23Gi        81Gi
Swap:           11Gi        11Gi       121Mi

Memory Stats

2024-12-30 09:11:22 [root@FPP01 /var/named/chroot/etc/]# vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st gu
0  1 12458476 43626964 710656 42012692 1354 1895 51616  8431 16514    6  1  2 88  7  0  3

How to see what is using swap

2024-12-30 09:11:45 [root@FPP01 /var/named/chroot/etc/]# smem -rs swap
PID User     Command                         Swap      USS      PSS      RSS
2903 qemu     /usr/bin/qemu-system-x86_64  4821840  3638976  3643669  3675164
3579 qemu     /usr/bin/qemu-system-x86_64  2282316  6237632  6242508  6275260
3418 qemu     /usr/bin/qemu-system-x86_64  2182844  2063528  2068041  2098292
3331 qemu     /usr/bin/qemu-system-x86_64  1398728  7078176  7082951  7115368
3940 qemu     /usr/bin/qemu-system-x86_64  1020944  4258144  4262757  4294080
3622 qemu     /usr/bin/qemu-system-x86_64   525272  7942284  7947159  7979876
25088 qemu     /usr/bin/qemu-system-x86_64   160456  8298900  8305130  8342252
2563 root     /usr/bin/python3 -Es /usr/s    11696     1332     4050    10872
2174 squid    (squid-1) --kid squid-1 --f     6944     4312     5200    10832
1329 root     /sbin/mount.ntfs-3g /dev/sd     5444    29636    29642    30224
24593 root     /usr/sbin/smbd --foreground     4940    16004    19394    31712
2686 root     /usr/sbin/libvirtd --timeou     4172    28704    30096    37964
2159 root     /usr/sbin/squid --foregroun     3340      152      763     4532
5454 root     /usr/sbin/smbd --foreground     3180      212      496     3552
2134 root     /usr/sbin/smbd --foreground     3008      208      598     4368
2157 root     /usr/sbin/smbd --foreground     2992      136      245     1504
2156 root     /usr/sbin/smbd --foreground     2912      212      304     1648
17963 root     /usr/sbin/smbd --foreground     2880      480     1603     8964
1631 named    /usr/sbin/named -u named -c     2820    60696    60896    63892
1424 polkitd  /usr/lib/polkit-1/polkitd -     2700      704      913     3864
4271 root     /usr/sbin/smbd --foreground     2644     1996     3106     8408
1 root     /usr/lib/systemd/systemd --     2220     4680     5826     9512
2766 root     /usr/sbin/virtlogd              1972      112      873     4548
30736 root     /usr/sbin/smbd --foreground     1864      884     3861    15756
31077 root     /usr/sbin/smbd --foreground     1844     1044     4189    16368
2453 root     /usr/lib/systemd/systemd --     1656      824     1707     4588
1446 root     /usr/sbin/NetworkManager --     1636     4748     5593    10348
1413 dbus     dbus-broker --log 4 --contr     1288      964     1072     2000
21904 root     sshd-session: root@pts/9        1028      644     1287     5412
1402 dbus     /usr/bin/dbus-broker-launch      968      456      571     1872
21900 root     sshd-session: root [priv]        848      488     1911     8588

DNF – Listing Contents of Package Prior to Installation

Voila! Well, install dnf-utils and then

[lisa@linux03 lisa]# repoquery –list gerbera
Last metadata expiration check: 0:00:48 ago on Mon 27 Dec 2024 12:04:05 PM EST.
/etc/gerbera
/etc/gerbera/config.xml
/etc/gerbera/gerbera.db
/etc/gerbera/gerbera.html
/etc/logrotate.d
/etc/logrotate.d/gerbera
/usr/bin/gerbera
/usr/lib/.build-id
/usr/lib/.build-id/8e
/usr/lib/.build-id/8e/cba8f3a7f9db93d01a462f31a8270f1c8ff975
/usr/lib/systemd/system/gerbera.service
/usr/lib/sysusers.d/gerbera.conf
/usr/share/doc/gerbera
/usr/share/doc/gerbera/AUTHORS
/usr/share/doc/gerbera/CONTRIBUTING.md
/usr/share/doc/gerbera/ChangeLog.md
/usr/share/licenses/gerbera
/usr/share/licenses/gerbera/LICENSE.md
/usr/share/man/man1/gerbera.1.gz
/var/log/gerbera

Getting K8s Secrets

A single line command to retrieve the the secrets from a namespace and decode the values:

k8shost:~ # kubectl get secret ca-secret -n mynamespace -o json | jq -r '.data | to_entries[] | "\(.key): \(.value | @base64d)"'
ACCESS_SECRET: X7aB-52p-p2y
API_USER: PM_USER
BASE_URL: https://apiserver.example.com/api/
COMPONENT_ID: 955_18
CPU_MEM_ID: 955_17
INTERFACE_ID: 955_16
INVENTORY_ID: 955_5
RAW_ID: 955_19

Wireshark Capture of Source and Dest Pairs

I was trying to use nethogs with a -t switch to see what is causing the large quantity of traffic that gets bucketed as “unknown TCP”. But the display jumped around a lot – I think because they’re attempting to increment the sums at the top of the “page” rather than just stream information to STDOUT. Figured I could more readily see what I wanted to see using Wireshark. Or, more accurately, tshark.

tshark -i any -f "not port 22" -Y "tcp or udp" -T fields -e ip.src -e tcp.srcport -e udp.srcport -e ip.dst -e tcp.dstport -e udp.dstport | tee /path/tonetcap.cap

Yields:

10.5.5.90 56572 10.5.5.91 3306
10.5.5.90 56572 10.5.5.91 3306
10.5.5.91 3306 10.5.5.90 56572
10.5.5.91 3306 10.5.5.90 56572
10.5.5.90 56572 10.5.5.91 3306
10.5.5.90 56572 10.5.5.91 3306
10.5.5.90 56572 10.5.5.91 3306
10.5.5.75 38552 10.5.5.85 443
10.5.5.75 38552 10.5.5.85 443
10.5.5.75 443 40.97.205.53 12160
10.5.5.75 443 40.97.205.53 12160
10.5.5.75 443 40.97.205.53 12160
10.5.5.75 443 40.97.205.53 12160
10.5.5.75 443 40.97.205.53 12160
10.5.5.75 443 40.97.205.53 12160
10.5.5.61 51389 255.255.255.255 6667
10.5.5.61 51389 255.255.255.255 6667
10.5.5.61 51389 255.255.255.255 6667
10.5.5.61 51389 255.255.255.255 6667
10.5.5.61 51389 255.255.255.255 6667

Listing Unit Files

I usually know what the name of the unit file for a service is … but sometimes you just need to ask what’s there. Or search for one that isn’t showing up with the expected name.

linux1505:~ # systemctl list-unit-files | grep zfs
zfs-import-cache.service                   enabled
zfs-import-scan.service                    disabled
zfs-import.service                         masked
zfs-load-key.service                       masked
zfs-mount.service                          enabled
zfs-scrub@.service                         static
zfs-share.service                          enabled
zfs-volume-wait.service                    enabled
zfs-zed.service                            enabled
zfs-import.target                          enabled
zfs-volumes.target                         disabled
zfs.target                                 enabled
zfs-scrub-monthly@.timer                   disabled
zfs-scrub-weekly@.timer                    disabled

Etomology, fake news, and circus kings

Turkish Journalist Sedef Kabaş said “There is also a saying that is the exact opposite: ‘When cattle go into a palace, they don’t become the king; but the palace becomes a barn’ — and, evidently, was arrested for it. The strange thing is that I heard this as “an old Turkish proverb” … since the quote comes from January 14th 2022, I’m not sure old really applies.

It then got repeated with all sorts of things and places – oxen, stables … and clowns with circuses. When a clown goes into a palace, they don’t become the king; the palace becomes a circus.

It would have made a great old proverb, although it looks like we’ll have to wait a few hundred years.