Month: March 2023

It is not a “witch hunt” if there are really witches

Our local paper had a political cartoon today with a dude exclaiming to his wife:

“Just think — if Donald Trump can be indited for misappropriating funds to pay hush money to a porn star he slept with while his wife was home with their newborn baby … WHAT’S TO PREVENT THAT FROM HAPPENING TO *ME*?”

And the cartoon answer? “Me, dear. Me, and the second amendment.”

But that skips the reality of the potential charge — what’s to stop that from happening to you? Are you running for office and raising money for your campaign? If not … then you CANNOT misappropriate funds (and lie about it) full stop. For the small percentage of Americans for whom the answer is “why, yes I am raising money for my political campaign”? Then the answer is YOU. Make your bribe and extortion payments from YOUR money. Dude isn’t being investigated for making a hush money payment. He’s being investigated for taking money that other people donated to help him get elected, using that for a hush money payment, and falsifying campaign expenditure reporting to hide that the money was used for that purpose.

It’s funny because I encountered something similar early in my career — the company had a lot of rules around spending money. But there were some things you could “expense” — basically use the corporate AmEx to cover & never have to account for. The IT manager had an agreement with a local supplier to submit invoices for items that qualified for expense purchasing — “LAN Cables”, “CD-R Media” — in spite of the fact he was actually picking something else up. I was sent to pick up the handheld radios he ordered, and I couldn’t because the invoice they wanted me to sign was for cat6 cables. I wasn’t trying to make a moral stand (at the time, I didn’t realize there was a moral stand to take) … I just didn’t see how I could submit an expense report with a receipt that didn’t match up with what we were purchasing. The manager explained it to me … and I’m sure he sent someone else to subvert company purchasing policies for him. But he wasn’t the manager for long after the lies were discussed with his manager — a new site manager was brought in & they discovered that the office had purchased tens of thousands of dollars of “fun stuff” — pool table, big screen TV, sofas, alcohol — and outfitted a hidden room in the warehouse. Employees were taking computers, TVs, etc home too. None of those people were fired for buying a TV or computer — they were fired for stealing from their employer and falsifying purchase records.

K8s 1.24.12 Upgrade

Trying to upgrade our dev Kubernetes environment to 1.24.12 … and we encountered what seems to be a fairly common error — unknown service runtime.v1alpha2.RuntimeService

kubeserver:~ # kubeadm init
I0323 13:53:26.492921   55320 version.go:256] remote version is much newer: v1.26.3; falling back to: stable-1.24
[init] Using Kubernetes version: v1.24.12
[preflight] Running pre-flight checks
        [WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists
        [ERROR FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists
        [ERROR FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists
        [ERROR FileAvailable--etc-kubernetes-manifests-etcd.yaml]: /etc/kubernetes/manifests/etcd.yaml already exists
        [ERROR CRI]: container runtime is not running: output: E0323 13:53:26.741684   55340 remote_runtime.go:948] "Status from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
time="2023-03-23T13:53:26-05:00" level=fatal msg="getting status of runtime: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
, error: exit status 1
        [ERROR DirAvailable--var-lib-etcd]: /var/lib/etcd is not empty
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

We found a lot of people online with the same issue who (1) removed the config.toml and tried again, (2) changed the SystemdCGroup setting in the config, or uninstalled and reinstalled some/all of the components until it worked. Unfortunately, removing or modifying the config didn’t help. And removing and reinstalling everything wasn’t particularly appealing. However, we noticed that the same error was reported directly from containerd:

kubeserver:~ # crictl ps
E0323 13:53:07.061777   55228 remote_runtime.go:557] "ListContainers with filter from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService" filter="&ContainerFilter{Id:,State:&ContainerStateValue{State:CONTAINER_RUNNING,},PodSandboxId:,LabelSelector:map[string]string{},}"
FATA[0000] listing containers: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService

Looking at the plugins, there were some in an error state

kubeserver:~ # ctr plugins ls
TYPE                                  ID                       PLATFORMS      STATUS
io.containerd.content.v1              content                  -              ok
io.containerd.snapshotter.v1          aufs                     linux/amd64    skip
io.containerd.snapshotter.v1          btrfs                    linux/amd64    skip
io.containerd.snapshotter.v1          devmapper                linux/amd64    error
io.containerd.snapshotter.v1          native                   linux/amd64    ok
io.containerd.snapshotter.v1          overlayfs                linux/amd64    error
io.containerd.snapshotter.v1          zfs                      linux/amd64    skip

So … it seemed reasonable to look for errors in the messages log from containerd. And, yeah, we had all sorts of errors. Including a rather scary one about reformatting the file system!

Mar 23 13:24:51 kubeserver containerd: time="2023-03-23T13:24:51.726984260-05:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.overlayfs" error="/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs does not support d_type. If the backing filesystem is xfs, please reformat with ftype=1 to enable d_type support"

That would do it — we have a dedicated partition for the k8s stuff … and that volume is formatted the right way — xfs_info confirmed ftype=1

kubeserver:~ # xfs_info /kubernetes/
meta-data=/dev/mapper/kubernetes-kubernetes isize=512    agcount=4, agsize=131071744 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=524286976, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=255999, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

However containerd doesn’t really know anything about this volume, does it? The default location that containerd wants to use isn’t set up to support d_type. Editing /etc/containerd/config.toml, root now tells containerd to use our special partition for ‘stuff’ …

And we were able to run kubeadm init without error. Everything came up as it should have, and our k8s server was upgraded!

Barn Cats First Trip to the Vet

We took the cats we trapped to the vet to get neutered. They were surprisingly chill in the car — not their favorite activity, but they were quiet and looking around as we drove. Dropped them off at a low cost / feral clinic in the morning, and we picked them all up a little after 4PM.

Each kitten has two days of pain killers & need to limit their activity. A neighbor will be taking the other gray kitten (the one we call Pebbles), but Ash and Fritter will be staying inside for a few more weeks so we can take them for a booster shot next month.

They are both recovering well so far — Fritter ended up having an incision in his belly, too, so Ash is recovering quicker. But they’re both happy to curl up together and nap.

2023 Garden — Seed Starting

We got some seeds started this evening — a lot of peppers, tomatoes, and a few other plants. There are a bunch of other peppers to get going, ground cherries, cucumbers, cauliflower, broccoli, and Brussels sprouts. And, once we can start planting outside, corn, beans, peas, and all sorts of stuff for birds.

But here’s what I’ve got planted so far:

Hungarian peppers, Guadillo pepper, Pasilla pepper
Aji Lemon Drop pepper, Peri-Peri pepper, Sugar Rush Peach pepper

Black Strawberry Cherry Tomato, Brandywine Tomato, Trophy Tomato,
Rhubarb, Sugar Baby Bush Watermelon, Mary Washington Asparagus