{"id":9880,"date":"2023-03-23T15:26:16","date_gmt":"2023-03-23T20:26:16","guid":{"rendered":"https:\/\/www.rushworth.us\/lisa\/?p=9880"},"modified":"2023-03-23T16:28:16","modified_gmt":"2023-03-23T21:28:16","slug":"k8s-1-24-upgrade","status":"publish","type":"post","link":"https:\/\/www.rushworth.us\/lisa\/?p=9880","title":{"rendered":"K8s 1.24.12 Upgrade"},"content":{"rendered":"\n<p>Trying to upgrade our dev Kubernetes environment to 1.24.12 &#8230; and we encountered what seems to be a fairly common error &#8212; unknown service runtime.v1alpha2.RuntimeService<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\nkubeserver:~ # kubeadm init\nI0323 13:53:26.492921   55320 version.go:256] remote version is much newer: v1.26.3; falling back to: stable-1.24\n&#x5B;init] Using Kubernetes version: v1.24.12\n&#x5B;preflight] Running pre-flight checks\n        &#x5B;WARNING Firewalld]: firewalld is active, please ensure ports &#x5B;6443 10250] are open or your cluster may not function correctly\nerror execution phase preflight: &#x5B;preflight] Some fatal errors occurred:\n        &#x5B;ERROR FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: \/etc\/kubernetes\/manifests\/kube-apiserver.yaml already exists\n        &#x5B;ERROR FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: \/etc\/kubernetes\/manifests\/kube-controller-manager.yaml already exists\n        &#x5B;ERROR FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: \/etc\/kubernetes\/manifests\/kube-scheduler.yaml already exists\n        &#x5B;ERROR FileAvailable--etc-kubernetes-manifests-etcd.yaml]: \/etc\/kubernetes\/manifests\/etcd.yaml already exists\n        &#x5B;ERROR CRI]: container runtime is not running: output: E0323 13:53:26.741684   55340 remote_runtime.go:948] &quot;Status from runtime service failed&quot; err=&quot;rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService&quot;\ntime=&quot;2023-03-23T13:53:26-05:00&quot; level=fatal msg=&quot;getting status of runtime: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService&quot;\n, error: exit status 1\n        &#x5B;ERROR DirAvailable--var-lib-etcd]: \/var\/lib\/etcd is not empty\n&#x5B;preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`\nTo see the stack trace of this error execute with --v=5 or higher\n\n<\/pre><\/div>\n\n\n<p>We found a lot of people online with the same issue who (1) removed the config.toml and tried again, (2) <a rel=\"noreferrer noopener\" href=\"https:\/\/github.com\/containerd\/containerd\/issues\/4581#issuecomment-1312083515\" target=\"_blank\">changed the SystemdCGroup setting in the config<\/a>, or uninstalled and reinstalled some\/all of the components until it worked. Unfortunately, removing or modifying the config didn&#8217;t help. And removing and reinstalling everything wasn&#8217;t particularly appealing. However, we noticed that the same error was reported directly from containerd:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\nkubeserver:~ # crictl ps\nE0323 13:53:07.061777   55228 remote_runtime.go:557] &quot;ListContainers with filter from runtime service failed&quot; err=&quot;rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService&quot; filter=&quot;&amp;ContainerFilter{Id:,State:&amp;ContainerStateValue{State:CONTAINER_RUNNING,},PodSandboxId:,LabelSelector:map&#x5B;string]string{},}&quot;\nFATA&#x5B;0000] listing containers: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService\n\n<\/pre><\/div>\n\n\n<p>Looking at the plugins, there <em>were<\/em> some in an error state<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\nkubeserver:~ # ctr plugins ls\nTYPE                                  ID                       PLATFORMS      STATUS\nio.containerd.content.v1              content                  -              ok\nio.containerd.snapshotter.v1          aufs                     linux\/amd64    skip\nio.containerd.snapshotter.v1          btrfs                    linux\/amd64    skip\nio.containerd.snapshotter.v1          devmapper                linux\/amd64    error\nio.containerd.snapshotter.v1          native                   linux\/amd64    ok\nio.containerd.snapshotter.v1          overlayfs                linux\/amd64    error\nio.containerd.snapshotter.v1          zfs                      linux\/amd64    skip\n\n<\/pre><\/div>\n\n\n<p>So &#8230; it seemed reasonable to look for errors in the messages log from containerd. And, yeah, we had all sorts of errors. Including a rather scary one about reformatting the file system!<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\nMar 23 13:24:51 kubeserver containerd: time=&quot;2023-03-23T13:24:51.726984260-05:00&quot; level=warning msg=&quot;failed to load plugin io.containerd.snapshotter.v1.overlayfs&quot; error=&quot;\/var\/lib\/containerd\/io.containerd.snapshotter.v1.overlayfs does not support d_type. If the backing filesystem is xfs, please reformat with ftype=1 to enable d_type support&quot;\n<\/pre><\/div>\n\n\n<p>That would do it &#8212; we have a dedicated partition for the k8s stuff &#8230; and <em>that<\/em> volume is formatted the right way &#8212; xfs_info confirmed ftype=1<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\nkubeserver:~ # xfs_info \/kubernetes\/\nmeta-data=\/dev\/mapper\/kubernetes-kubernetes isize=512\u00a0\u00a0\u00a0 agcount=4, agsize=131071744 blks\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 =\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 sectsz=512\u00a0\u00a0 attr=2, projid32bit=1\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 =\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 crc=1\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 finobt=0 spinodes=0\ndata\u00a0\u00a0\u00a0\u00a0 =\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 bsize=4096\u00a0\u00a0 blocks=524286976, imaxpct=5\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 =\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 sunit=0\u00a0\u00a0\u00a0\u00a0\u00a0 swidth=0 blks\nnaming\u00a0\u00a0 =version 2\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 bsize=4096\u00a0\u00a0 ascii-ci=0 ftype=1\nlog\u00a0\u00a0\u00a0\u00a0\u00a0 =internal\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 bsize=4096\u00a0\u00a0 blocks=255999, version=2\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 =\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 sectsz=512\u00a0\u00a0 sunit=0 blks, lazy-count=1\nrealtime =none\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 extsz=4096\u00a0\u00a0 blocks=0, rtextents=0\n<\/pre><\/div>\n\n\n<p>However containerd doesn&#8217;t really know anything about this volume, does it? The <em>default<\/em> location that containerd wants to use <em>isn&#8217;t<\/em> set up to support d_type. Editing \/etc\/containerd\/config.toml, <a rel=\"noreferrer noopener\" href=\"https:\/\/github.com\/containerd\/containerd\/blob\/main\/docs\/man\/containerd-config.toml.5.md\" data-type=\"URL\" data-id=\"https:\/\/github.com\/containerd\/containerd\/blob\/main\/docs\/man\/containerd-config.toml.5.md\" target=\"_blank\">root<\/a> now tells containerd to <em>use <\/em>our special partition for &#8216;stuff&#8217; &#8230; <\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><a href=\"https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2023\/03\/ContainerDConfigForK8s.png\"><img loading=\"lazy\" decoding=\"async\" width=\"199\" height=\"175\" src=\"https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2023\/03\/ContainerDConfigForK8s.png\" alt=\"\" class=\"wp-image-9881\"\/><\/a><\/figure>\n\n\n\n<p>And we were able to run kubeadm init without error. Everything came up as it should have, and our k8s server was upgraded! <\/p>\n","protected":false},"excerpt":{"rendered":"<p>Trying to upgrade our dev Kubernetes environment to 1.24.12 &#8230; and we encountered what seems to be a fairly common error &#8212; unknown service runtime.v1alpha2.RuntimeService We found a lot of people online with the same issue who (1) removed the config.toml and tried again, (2) changed the SystemdCGroup setting in the config, or uninstalled and &hellip;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1541],"tags":[1823,1542,615],"class_list":["post-9880","post","type-post","status-publish","format-standard","hentry","category-kubernetes","tag-containerd","tag-k8s","tag-kubernetes"],"_links":{"self":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9880","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=9880"}],"version-history":[{"count":2,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9880\/revisions"}],"predecessor-version":[{"id":9886,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9880\/revisions\/9886"}],"wp:attachment":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=9880"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=9880"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=9880"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}