Kubernetes (minikube)

Configure Liveness, Readiness and Startup Probes

Liveness

ポッド(コンテナ)の状態を監視し不意に停止してしまったときに再起動をかけるためのポッドの設定条件

以下 livenessProbe/tmp/healthy を5秒 (periodSeconds) ごとに cat するデモ
initialDelaySecondskubelet が最初に cat コマンドを実行するまでの待ち時間
/tmp/healthy ファイルが作成後30秒後に削除されます)

apiVersion: v1
kind: Pod
metadata:
  labels:
    test: liveness
  name: liveness-exec
spec:
  containers:
  - name: liveness
    image: registry.k8s.io/busybox
    args:
    - /bin/sh
    - -c
    - touch /tmp/healthy; sleep 30; rm -f /tmp/healthy; sleep 600
    livenessProbe:
      exec:
        command:
        - cat
        - /tmp/healthy
      initialDelaySeconds: 5
      periodSeconds: 5

上記ポッドの作成・起動
注) 既存のポッドが起動していて変更した設定を適用する場合は kubectl apply を指定するが、既存のポッドが存在しない場合には $ kubectl apply または $ kubectl create でも可

$ kubectl apply -f https://k8s.io/examples/pods/probe/exec-liveness.yaml
pod/liveness-exec created

30秒後にポッドが再起動していることを確認

$ kubectl describe pod liveness-exec
.....
.....
.....
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  65s                default-scheduler  Successfully assigned default/liveness-exec to minikube
  Normal   Pulling    61s                kubelet            Pulling image "registry.k8s.io/busybox"
  Normal   Pulled     54s                kubelet            Successfully pulled image "registry.k8s.io/busybox" in 6.604873952s
  Normal   Created    51s                kubelet            Created container liveness
  Normal   Started    50s                kubelet            Started container liveness
  Warning  Unhealthy  10s (x3 over 20s)  kubelet            Liveness probe failed: cat: can't open '/tmp/healthy': No such file or directory
  Normal   Killing    10s                kubelet            Container liveness failed liveness probe, will be restarted

ポッドの再起動回数確認

$ kubectl get pod liveness-exec
NAME            READY   STATUS    RESTARTS      AGE
liveness-exec   1/1     Running   3 (34s ago)   4m40s

Volumes


Types of Persistent Volumes

PersistentVolume types are implemented as plugins. Kubernetes currently supports the following plugins:

  • cephfs - CephFS volume
  • csi - Container Storage Interface (CSI)
  • fc - Fibre Channel (FC) storage
  • hostPath - HostPath volume (for single node testing only; WILL NOT WORK in a multi-node cluster; consider using local volume instead)
  • iscsi - iSCSI (SCSI over IP) storage
  • local - local storage devices mounted on nodes.
  • nfs - Network File System (NFS) storage
  • rbd - Rados Block Device (RBD) volume

The following types of PersistentVolume are deprecated. This means that support is still available but will be removed in a future Kubernetes release.


2つのイメージ(コンテナ)で hostPath ボリュームを共有(マウント)するデモ

2つのイメージ(コンテナ)を1つのポッドにデプロイするマニフェスト(設定ファイル)を作成

各コンテナはマウントするボリューム名 host-volume を指定、マウントボリュームの hostPath が各コンテナの mountPath に該当

$ vi app-blue-shared-vol.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: blue-app
  name: blue-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: blue-app
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: blue-app
        type: canary
    spec:
      volumes:
      - name: host-volume
        hostPath:
          path: /home/docker/blue-shared-volume
      containers:
      - image: nginx
        name: nginx
        ports:
        - containerPort: 80
        volumeMounts:
        - mountPath: /usr/share/nginx/html
          name: host-volume
      - image: debian
        name: debian
        volumeMounts:
        - mountPath: /host-vol
          name: host-volume
        command: ["/bin/sh", "-c", "echo Welcome to BLUE App! > /host-vol/index.html ; sleep infinity"]
status: {}

ポッドの起動

$ kubectl apply -f app-blue-shared-vol.yaml
deployment.apps/blue-app created

ノードのポートを開放するサービスの追加

$ kubectl expose deployment blue-app --type=NodePort
service/blue-app exposed

デプロイ、ポッド、サービスの確認

$ kubectl get deploy,po,svc
NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/blue-app   1/1     1            1           5m11s

NAME                           READY   STATUS    RESTARTS   AGE
pod/blue-app-dbcdfc9fd-k2j4q   2/2     Running   0          5m11s

NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
service/blue-app     NodePort    10.110.129.230   <none>        80:30488/TCP   62s
service/kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP        22d

URLの確認(ブラウザで Welcome to BLUE App! が表示されること確認して下さい)

$ minikube service list
|----------------------|------------------------------------|--------------|---------------------------|
|      NAMESPACE       |                NAME                | TARGET PORT  |            URL            |
|----------------------|------------------------------------|--------------|---------------------------|
| default              | blue-app                           |           80 | http://192.168.49.2:30488 |
| default              | kubernetes                         | No node port |

curlコマンドによる確認

$ curl http://192.168.49.2:30488
Welcome to BLUE App!

作成された index.html ファイルの確認

kubernetesのホストマシンのディレクトリが、ポッドの2つのコンテナのディレクトリと共有されているか確認します。

ホストマシン(kubernetes)へのアクセス

minikube ssh でホストのシェルへ移動 —> ls コマンドでファイルを確認 —> シェルから退場

$ minikube ssh
Last login: Fri Sep  9 06:13:09 2022 from 192.168.49.1
docker@minikube:~$ ls /home/docker/blue-shared-volume
index.html
docker@minikube:~$ exit

ポッド内コンテナのシェルへ移動

ポッドの詳細からコンテナ名をチェック

$ kubectl describe pod/blue-app-dbcdfc9fd-k2j4q
.....
.....
Containers:
  nginx:
  .....
  .....
  debian:
  .....
  .....

コンテナ名を指定して各コンテナ( nginx, debian )のシェルへ移動 —> ls コマンドでファイルを確認 —> シェルから退場

$ kubectl exec -ti blue-app-dbcdfc9fd-k2j4q -c nginx -- bash
# ls /usr/share/nginx/html
index.html
# exit

$ kubectl exec -ti blue-app-dbcdfc9fd-k2j4q -c debian -- bash
# ls /host-vol
index.html
# exit

YAML basics in Kubernetes

参考)yamlの構文ルール

https://developer.ibm.com/tutorials/yaml-basics-and-usage-in-kubernetes/

  1. Key Value Pair — The basic type of entry in a YAML file is of a key value pair. After the Key and colon there is a space and then the value.
  2. Arrays/Lists — Lists would have a number of items listed under the name of the list. The elements of the list would start with a -. There can be a n of lists, however the indentation of various elements of the array matters a lot.
  3. Dictionary/Map — A more complex type of YAML file would be a Dictionary and Map.

ConfigMaps

ポッド内のコンテナの環境変数や実行コマンドなどを定義

ConfigMap の作成

$ kubectl create configmap my-config --from-literal=key1=value1 --from-literal=key2=value2
configmap/my-config created

yaml フォーマットで詳細表示

$ kubectl get configmaps my-config -o yaml
apiVersion: v1
data:
  key1: value1
  key2: value2
kind: ConfigMap
metadata:
  creationTimestamp: "2022-09-12T01:40:31Z"
  name: my-config
  namespace: default
  resourceVersion: "719705"
  uid: a085e999-450b-4fdc-bd19-918df195a978

ConfigMap によるコンテナアプリが読み込むファイルの指定(デモ)

ConfigMapでは、アプリが読み込むファイル定義も可能です。

Nginx のコンテナで ConfigMap で定義した index.html ファイルを読み込みます。

index.html

<!DOCTYPE html>
<html>
<head>
<title>Welcome to GREEN App!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
        background-color: GREEN;
    }
</style>
</head>
<body>
<h1 style=\"text-align: center;\">Welcome to GREEN App!</h1>
</body>
</html>

上記ファイルから ConfigMap を作成します。

$ kubectl create configmap green-web-cm --from-file=index.html
configmap/green-web-cm created

ConfigMap(cm) リスト

$ kubectl get cm
NAME               DATA   AGE
green-web-cm       1      40s
kube-root-ca.crt   1      25d

ConfigMap:green-web-cm 詳細確認。Dataセクションに index.html ファイルの内容が反映されています。

$ kubectl describe cm green-web-cm
Name:         green-web-cm
Namespace:    default
Labels:       <none>
Annotations:  <none>

Data
====
index.html:
----
<!DOCTYPE html>
<html>
<head>
<title>Welcome to GREEN App!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
        background-color: GREEN;
    }
</style>
</head>
<body>
<h1 style=\"text-align: center;\">Welcome to GREEN App!</h1>
</body>
</html>


BinaryData
====

Events:  <none>

ConfigMap を指定した Nginxコンテナをデプロイするマニフェストの作成

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: green-web
  name: green-web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: green-web
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: green-web
    spec:
      volumes:
      - name: web-config
        configMap:
          name: green-web-cm
      containers:
      - image: nginx
        name: nginx
        ports:
        - containerPort: 80
        volumeMounts:
        - mountPath: /usr/share/nginx/html
          name: web-config
status: {}

上記マニフェストを指定してポッドを起動

$ kubectl apply -f web-green-with-cm.yaml
deployment.apps/green-web created

外部からアクセスするためノードのポートを開放します。

$ kubectl expose deployment green-web --type=NodePort
service/green-web exposed

Pod, Deploy, Service, Replicaset のリスト一覧

$ kubectl get all
NAME                             READY   STATUS    RESTARTS   AGE
pod/green-web-56f5c8655d-j5hb6   1/1     Running   0          4m36s

NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
service/green-web    NodePort    10.97.55.109   <none>        80:32101/TCP   22s
service/kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP        25d

NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/green-web   1/1     1            1           4m36s

NAME                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/green-web-56f5c8655d   1         1         1       4m36s

URLの確認

$ minikube service list  -n default
|-----------|------------|--------------|---------------------------|
| NAMESPACE |    NAME    | TARGET PORT  |            URL            |
|-----------|------------|--------------|---------------------------|
| default   | green-web  |           80 | http://192.168.49.2:32101 |
| default   | kubernetes | No node port |
|-----------|------------|--------------|---------------------------|

curl コマンドまたはブラウザで上記 URL:http://192.168.49.2:32101 にアクセスして表示内容を確認して下さい。

Secrets

Managing Secrets using kubectl

ユーザとパスワードを別々のファイルで保存(-n オプションで改行なし)

$ echo -n 'admin' > ./username.txt
$ echo -n '1f2d1e2e67df' > ./password.txt

上記ファイルからSecretオブジェクト db-user-pass を作成

$ kubectl create secret generic db-user-pass \
  --from-file=./username.txt \
  --from-file=./password.txt
secret/db-user-pass created

keyを明示する場合 : --from-file=[key=]source

$ kubectl create secret generic db-user-pass \
  --from-file=username=./username.txt \
  --from-file=password=./password.txt

key:value を直接指定する場合: --from-literal=<key>=<value>

$ kubectl create secret generic db-user-pass \
  --from-literal=username=devuser \
  --from-literal=password='S!B\*d$zDsb='

Secretオブジェクトリスト

$ kubectl get secret
NAME            TYPE                                  DATA   AGE
db-user-pass    Opaque                                2      56s

詳細確認

$ kubectl describe secrets/db-user-pass
Name:         db-user-pass
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
password:  12 bytes
username:  7 bytes

エンコードされた username:password を抽出

$ kubectl get secret db-user-pass -o jsonpath='{.data}'
{"password":"UyFCXCpkJHpEc2I9","username":"ZGV2dXNlcg=="}

上記エンコードされた値を入力しデコード

$ echo 'xxxxxxxxxxxxxxxxx' | base64 --decode

または、次の書式でエンコードされた値をデコード(passwordをデコード)

$ kubectl get secret db-user-pass -o jsonpath='{.data.password}' | base64 --decode
S!B\*d$zDsb=

削除

$ kubectl delete secret ds-user-pass

その他Secretオプション


Managing Secrets using Configuration File


Managing Secrets using Kustomize

kubectl edit デフォルトエディターの変更

Linuxでは kubectl edit コマンドのデフォルトエディターは vi ですが、コマンド実行前に環境変数 KUBE_EDITOR を指定することで任意のエディターに変更できます。

nano を指定

$ export KUBE_EDITOR=/usr/bin/nano

vs-code を指定

$ export KUBE_EDITOR='code --wait'

継続使用する場合は .bashrc に上記 export ~ を追加すること。

Ingress

Name-Based Virtual Hosting

Ingress によりホスト名でアクセスするサーバを振分ける

apiVersion: networking.k8s.io/v1 
kind: Ingress
metadata:
  name: virtual-host-ingress
  namespace: default
spec:
  rules:
  - host: blue.example.com
    http:
      paths:
      - backend:
          service:
            name: webserver-blue-svc
            port:
              number: 80
        path: /
        pathType: ImplementationSpecific
  - host: green.example.com
    http:
      paths:
      - backend:
          service:
            name: webserver-green-svc
            port:
              number: 80
        path: /
        pathType: ImplementationSpecific

Fanout

Ingress によりホストのディレクトリでアクセスするサーバを振分ける

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: fan-out-ingress
  namespace: default
spec:
  rules:
  - host: example.com
    http:
      paths:
      - path: /blue
        backend:
          service:
            name: webserver-blue-svc
            port:
              number: 80
        pathType: ImplementationSpecific
      - path: /green
        backend:
          service:
            name: webserver-green-svc
            port:
              number: 80
        pathType: ImplementationSpecific


Kubernetes Ingress Nginx Controller

ingress-nginx is an Ingress controller for Kubernetes using NGINX as a reverse proxy and load balancer.

標準では無効になっているため、minikube のアドオンから有効化する必要があります。

$ minikube addons enable ingress
$ minikube addons list
|-----------------------------|----------|--------------|--------------------------------|
|         ADDON NAME          | PROFILE  |    STATUS    |           MAINTAINER           |
|-----------------------------|----------|--------------|--------------------------------|
| ambassador                  | minikube | disabled     | 3rd party (Ambassador)         |
| auto-pause                  | minikube | disabled     | Google                         |
| csi-hostpath-driver         | minikube | disabled     | Kubernetes                     |
| dashboard                   | minikube | enabled ✅   | Kubernetes                     |
| default-storageclass        | minikube | enabled ✅   | Kubernetes                     |
| efk                         | minikube | disabled     | 3rd party (Elastic)            |
| freshpod                    | minikube | disabled     | Google                         |
| gcp-auth                    | minikube | disabled     | Google                         |
| gvisor                      | minikube | disabled     | Google                         |
| headlamp                    | minikube | disabled     | 3rd party (kinvolk.io)         |
| helm-tiller                 | minikube | disabled     | 3rd party (Helm)               |
| inaccel                     | minikube | disabled     | 3rd party (InAccel             |
|                             |          |              | [[email protected]])            |
| ingress                     | minikube | enabled ✅   | Kubernetes                     |
| ingress-dns                 | minikube | disabled     | Google                         |
| istio                       | minikube | disabled     | 3rd party (Istio)              |
| istio-provisioner           | minikube | disabled     | 3rd party (Istio)              |
| kong                        | minikube | disabled     | 3rd party (Kong HQ)            |
| kubevirt                    | minikube | disabled     | 3rd party (KubeVirt)           |
| logviewer                   | minikube | disabled     | 3rd party (unknown)            |
| metallb                     | minikube | disabled     | 3rd party (MetalLB)            |
| metrics-server              | minikube | enabled ✅   | Kubernetes                     |
| nvidia-driver-installer     | minikube | disabled     | Google                         |
| nvidia-gpu-device-plugin    | minikube | disabled     | 3rd party (Nvidia)             |
| olm                         | minikube | disabled     | 3rd party (Operator Framework) |
| pod-security-policy         | minikube | disabled     | 3rd party (unknown)            |
| portainer                   | minikube | disabled     | 3rd party (Portainer.io)       |
| registry                    | minikube | disabled     | Google                         |
| registry-aliases            | minikube | disabled     | 3rd party (unknown)            |
| registry-creds              | minikube | disabled     | 3rd party (UPMC Enterprises)   |
| storage-provisioner         | minikube | enabled ✅   | Google                         |
| storage-provisioner-gluster | minikube | disabled     | 3rd party (Gluster)            |
| volumesnapshots             | minikube | disabled     | Kubernetes                     |
|-----------------------------|----------|--------------|--------------------------------|

Annotations 注釈

Ex)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: webserver
  annotations:
    description: Deployment based PoC dates 2nd Mar'2022
$ kubectl describe deployment webserver

Name:                webserver
Namespace:           default
CreationTimestamp:   Fri, 25 Mar 2022 05:10:38 +0530
Labels:              app=webserver
Annotations:         deployment.kubernetes.io/revision=1
                     description=Deployment based PoC dates 2nd Mar'2022
... 

Horizontal Pod Autoscaler (HPA)

HPA is an algorithm-based controller API resource which automatically adjusts the number of replicas in a ReplicaSet, Deployment or Replication Controller based on CPU utilization.

Vertical Pod Autoscaler (VPA)

VPA automatically sets Container resource requirements (CPU and memory) in a Pod and dynamically adjusts them in runtime, based on historical utilization data, current resource availability and real-time events.

Cluster Autoscaler

Cluster Autoscaler automatically re-sizes the Kubernetes cluster when there are insufficient resources available for new Pods expecting to be scheduled or when there are underutilized nodes in the cluster.


Resource Quotas リソースの割当て


Jobs

  • parallelism - to set the number of pods allowed to run in parallel;
  • completions - to set the number of expected completions;
  • activeDeadlineSeconds - to set the duration of the Job;
  • backoffLimit - to set the number of retries before Job is marked as failed;
  • ttlSecondsAfterFinished - to delay the clean up of the finished Jobs.

Cronjob

  • startingDeadlineSeconds - to set the deadline to start a Job if scheduled time was missed;
  • concurrencyPolicy - to allow or forbid concurrent Jobs or to replace old Jobs with new ones.

StatefulSets

Like a Deployment, a StatefulSet manages Pods that are based on an identical container spec. Unlike a Deployment, a StatefulSet maintains a sticky identity for each of their Pods. These pods are created from the same spec, but are not interchangeable: each has a persistent identifier that it maintains across any rescheduling.

If you want to use storage volumes to provide persistence for your workload, you can use a StatefulSet as part of the solution. Although individual Pods in a StatefulSet are susceptible to failure, the persistent Pod identifiers make it easier to match existing volumes to the new Pods that replace any that have failed.

Ex)

apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  selector:
    matchLabels:
      app: nginx # has to match .spec.template.metadata.labels
  serviceName: "nginx"
  replicas: 3 # by default is 1
  minReadySeconds: 10 # by default is 0
  template:
    metadata:
      labels:
        app: nginx # has to match .spec.selector.matchLabels
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: nginx
        image: registry.k8s.io/nginx-slim:0.8
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: www
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "my-storage-class"
      resources:
        requests:
          storage: 1Gi

In the above example:

  • A Headless Service, named nginx, is used to control the network domain.
  • The StatefulSet, named web, has a Spec that indicates that 3 replicas of the nginx container will be launched in unique Pods.
  • The volumeClaimTemplates will provide stable storage using PersistentVolumes provisioned by a PersistentVolume Provisioner.

The name of a StatefulSet object must be a valid DNS subdomain name.


Custom Resource Definitions (CRDs)

This is the easiest way to add custom resources and it does not require any programming knowledge. However, building the custom controller would require some programming.

API Aggregation

For more fine-grained control, we can write API Aggregators. They are subordinate API services which sit behind the primary API Server. The primary API Server acts as a proxy for all incoming API requests - it handles the ones based on its capabilities and proxies over the other requests meant for the subordinate API services.

セキュリティとネットワークポリシー

Pod Security Admission

is a built in admission controller for Pod Security that is enabled by default in the API Server.

It can enforce the three Pod Security Standards

Pod Security Standards

at namespace level, by automating the security context restriction to pods when they are deployed.

Profile Description
Privileged Unrestricted policy, providing the widest possible level of permissions. This policy allows for known privilege escalations.
Baseline Minimally restrictive policy which prevents known privilege escalations. Allows the default (minimally specified) Pod configuration.
Restricted Heavily restricted policy, following current Pod hardening best practices.

Network Policies

are sets of rules which define how Pods are allowed to talk to other Pods and resources inside and outside the cluster. Pods not covered by any Network Policy will continue to receive unrestricted traffic from any endpoint.

ログ取得とモニタリング

Metrics Server

is a cluster-wide aggregator of resource usage data - a relatively new feature in Kubernetes.

Prometheus

Managing Resouces

サービス、デプロイなどのリソースのマニフェストを1つのファイルに纏めることも出来ます。
--- で仕切り)

nginx-app.yaml

apiVersion: v1
kind: Service
metadata:
  name: my-nginx-svc
  labels:
    app: nginx
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: nginx
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

上記マニフェストからサービスとデプロイのリソースを同時に作成

$ kubectl apply -f https://k8s.io/examples/application/nginx-app.yaml
service/my-nginx-svc created
deployment.apps/my-nginx created

複数のマニフェストやそれらが格納されているディレクトリを指定して複数のリソースを作成することも出来ます。

$ kubectl apply -f https://k8s.io/examples/application/nginx/nginx-svc.yaml -f https://k8s.io/examples/application/nginx/nginx-deployment.yaml
$ kubectl apply -f https://k8s.io/examples/application/nginx/

What Happens During a Kubernetes Control Plane Failure?

クラスター全体を管理する Control Plane (マスターノード)をインストールしているマシンに不具合が発生した場合、既に稼働しているワーカーノードには直ぐに影響は及びません。ただし、ワーカーノード内でのレプリカ作成やスケジューリングなどを Control Plane (マスターノード)が担っているため事前に以下の対策が必要となります。

Options for Highly Available Topology

Stacked control plane and etcd nodes

kubeadm-ha-topology-stacked-etcd

External etcd nodes

kubeadm-ha-topology-external-etcd

ホストのローカル環境でビルドしたイメージをminikubeへ移行するオプション

Method Supported Runtimes Performance Load Build
docker-env command docker & containerd good yes yes
cache command all ok yes no
podman-env command only cri-o good yes yes
registry addon all ok yes no
minikube ssh all best yes* yes*
ctr/buildctl command only containerd good yes yes
image load command all ok yes no
image build command all ok no yes

上記のImage loadを利用

Podmanでビルドしたイメージをエクスポート

$ podman save localhost/php-fpm-flexisip:20250514 -o php-fpm.tar

minikubeへエクスポート

$ minikube image load php-fpm.tar

確認

$ minikube image ls
registry.k8s.io/pause:3.10
registry.k8s.io/kube-scheduler:v1.33.1
registry.k8s.io/kube-proxy:v1.33.1
registry.k8s.io/kube-controller-manager:v1.33.1
registry.k8s.io/kube-apiserver:v1.33.1
registry.k8s.io/etcd:3.5.21-0
registry.k8s.io/coredns/coredns:v1.12.0
quay.io/jetstack/cert-manager-webhook:v1.17.2
quay.io/jetstack/cert-manager-controller:v1.17.2
quay.io/jetstack/cert-manager-cainjector:v1.17.2
.....
.....
localhost/php-fpm-8.2:20250514
.....
.....

マニフェストではローカルイメージを使用するように指定

imagePullPolicy: Never

containers:
- name: flexisip
  image: localhost/php-fpm:20250514
  imagePullPolicy: Never

ホストマシンに直接minikubeを配備:driver=none

たとえば --driver=docker を使用すると、Minikube VM はDockerコンテナとして実行され、ホストと異なるネットワーク空間に配置されます。そのため、RTP通信などで必要とされる複数ポート(UDPなど)をホスト側で直接バインドすることが難しくなります。これが問題となる場合は、--driver=none--driver=virtualbox などのホスト型ドライバの使用を検討する必要があります。

Conntrackのインストール

conntrack はネットワークコネクションの状態を追跡するためのツールで、Kubernetes のコンポーネント(特に kube-proxy)がネットワークルーティングを正しく管理するために使われます。

$ sudo apt install conntrack

crictlのインストール

crictl は、Kubernetes が containerdCRI-O などのコンテナランタイムとやり取りするための CLI ツールです。

$ VERSION="v1.33.0"
wget https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-$VERSION-linux-amd64.tar.gz
sudo tar zxvf crictl-$VERSION-linux-amd64.tar.gz -C /usr/local/bin

CNI (Container Network Interface) のインストール

$ CNI_VERSION="v1.7.1"
$ curl -L -o cni-plugins.tgz https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-linux-amd64-${CNI_VERSION}.tgz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 53.3M  100 53.3M    0     0  4017k      0  0:00:13  0:00:13 --:--:-- 4274k
$ sudo mkdir -p /opt/cni/bin
$ sudo tar -C /opt/cni/bin -xzf cni-plugins.tgz
$ ls /opt/cni/bin
bandwidth  bridge  dhcp  dummy  firewall  host-device  host-local  ipvlan  LICENSE  loopback  macvlan  portmap  ptp  README.md  sbr  static  tap  tuning  vlan  vrf

コメントアウト: disabled_plugins = [“cri”]

https://serverfault.com/questions/1074008/containerd-1-4-9-unimplemented-desc-unknown-service-runtime-v1alpha2-runtimese

$ sudo nano /etc/containerd/config.toml
.....
.....
# disabled_plugins = ["cri"]
.....
.....

minikube起動

$ sudo minikube start --driver=none --container-runtime=containerd
$ sudo kubectl get pod -A
NAMESPACE     NAME                                   READY   STATUS    RESTARTS   AGE
kube-system   coredns-674b8bbfcf-7x7l4               1/1     Running   0          33s
kube-system   coredns-674b8bbfcf-89vxh               1/1     Running   0          33s
kube-system   etcd-hp-elitedesk                      1/1     Running   0          38s
kube-system   kube-apiserver-hp-elitedesk            1/1     Running   0          38s
kube-system   kube-controller-manager-hp-elitedesk   1/1     Running   0          38s
kube-system   kube-proxy-lkvdl                       1/1     Running   0          33s
kube-system   kube-scheduler-hp-elitedesk            1/1     Running   0          38s
kube-system   storage-provisioner                    1/1     Running   0          37s

Ingressアドオン追加

$ sudo minikube addons enable ingress
💡  ingress is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
    ▪ Using image registry.k8s.io/ingress-nginx/controller:v1.12.2
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.5.3
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.5.3
🔎  Verifying ingress addon...
🌟  The 'ingress' addon is enabled