title: 78.K8S不使用Docker  
CreateTime: 2022-06-24 16:00:00  
UpdateTime: 2024-12-29 15:49:23  
CategoryName: CloudNative  
---

---
title: "78.K8S不使用Docker"
date: 2022-06-24T16:00:00+08:00
draft: false
tags: ["cloudnative"]
categories: ["cloudnative"]
author: "springrain"
---


# 说明 
k8s v1.24.0 版本之后,删除了docker的代码,运行时推荐使用```containerd```.悲伤的是没有```docker build```命令,无法build镜像,需要使用 ```nerdctl``` + ```buildkit```组合实现build功能.  

# 镜像差异
 ```nerdctl``` 不会从本地寻找镜像,例如已经存在 ```test:1.0.0``` 的image,如果dockerfile里 ```from test:1.0.0``` docker可以正常从本地获取镜像,nerdctl会从公网查找镜像,需要把镜像发布到```registry```内网仓库,通过https访问获取.  
 ```crictl images```是k8s的使用的镜像,和```nerdctl images```不是一个,需要单独清理.也可以本地导入镜像  
 ```shell
# docker
docker save -o nginx.tar nginx:1.24.0
docker load -i nginx.tar
# containerd
ctr -n=k8s.io image export  nginx.tar nginx:1.24.0
ctr -n=k8s.io image import  nginx.tar
```

# 安装containerd和配置  
yum安装  
```shell
# 安装需要的软件包, yum-util 提供yum-config-manager功能,另外两个是devicemapper驱动依赖的
yum install -y yum-utils device-mapper-persistent-data lvm2
# 设置 yum 源
# yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum install -y containerd 

```
二进制安装  
```shell
### 安装runc
yum install -y runc

### 下载
wget https://github.com/containerd/containerd/releases/download/v1.7.8/containerd-1.7.8-linux-amd64.tar.gz

### 解压
tar -zxvf containerd-1.7.8-linux-amd64.tar.gz

### copy到 /usr/bin/
cp -rf ./bin/* /usr/bin/

### 生成默认的配置文件
mkdir /etc/containerd
containerd config default > /etc/containerd/config.toml

#注意修改sandbox_image的pause镜像版本最好和K8S的保持一致!!!!

```
镜像加速,修改```/etc/containerd/config.toml```  
```shell
    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = ""

      [plugins."io.containerd.grpc.v1.cri".registry.auths]

      [plugins."io.containerd.grpc.v1.cri".registry.configs]
	    ### 跳过tls验证
        [plugins."io.containerd.grpc.v1.cri".registry.configs."registry.jiagou.com:5000".tls]
          insecure_skip_verify = true

      [plugins."io.containerd.grpc.v1.cri".registry.headers]

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]

        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."registry.jiagou.com:5000"]
          endpoint = ["https://registry.jiagou.com:5000"]
        ### 表示需要配置 mirror 的镜像仓库原镜像仓库,endpoint表示提供 mirror 的镜像加速服务.
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
          endpoint = ["https://xuachqgw.mirror.aliyuncs.com"]

```


编写服务脚本 ```/usr/lib/systemd/system/containerd.service```  
```shell
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target

[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/bin/containerd

Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=infinity
# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target
```
注册服务: ```systemctl enable containerd```  
启动服务: ```systemctl start containerd```

如果断电重启,可能会出现类似问题: https://github.com/containerd/containerd/issues/3347

```shell
find /var/lib/containerd/ -type f -size -5M -name '*.db' |grep -v overlay
##/var/lib/containerd/io.containerd.metadata.v1.bolt/meta.db

### 会丢失镜像记录,需要重新pull镜像!!!!!!
mv /var/lib/containerd/io.containerd.metadata.v1.bolt/meta.db /var/lib/containerd/io.containerd.metadata.v1.bolt/meta.db.bak

### 重启
systemctl restart containerd
```

# 安装buildkit
```shell
mkdir -p /usr/local/buildkit/
cd /usr/local/buildkit/
wget https://github.com/moby/buildkit/releases/download/v0.12.3/buildkit-v0.12.3.linux-amd64.tar.gz

tar -zxvf buildkit-v0.12.3.linux-amd64.tar.gz 

### 创建软连接
ln -s /usr/local/buildkit/bin/buildctl /usr/local/bin/buildctl
```

配置```/etc/buildkit/buildkitd.toml```,非root用户配置文件是```~/.config/buildkit/buildkitd.toml```
```toml
[worker.oci]
  enabled = false

[worker.containerd]
  enabled = true
  # 设置默认命名空间
  #namespace = "default"

   
# optionally mirror configuration can be done by defining it as a registry.
[registry."registry.jiagou.com:5000"]
  ###允许http访问
  http = true
  ###忽略检查https证书,用于自签证书
  insecure=true

```

编写服务脚本 ```/etc/systemd/system/buildkit.service```  
```shell
[Unit]
Description=BuildKit
Documentation=https://github.com/moby/buildkit

[Service]
Type=notify
NotifyAccess=all
ExecStart=/usr/local/buildkit/bin/buildkitd --oci-worker=false --containerd-worker=true

[Install]
WantedBy=multi-user.target
```
注册服务: ```systemctl enable buildkit```  
启动服务: ```systemctl start buildkit```

# 安装nerdctl  

```shell
wget https://github.com/containerd/nerdctl/releases/download/v1.6.2/nerdctl-1.6.2-linux-amd64.tar.gz

tar -zxvf nerdctl-1.6.2-linux-amd64.tar.gz

cp -rf ./nerdctl /usr/local/bin/nerdctl
### 伪装成 docker命令
ln -s /usr/local/bin/nerdctl /usr/local/bin/docker

##使用 docker pull 和 docker push 时如果出现X509异常,可以添加 --insecure-registry 参数,使用自签证书.
## docker --insecure-registry  pull/push
```

# 安装配置registry
```nerdctl``` 默认不解析本地已经存在的镜像,会从公网搜索,所以还是要使用镜像仓库,默认使用```registry```,配置https证书.

## 生成SANs https证书
高版本的Chrome浏览器会要求设置subjectAltName,如果没有设置SAN会报证书错误  
参考openssl配置文件,Linux服务器上通常在```/etc/pki/tls/openssl.cnf```  
新建文件```jiagou.conf```  
```conf
[ req ]
default_bits = 4096
distinguished_name = req_distinguished_name
req_extensions = v3_req

[req_distinguished_name]
countryName = country - CN
stateOrProvinceName = province - henan
localityName = city - zhengzhou
organizationName = company name - jiaogu
commonName = domain name or ip - *.jiagou.com

[ v3_req ]
subjectAltName = @alt_names

[ alt_names ]
DNS.1=*.jiagou.com #可以使用通配符
#IP.1=xxx.xxx.xxx.xxx
```
使用openssl生成证书  
```shell
#1.生成根证书密钥
openssl genrsa -out ca.key 4096   #建议长度为4096,1024长度已经被列为不安全.
#2.生成自签名根证书
openssl req -new -x509 -days 36500 -key ca.key -out ca.crt
#3.生成证书密钥
openssl genrsa -out jiagou.key 4096
#向根证书请求签名一个新的证书,由于用户信任了你的根证书,所以根证书签名的其它证书也会被信任
#4.生成csr 注意要使用sha256算法(推荐是sha256算法,默认算法浏览器会报弱加密算法错误)
openssl req -new -key jiagou.key -out jiagou.csr -config ./jiagou.conf -sha256
#5.使用根证书按照csr给证书签名,生成新证书jiagou.crt
openssl x509 -req -days 36500 -in jiagou.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out jiagou.crt -extfile ./jiagou.conf -extensions v3_req

#6.查看证书信息
openssl x509 -text -in jiagou.crt
```

## 配置registry.yaml
```yaml
    spec:
      containers:
      - name: registry
        image: registry:2.8.1
        ###
        imagePullPolicy: IfNotPresent
        env:
        ###设置时区###
        - name: TZ
          value: Asia/Shanghai  

        ###配置生成的域名证书,文件外部挂载
        - name: REGISTRY_HTTP_TLS_CERTIFICATE
          value: /certs/jiagou.crt                     
        - name: REGISTRY_HTTP_TLS_KEY
          value: /certs/jiagou.key       

```
## 本地解析
修改```/etc/hosts```,增加本地的域名解析
```conf
10.98.239.102 registry.jiagou.com
```


​