K8S-环境部署

1 基础环境配置

主机名规划

序号	主机ip	主机名规划
1	10.0.0.12	kubernetes-master.sswang.com kubernetes-master
2	10.0.0.15	kubernetes-node1.sswang.com kubernetes-node1
3	10.0.0.16	kubernetes-node2.sswang.com kubernetes-node2
4	10.0.0.17	kubernetes-node3.sswang.com kubernetes-node3
5	10.0.0.20	kubernetes-register.sswang.com kubernetes-register

跨主机免密码认证

生成秘钥对
ssh-keygen -t rsa 

跨主机免密码认证
ssh-copy-id root@远程主机ip地址

Swap环境配置(所有主机操作)

临时禁用
swapoff -a

永久禁用
sed -i 's/.*swap.*/#&/' /etc/fstab 

内核参数调整
cat >> /etc/sysctl.d/k8s.conf << EOF
vm.swappiness=0
EOF
sysctl -p /etc/sysctl.d/k8s.conf

网络参数调整(所有主机操作)

配置iptables参数，使得流经网桥的流量也经过iptables/netfilter防火墙
cat >> /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

配置生效
modprobe br_netfilter
modprobe overlay
sysctl -p /etc/sysctl.d/k8s.conf

2 容器环境操作

注意：所有主机操作

部署docker软件源

定制软件源
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

安装最新版docker
yum list docker-ce --showduplicates | sort -r
yum install -y docker-ce
systemctl enable docker
systemctl start docker

docker加速器配置

配置加速器文件
]# cat >> /etc/docker/daemon.json <<-EOF
{
  "registry-mirrors": [
    "http://74f21445.m.daocloud.io",
    "https://registry.docker-cn.com",
    "http://hub-mirror.c.163.com",
    "https://docker.mirrors.ustc.edu.cn"
  ], 
  "insecure-registries": ["kubernetes-register.sswang.com"], 
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

重启docker服务
systemctl restart docker

3 cri环境操作

注意：所有主机操作

获取软件

下载软件
mkdir /data/softs && cd /data/softs
wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.2/cri-dockerd-0.3.2.amd64.tgz

解压软件
tar xf cri-dockerd-0.3.2.amd64.tgz
mv cri-dockerd/cri-dockerd /usr/local/bin/

检查效果
cri-dockerd --version

定制配置

定制配置文件
cat > /etc/systemd/system/cri-dockerd.service<<-EOF
[Unit]
Description=CRI Interface for Docker Application Container Engine
Documentation=https://docs.mirantis.com
After=network-online.target firewalld.service docker.service
Wants=network-online.target
[Service]
Type=notify
ExecStart=/usr/local/bin/cri-dockerd --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9
 --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --cri-dockerd-root-directory=/var/lib/dockershim --docker-endpoint=unix:///var/run/docker.sock --cri-dockerd-root-directory=/var/lib/docker
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
StartLimitBurst=3
StartLimitInterval=60s
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
Delegate=yes
KillMode=process
[Install]
WantedBy=multi-user.target
EOF

定制配置
cat > /etc/systemd/system/cri-dockerd.socket <<-EOF
[Unit]
Description=CRI Docker Socket for the API
PartOf=cri-docker.service

[Socket]
ListenStream=/var/run/cri-dockerd.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker

[Install]
WantedBy=sockets.target
EOF

设置服务开机自启动
systemctl daemon-reload
systemctl enable cri-dockerd.service
systemctl restart cri-dockerd.service

4 harbor仓库操作

准备工作

安装docker环境
参考 上一节docker环境部署

安装docker-compose
yum install -y docker-compose

获取软件

下载软件
mkdir /data/{softs,server} -p && cd /data/softs
wget https://github.com/goharbor/harbor/releases/download/v2.5.0/harbor-offline-installer-v2.5.0.tgz

解压软件
tar -zxvf harbor-offline-installer-v2.5.0.tgz -C  /data/server/
cd /data/server/harbor/

加载镜像
docker load < harbor.v2.5.0.tar.gz
docker images

备份配置
cp harbor.yml.tmpl harbor.yml

修改配置

修改配置
[root@kubernetes-register /data/server/harbor]# vim harbor.yml.tmpl
    # 修改主机名
    hostname: kubernetes-register.sswang.com
    http:
      port: 80
    #https:  注释ssl相关的部分
      #  port: 443
      #  certificate: /your/certificate/path
      #  private_key: /your/private/key/path
    # 修改harbor的登录密码
    harbor_admin_password: 123456
    # 设定harbor的数据存储目录
    data_volume: /data/server/harbor/data

配置harbor
./prepare

启动harbor
./install.sh

检查效果
docker-compose ps

定制服务启动文件

定制服务启动文件 /etc/systemd/system/harbor.service
[Unit]
Description=Harbor
After=docker.service systemd-networkd.service systemd-resolved.service
Requires=docker.service
Documentation=http://github.com/vmware/harbor

[Service]
Type=simple
Restart=on-failure
RestartSec=5
#需要注意harbor的安装位置
ExecStart=/usr/bin/docker-compose --file /data/server/harbor/docker-compose.yml up
ExecStop=/usr/bin/docker-compose --file /data/server/harbor/docker-compose.yml down

[Install]
WantedBy=multi-user.target

加载服务配置文件
systemctl daemon-reload
启动服务
systemctl start harbor
检查状态
systemctl status harbor
设置开机自启动
systemctl enable harbor

harbor仓库定制

浏览器访问域名，用户名: admin, 密码：123456
创建sswang用户专用的项目仓库，名称为 sswang，权限为公开的
harbor仓库测试

登录仓库
# docker login kubernetes-register.sswang.com -u sswang
Password:   # 输入登录密码 A12345678a

下载镜像
docker pull busybox

定制镜像标签
docker tag busybox kubernetes-register.sswang.com/sswang/busybox:v0.1

推送镜像
docker push kubernetes-register.sswang.com/sswang/busybox:v0.1

5 k8s集群初始化

软件部署

定制阿里云的关于kubernetes的软件源
]# cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

更新软件源
yum makecache fast

master环境软件部署
yum install kubeadm kubectl kubelet -y

node环境软件部署
yum install kubeadm kubectl kubelet -y

确认基本配置

检查镜像文件列表
kubeadm config images list

获取镜像文件
images=$(kubeadm config images list --kubernetes-version=1.27.3 | awk -F "/" '{print $NF}')
for i in ${images}
do
  docker pull registry.aliyuncs.com/google_containers/$i
  docker tag registry.aliyuncs.com/google_containers/$i kubernetes-register.sswang.com/google_containers/$i
  docker push kubernetes-register.sswang.com/google_containers/$i
  docker rmi registry.aliyuncs.com/google_containers/$i
done

master节点初始化

环境初始化命令
kubeadm init --kubernetes-version=1.27.3 \
--apiserver-advertise-address=10.0.0.12 \
--image-repository kubernetes-register.superopsmsb.com/google_containers \
--service-cidr=10.96.0.0/12 \
--pod-network-cidr=10.244.0.0/16 \
--ignore-preflight-errors=Swap \
--cri-socket=unix:///var/run/cri-dockerd.sock

node节点加入集群

复制join命令，加入到master集群
kubeadm join 10.0.0.12:6443 --token vudfvt.fwpohpbb7yw2qy49 \
        --discovery-token-ca-cert-hash sha256:1... ...48545 --cri-socket=unix:///var/run/cri-dockerd.sock

6 k8s环境收尾操作

权限操作

定制kubernetes的登录权限
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

命令补全

放到master主机的环境文件中
echo "source <(kubectl completion bash)" >> ~/.bashrc
echo "source <(kubeadm completion bash)" >> ~/.bashrc
source ~/.bashrc

网络环境

网络定制
mkdir /data/kubernetes/flannel -p
cd /data/kubernetes/flannel

获取配置文件
wget https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

定制镜像标签
for i in $(grep image kube-flannel.yml | grep -v '#' | awk -F '/' '{print $NF}')
do
    docker pull flannel/$i
    docker tag flannel/$i kubernetes-register.superopsmsb.com/google_containers/$i
    docker push kubernetes-register.superopsmsb.com/google_containers/$i
    docker rmi flannel/$i
done

备份配置文件
cp kube-flannel.yml{,.bak}

修改配置文件
sed -i '/ image:/s/docker.io\/flannel/kubernetes-register.sswang.com\/google_containers/' kube-flannel.yml

应用配置文件
kubectl apply -f kube-flannel.yml

检查效果
kubectl get node