kubernetes-1.18.8-UOS-龙芯mips64le架构适配
2023-12-13 09:34:15
kubernetes-1.18.8-UOS-龙芯mips64le架构适配
一.适配环境
操作系统：UOS 20
CPU架构：mips64le
服务器厂家：
K8S版本：v1.18.8
docker版本：docker-ce 19.03
二.适配步骤
1. 安装docker
由于UOS之前已与docker做过适配，因此可通过uos官方的软件源，安装docker,官方提供给的版本为docker-ce 19.03,若需要其他版本，需要自行进行源码编译安装，本文档仅提供使用uos官方软件源进行安装：
apt-get install -y docker-ce
注意：以下版本中在安装docker-ce的过程中，发现安装后docker无法正常运行，由uos方工程师确认为内核bug目前已合入最新内核。
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-0RY55WFB-1618380763672)(C:\Users\Lenovo\AppData\Roaming\Typora\typora-user-images\image-20200911174533750.png)]
2.源码编译
2.1 安装其他依赖软件
yum install gcc make -y
yum install rsync jq -y
安装go环境：
wget -c https://golang.google.cn/dl/go1.14.6.linux-amd64.tar.gz /opt/
cd /opt/
tar -C /usr/local -xzf go1.14.6.linux-amd64.tar.gz
echo "export PATH=$PATH:/usr/local/go/bin" >> /etc/profile && source /etc/profile
echo "export GOPATH=/home/go" >> /etc/profile && source /etc/profile # 配置GOPATH
mkdir -p $GOPATH
2.2 下载源码
可根据自己的需要去下载不同版本的源码，本文主要编译的为v1.18.6.
mkdir -p $GOPATH/src/k8s.io
cd $GOPATH/src/k8s.io
git clone https://github.com/kubernetes/kubernetes -b v1.18.8
cd kubernetes
国内受限于网速，大概率会下载失败，可参考其他帖子在码云 上做仓库同步，将github的kubernetes仓库同步到码云，目前已经有其他童鞋做过同步，可直接进行下载。
cd $GOPATH/src/k8s.io
git clone https://gitee.com/mirrors/Kubernetes.git -b v1.18.8
cd Kubernetes
2.3 编译需要的资源
2.3.1 mips64le架构平台镜像
- 查看kube-cross的TAG版本号
root@b529f9ce0ca9:/go/src/k8s.io/kubernetes# cat ./build/build-image/cross/VERSION
v1.13.15-1
- 查看debian_iptables_version版本号
root@b529f9ce0ca9:/go/src/k8s.io/kubernetes# egrep -Rn "debian_iptables_version=" ./
./build/common.sh:98: local debian_iptables_version=v12.1.2
./build/dependencies.yaml:112: match: debian_iptables_version=
- 查看debian_base_version版本号
root@b529f9ce0ca9:/go/src/k8s.io/kubernetes# egrep -Rn "debian_base_version=" ./
./build/common.sh:97: local debian_base_version=v2.1.3
./build/dependencies.yaml:84: match: debian_base_version=
目前还无法从官方下载到这两种tag的镜像，只能通过其他镜像替代，如拉取以下镜像：
docker pull loongnixk8s/debian-iptables-mips64le:v12.1.0
docker pull loongnixk8s/debian-base-mips64le:v2.1.0
docker pull registry.aliyuncs.com/google_containers/kube-cross:v1.13.6-1
docker pull loongnixk8s/pause-mips64le:3.1
docker tag loongnixk8s/pause-mips64le:3.1 k8s.gcr.io/pause-mips64le:3.2
docker tag registry.aliyuncs.com/google_containers/kube-cross:v1.13.6-1 us.gcr.io/k8s-artifacts-prod/build-image/kube-cross:v1.13.15-1
docker tag loongnixk8s/debian-base-mips64le:v2.1.0 k8s.gcr.io/debian-base-mips64le:v2.1.3
docker tag loongnixk8s/debian-iptables-mips64le:v12.1.0 k8s.gcr.io/debian-iptables-mips64le:v12.1.2
docker rmi loongnixk8s/debian-iptables-mips64le:v12.1.0
docker rmi loongnixk8s/debian-base-mips64le:v2.1.0
docker rmi registry.aliyuncs.com/google_containers/kube-cross:v1.13.6-1
docker rmi loongnixk8s/pause-mips64le:3.1
4）修改编译脚本
k8s官方并未对mips64le操作指令集的架构进行适配，所以在编译脚本中也不支持对构建此架构的镜像，需要对以下脚本的内容进行修改。
vim hack/lib/version.sh
if [[ -z ${KUBE_GIT_TREE_STATE-} ]]; then
# Check if the tree is dirty. default to dirty
if git_status=$("${git[@]}" status --porcelain 2>/dev/null) && [[ -z ${git_status} ]]; then
KUBE_GIT_TREE_STATE="clean"
else
KUBE_GIT_TREE_STATE="clean" # dirty修改为clean,否则修改代码后编译出的版本号会dirty标记
fi
fi
另外在vendor/github.com/google/cadvisor/fs/fs.go这个包中，有一个数据类型在mips64le架构中不兼容，需将buf.Dev改为uint64(buf.Dev)
将hack/lib/golang.sh中的KUBE_SUPPORTED_SERVER_PLATFORMS、KUBE_SUPPORTED_NODE_PLATFORMS、KUBE_SUPPORTED_CLIENT_PLATFORMS、KUBE_SUPPORTED_TEST_PLATFORMS中添加上mips64le
- 执行编译命令，
在编译过程中会有一个报错，需要将对应的数据类型强制转换一下
KUBE_BASE_IMAGE_REGISTRY=k8s.gcr.io GOOS=linux GOARCH=mips64le KUBE_BUILD_PLATFORMS=linux/mips64le KUBE_BUILD_CONFORMANCE=n KUBE_BUILD_HYPERKUBE=n make release-images GOFLAGS=-v GOGCFLAGS="-N -l" KUBE_BUILD_PULL_LATEST_IMAGES=false
2.3.2 编译kubelet、kubeadm、kubectl二进制
执行如下编译命令：
docker run --rm -v /home/go/src/k8s.io/Kubernetes:/go/src/k8s.io/kubernetes -it us.gcr.io/k8s-artifacts-prod/build-image/kube-cross:v1.13.15-1 bash
cd /go/src/k8s.io/kubernetes
GOOS=linux GOARCH=mips64le KUBE_BUILD_PLATFORMS=linux/mips64le make all GOFLAGS=-v GOGCFLAGS="-N -l" WHAT=cmd/kubeadm # 再分别编译kubectl和kubelet
2.3.3 使用编译好的镜像
将制作好的镜像上传到指定服务器，导入kube-apiserver.tar镜像，并更新环境上部署的kube-apiserver镜像，其他的操作类似。
# docker load -i kube-apiserver.tar
b1d170ccb364: Loading layer [==================================================>] 162.4MB/162.4MB
3.安装部署
3.1 安装kubectl、kubelet、kubectl
将上述编译好的二进制文件上传到指定服务器，将上述而你简直文件移入到/usr/bin目录下，为kubelet配置启动脚本和配置文件。
[root@k8s-master ~]# cat /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
[root@k8s-master ~]# cat /usr/lib/systemd/system/kubelet.service
[Unit]
Description=kubelet: The Kubernetes Node Agent
Documentation=https://kubernetes.io/docs/
Wants=network-online.target
After=network-online.target
[Service]
ExecStart=/usr/bin/kubelet
Restart=always
StartLimitInterval=0
RestartSec=10
[Install]
WantedBy=multi-user.target
3.2 部署k8s集群
3.2.1 下载相关镜像
需要提前下载好以下镜像：
docker pull gebilaoyao/pause-mips64le:3.1
docker pull gebilaoyao/etcd-mips64le:3.3.11
docker pull gebilaoyao/flannel-mips64le:0.10.0
docker pull gebilaoyao/coredns-mips64le:v1.6.7
docker pull gebilaoyao/kube-scheduler-mips64le:v1.18.8
docker pull gebilaoyao/kube-apiserver-mips64le:v1.18.8
docker pull gebilaoyao/kube-controller-manager-mips64le:v1.18.8
docker pull gebilaoyao/kube-proxy-mips64le:v1.18.8
下载好以后需要将镜像标签修改为指定的版本，可通过kubeadm init的过程中通过查看日志看到具体版本，修改为指定版本后在进行集群部署。
3.2.2 部署
具体部署后期会通过自动化工具进行部署，本次适配过程仍为命令行操作，再次不做具体说明
3.3.3 安装网络插件（flannel）
核心组件安装部署完成后，通过kubectl get pod -A ,会发现coredns这个pod一直处于pending状态，这是因为没有安装网络插件，此时通过kubectl get node查看node也会看到节点处于NoReady状态。可通过下面的yaml文件安装flannel网络插件。
vim flannel-mips64le.yaml
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: psp.flannel.unprivileged
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
privileged: false
volumes:
- configMap
- secret
- emptyDir
- hostPath
allowedHostPaths:
- pathPrefix: "/etc/cni/net.d"
- pathPrefix: "/etc/kube-flannel"
- pathPrefix: "/run/flannel"
readOnlyRootFilesystem: false
# Users and groups
runAsUser:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
fsGroup:
rule: RunAsAny
# Privilege Escalation
allowPrivilegeEscalation: false
defaultAllowPrivilegeEscalation: false
# Capabilities
allowedCapabilities: ['NET_ADMIN']
defaultAddCapabilities: []
requiredDropCapabilities: []
# Host namespaces
hostPID: false
hostIPC: false
hostNetwork: true
hostPorts:
- min: 0
max: 65535
# SELinux
seLinux:
# SELinux is unused in CaaSP
rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: flannel
rules:
- apiGroups: ['extensions']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames: ['psp.flannel.unprivileged']
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- ""
resources:
- nodes
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes/status
verbs:
- patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: flannel
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: flannel
subjects:
- kind: ServiceAccount
name: flannel
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: flannel
namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
name: kube-flannel-cfg
namespace: kube-system
labels:
tier: node
app: flannel
data:
cni-conf.json: |
{
"name": "cbr0",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds-mips64le
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/os
operator: In
values:
- linux
- key: kubernetes.io/arch
operator: In
values:
- mips64le
hostNetwork: true
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: gebilaoyao/flannel-mips64le:v0.10.0
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: gebilaoyao/flannel-mips64le:v0.10.0
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "300m"
memory: "500Mi"
limits:
cpu: "300m"
memory: "500Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
执行kubectl apply -f flannel-mips64le.yaml创建pod，此时通过查看pod发现flannel可以正常运行，但是coredns仍然无法正常运行，通过查看node可以发现，此时node仍处于NoReady状态，可以在systemctl status kubelet的信息中看到有no valid networks found in /etc/cni/net.d的报错，同时报在报无法在/opt/cni/bin下找到合适的插件，可以通过如下方法进行解决：
cd $GOPATH/src
git clone https://github.com/containernetworking/plugins.g
cd containernetworking
./build_linux.sh
cp bin/* /
适配结果
目前已成功将K8S核心组件在基于龙芯CPU（mips64le架构）的UO操作系统的服务器上成功运行，具体情况如下：
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-69zzTnTF-1618380763675)(C:\Users\Lenovo\AppData\Roaming\Typora\typora-user-images\image-20200911184548998.png)]
创建pod，此时通过查看pod发现flannel可以正常运行，但是coredns仍然无法正常运行，通过查看node可以发现，此时node仍处于NoReady状态，可以在systemctl status kubelet的信息中看到有no valid networks found in /etc/cni/net.d的报错，同时报在报无法在/opt/cni/bin`下找到合适的插件，可以通过如下方法进行解决：
cd $GOPATH/src
git clone https://github.com/containernetworking/plugins.g
cd containernetworking
./build_linux.sh
cp bin/* /
适配结果
目前已成功将K8S核心组件在基于龙芯CPU（mips64le架构）的UO操作系统的服务器上成功运行，具体情况如下：
文章来源:https://blog.csdn.net/weixin_42143049/article/details/115697491
本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。 如若内容造成侵权/违法违规/事实不符，请联系我的编程经验分享网邮箱：veading@qq.com进行投诉反馈，一经查实，立即删除！
本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。 如若内容造成侵权/违法违规/事实不符，请联系我的编程经验分享网邮箱：veading@qq.com进行投诉反馈，一经查实，立即删除！