从零搭建高可用 Kubernetes 集群 说明：之前我们搭建的集群，只有一个 master 节点，当 master 节点宕机的时候，通过 node 节点将无法继续访问，而 master 主要是管理作用，所以整个集群将无法提供服务
1.1 高可用集群架构
在 node 节点和 master 节点之间，需要一个 LoadBalancer 组件
【作用 1】负载
【作用 2】检查 master 节点的状态
对外需要一个统一的 VIP
1.2 高可用集群技术细节
keepalived：配置虚拟 ip，检查节点的状态
haproxy：负载均衡服务【类似于 nginx】
apiserver
controller
manager
scheduler
1.3 高可用集群搭建 我们采用 2 个 master 节点，一个 node 节点来搭建高可用集群
1.3.1 安装步骤 使用二进制包方式搭建 Kubernetes 集群主要分为以下几步：
【环境准备】准备四台虚拟机，并安装操作系统 CentOS 7.x
【系统初始化】对四个刚安装好的操作系统进行初始化操作
【安装 docker、kubectl、kubeadm、kubectl】对四个节点进行安装
【配置高可用 VIP】对 master 节点安装keepalived和haproxy
【部署 master 组件】在 master 节点上安装kube-apiserver、kube-controller-manager、kube-scheduler
【安装网络插件】配置 CNI 网络插件，用于节点之间的连通
【测试集群】通过拉取一个 nginx 进行测试，能否进行外网测试
1.3.2 安装要求 在开始之前，部署 Kubernetes 集群机器需要满足以下几个条件：
一台或多台机器，操作系统 CentOS7.x-86_x64
硬件配置：2GB 或更多 RAM，2 个 CPU 或更多 CPU，硬盘 30GB 或更多【注意】【注意】【注意】【master 需要两核】
可以访问外网，需要拉取镜像，如果服务器不能上网，需要提前下载镜像并导入节点
禁止 swap 分区
1.3.3 准备环境
角色
IP
配置
步骤
k8sLoadBalancer
192.168.60.150
2CPU 1G
init docker kubectl kubeadm kubectl
k8smaster1
192.168.60.151
2CPU 2G
init docker kubectl kubeadm kubectl keepalived haproxy
k8smaster2
192.168.60.152
2CPU 2G
init docker kubectl kubeadm kubectl keepalived haproxy
k8snode1
192.168.60.153
2CPU 2G
init docker kubectl kubeadm kubectl
1.3.4 系统初始化 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 # 关闭防火墙 systemctl stop firewalld # 禁用 firewalld 服务 systemctl disable firewalld # 关闭 selinux # 临时关闭【立即生效】告警，不启用，Permissive，查看使用 getenforce 命令 setenforce 0 # 永久关闭【重启生效】 sed -i 's/SELINUX=enforcing/\SELINUX=disabled/' /etc/selinux/config # 关闭 swap # 临时关闭【立即生效】查看使用 free 命令 swapoff -a # 永久关闭【重启生效】 sed -ri 's/.*swap.*/#&/' /etc/fstab # 在主机名静态查询表中添加 4 台主机 cat >> /etc/hosts << EOF 192.168.60.150 k8sLoadBalancer 192.168.60.151 k8smaster1 192.168.60.152 k8smaster2 192.168.60.153 k8snode1 EOF # 将桥接的 IPv4 流量传递到 iptables 的链【3 个节点上都执行】 cat > /etc/sysctl.d/k8s.conf << EOF net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 EOF # 生效 sysctl --system # 时间同步 yum install ntpdate -y ntpdate time.windows.com # 根据规划设置主机名【k8sLoadBalancer 节点上操作】 hostnamectl set-hostname k8sLoadBalancer # 根据规划设置主机名【k8smaster1 节点上操作】 hostnamectl set-hostname ks8master1 # 根据规划设置主机名【k8smaster2 节点上操作】 hostnamectl set-hostname k8smaster2 # 根据规划设置主机名【k8snode1 节点操作】 hostnamectl set-hostname k8snode1
1.3.5 安装 docker、kubelet、kubeadm、kubectl 所有节点安装 docker、kubelet、kubeadm、kubectl，Kubernetes 默认 CRI（容器运行时）为 docker，因此先安装 docker
1 2 3 4 5 6 7 8 cat >/etc/yum.repos.d/docker.repo<<EOF [docker-ce-edge] name=Docker CE Edge - \$basearch baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/\$basearch/edge enabled=1 gpgcheck=1 gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg EOF
（2）然后 yum 方式安装 docker
1 2 3 4 5 # yum 安装 yum -y install docker-ce # 查看 docker 版本 docker --version
（3）配置 docker 的镜像源【阿里云】
1 2 3 4 5 cat >> /etc/docker/daemon.json << EOF { "registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"] } EOF
（4）然后启动 docker
1 2 3 systemctl start docker systemctl enable docker systemctl status docker
2、安装 kubeadm，kubelet 和 kubectl
1 2 3 4 5 6 7 8 9 cat > /etc/yum.repos.d/kubernetes.repo << EOF [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=0 repo_gpgcheck=0 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF
（2）yum 方式安装，由于版本更新频繁，这里指定版本号部署
1 2 3 4 5 6 7 # 查看版本 yum list kubeadm --showduplicates # 安装 kubelet、kubeadm、kubectl，同时指定版本 yum install -y kubelet-1.18.0 kubeadm-1.18.0 kubectl-1.18.0 # 设置开机启动【这里先不启动】 systemctl enable kubelet
1.3.6 配置高可用 VIP【haproxy+keepalived】 【k8smaster1 + k8smaster2 上操作】
1、安装 haproxy + keepalived我们需要在所有的 master 节点【k8smaster1 和 k8smaster2】上部署 haproxy + keepAlive
1 yum install -y haproxy keepalived
2、配置 haproxy/etc/haproxy/haproxy.cfg
配置中声明了后端代理的两个 master 节点服务器，指定了 haproxy 运行的端口为 16443 等，因此 16443 端口为集群的入口
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 cat > /etc/haproxy/haproxy.cfg << EOF # --------------------------------------------------------------------- # Global settings # --------------------------------------------------------------------- global # to have these messages end up in /var/log/haproxy.log you will # need to: # 1) configure syslog to accept network log events. This is done # by adding the '-r' option to the SYSLOGD_OPTIONS in # /etc/sysconfig/syslog # 2) configure local2 events to go to the /var/log/haproxy.log # file. A line like the following can be added to # /etc/sysconfig/syslog # # local2.* /var/log/haproxy.log # log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon # turn on stats unix socket stats socket /var/lib/haproxy/stats # --------------------------------------------------------------------- # common defaults that all the 'listen' and 'backend' sections will # use if not designated in their block # --------------------------------------------------------------------- defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000 # --------------------------------------------------------------------- # kubernetes apiserver frontend which proxys to the backends # --------------------------------------------------------------------- frontend kubernetes-apiserver mode tcp bind *:16443 option tcplog default_backend kubernetes-apiserver # --------------------------------------------------------------------- # round robin balancing between the various backends # --------------------------------------------------------------------- backend kubernetes-apiserver mode tcp balance roundrobin server k8smaster1 192.168.60.151:6443 check server k8smaster2 192.168.60.152:6443 check # --------------------------------------------------------------------- # collection haproxy statistics message # --------------------------------------------------------------------- listen stats bind *:10080 stats auth admin:awesomePassword stats refresh 5s stats realm HAProxy\ Statistics stats uri /admin?stats EOF
3、配置 keepalived
keepalived中使用track_script机制来配置脚本进行探测kubernetes的master节点是否宕机，并以此切换节点实现高可用
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 cat > /etc/keepalived/keepalived.conf <<EOF ! Configuration File for keepalived global_defs { router_id k8s } vrrp_script check_haproxy { script "killall -0 haproxy" interval 3 weight -2 fall 10 rise 2 } vrrp_instance VI_1 { state MASTER interface ens33 mcast_src_ip 192.168.60.151 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass ceb1b3ec013d66163d6ab } virtual_ipaddress { 192.168.60.150 } track_script { check_haproxy } } EOF
需要注意几点（前两点记得修改）：
mcast_src_ip：配置多播源地址，此地址是当前主机的 ip 地址
priority：keepalived根据此项参数的大小仲裁master节点。我们这里让 master 节点为kubernetes提供服务，其他两个节点暂时为备用节点。因此k8smaster1节点设置为100，k8smaster2节点设置为99
state：我们将k8smaster1节点的state字段设置为MASTER，其他节点字段修改为BACKUP
（2）配置 k8smaster2 节点
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 cat > /etc/keepalived/keepalived.conf <<EOF ! Configuration File for keepalived global_defs { router_id k8s } vrrp_script check_haproxy { script "killall -0 haproxy" interval 3 weight -2 fall 10 rise 2 } vrrp_instance VI_1 { state BACKUP interface ens33 mcast_src_ip 192.168.60.152 virtual_router_id 51 priority 99 advert_int 1 authentication { auth_type PASS auth_pass ceb1b3ec013d66163d6ab } virtual_ipaddress { 192.168.60.150 } track_script { check_haproxy } } EOF
4、启动和检查 【k8smaster1 和 k8smaster2 均要启动】
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 # 启动 haproxy systemctl start haproxy systemctl enable haproxy systemctl status haproxy # 启动 keepalived systemctl start keepalived.service systemctl enable keepalived.service systemctl status keepalived.service # 启动后查看 master 网卡信息 ip a s ens33 # 检查是否可以 ping 通 ping 192.168.60.150 # 如果出错 # 初始化一下！！！并重新启动！！！ systemctl stop firewalld setenforce 0 swapoff -a
1.3.7 部署 Kubernetes Master 组件 【k8smaster1 + k8smaster2 + k8snode1 上操作】
1、k8smaster1 节点
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 # 导出初始化配置文件，然后修改配置，再进行初始化 kubeadm config print init-defaults > kubeadm-init.yaml # 这里直接写入配置，并初始化 cat > kubeadm-init.yaml << EOF apiVersion: kubeadm.k8s.io/v1beta2 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: advertiseAddress: 192.168.60.150 # k8sLoadBalancer ip bindPort: 6443 nodeRegistration: criSocket: /var/run/dockershim.sock name: k8sloadbalancer taints: - effect: NoSchedule key: node-role.kubernetes.io/master --- apiServer: # 添加两行配置 certSANs: - "192.168.60.150" # k8sLoadBalancer ip 即 VIP 的地址 timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta2 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controllerManager: {} dns: type: CoreDNS etcd: local: dataDir: /var/lib/etcd imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers # 阿里云的镜像站点 controlPlaneEndpoint: "192.168.60.150:16443" # VIP 的地址和端口 kind: ClusterConfiguration kubernetesVersion: v1.18.0 networking: dnsDomain: cluster.local serviceSubnet: 10.96.0.0/12 podSubnet: 10.244.0.0/16 # 添加 pod 网段 scheduler: {} EOF # 直接 kubeadm init 初始化，中间会拉取镜像，速度较慢，分为两步来做 # （1）提前拉取镜像 kubeadm config images pull --config kubeadm-init.yaml # （2）初始化 kubeadm init --config kubeadm-init.yaml --upload-certs # Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can now join any number of the control-plane node running the following command on each as root: kubeadm join 192.168.60.150:16443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:68d59df77d9109c44a60a8ca4e7f0932d8cd270c5d0a8adc83c9a1a7d72de73a \ --control-plane --certificate-key b84d54cf9015ef8252e38d68ae96be4b7e41fc9380d8dc2b9ac9ae916b0e9cda Please note that the certificate-key gives access to cluster sensitive data, keep it secret! As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use "kubeadm init phase upload-certs --upload-certs" to reload certs afterward. Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.60.150:16443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:68d59df77d9109c44a60a8ca4e7f0932d8cd270c5d0a8adc83c9a1a7d72de73a
（2）按照提示信息，执行下方命令
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 # 执行下方命令 mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config # 查看节点 kubectl get nodes # 查看 pod kubectl get pods -n kube-system # [root@ks8master1 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8sloadbalancer NotReady master 3m58s v1.18.0 [root@ks8master1 ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-546565776c-skjzz 0/1 Pending 0 3m50s coredns-546565776c-xm8wf 0/1 Pending 0 3m50s etcd-k8sloadbalancer 1/1 Running 0 4m5s kube-apiserver-k8sloadbalancer 1/1 Running 0 4m5s kube-controller-manager-k8sloadbalancer 1/1 Running 0 4m5s kube-proxy-gbjmm 1/1 Running 0 3m48s kube-scheduler-k8sloadbalancer 1/1 Running 0 4m5s
2、k8smaster2 节点
1 2 3 4 5 6 7 8 9 10 # k8smaster2 加入集群 kubeadm join 192.168.60.150:16443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:68d59df77d9109c44a60a8ca4e7f0932d8cd270c5d0a8adc83c9a1a7d72de73a \ --control-plane --certificate-key b84d54cf9015ef8252e38d68ae96be4b7e41fc9380d8dc2b9ac9ae916b0e9cda # 查看集群状态 kubectl get cs # 查看 pod kubectl get pods -n kube-system
3、k8snode1 节点
1 2 kubeadm join 192.168.60.150:16443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:68d59df77d9109c44a60a8ca4e7f0932d8cd270c5d0a8adc83c9a1a7d72de73a
1.3.8 安装集群网络 从官方地址获取到 flannel 的 yaml，在 k8smaster1 上执行
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 # 下载 yaml 文件 wget -c https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml # 安装 flannel 网络 kubectl apply -f kube-flannel.yml # 检查 kubectl get pods -n kube-system # [root@ks8master1 ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-546565776c-skjzz 0/1 Pending 0 21m coredns-546565776c-xm8wf 0/1 Pending 0 21m etcd-k8sloadbalancer 1/1 Running 2 21m etcd-k8smaster2 1/1 Running 0 7m58s kube-apiserver-k8sloadbalancer 1/1 Running 3 21m kube-apiserver-k8smaster2 1/1 Running 1 7m58s kube-controller-manager-k8sloadbalancer 1/1 Running 2 21m kube-controller-manager-k8smaster2 1/1 Running 1 7m58s kube-flannel-ds-cv84g 0/1 Init:1/2 0 91s kube-flannel-ds-j9mbn 0/1 Init:1/2 0 91s kube-flannel-ds-qplqm 0/1 Init:1/2 0 91s kube-proxy-gbjmm 1/1 Running 0 21m kube-proxy-qqdl5 1/1 Running 0 13m kube-proxy-s8bvq 1/1 Running 0 6m27s kube-scheduler-k8sloadbalancer 1/1 Running 2 21m kube-scheduler-k8smaster2 1/1 Running 1 7m58s
1.3.9 测试 kubernetes 集群 在 Kubernetes 集群中创建一个 pod，验证是否正常运行：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 # 创建 nginx deployment kubectl create deployment nginx --image=nginx # 暴露端口 kubectl expose deployment nginx --port=80 --type=NodePort # 查看状态 kubectl get pod,svc # # 浏览器访问： # 192.168.60.151:32594 # 192.168.60.152:32594 # 192.168.60.153:32594 [root@ks8master1 ~]# kubectl get pod,svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 24m service/nginx NodePort 10.109.174.226 <none> 80:32594/TCP 8s
然后我们通过任何一个节点，都能够访问我们的 nginx 页面
