千云物流 - 使用k8s负载均衡openelb

openelb的介绍

具体根据官方文档进行安装官方文档,这里作为测试环境的安装使用.
OpenELB 是一个开源的云原生负载均衡器实现，可以在基于裸金属服务器、边缘以及虚拟化的 Kubernetes 环境中使用 LoadBalancer 类型的 Service 对外暴露服务。OpenELB 项目最初由 KubeSphere 社区发起，目前已作为 CNCF 沙箱项目加入 CNCF 基金会，由 OpenELB 开源社区维护与支持。
与MetalLB类似，OpenELB也拥有两种主要工作模式：Layer2模式和BGP模式。OpenELB的BGP模式目前暂不支持IPv6。

layer2 Mode
BGP Mode

准备k8s的环境

千云物流测试环境部署使用openelb部署.

所需要的软件&版本	对应依赖软件版本
linux [CentOS]	7.9.2009
kubernetes	v1.22.12
docker [20.10.8]	20.10.8
openelb	kubesphere/openelb:v0.5.1

准备Layer2 Mode配置

配置ARP参数
部署Layer2模式需要把k8s集群中的ipvs配置打开strictARP，

strict ARP configure arp_ignore and arp_announce to avoid answering ARP queries from kube-ipvs0 interface

# 查看kube-proxy中的strictARP配置
$ kubectl get configmap -n kube-system kube-proxy -o yaml | grep strictARP
#strictARP: false

# 手动修改strictARP配置为true
$ kubectl edit configmap -n kube-system kube-proxy
configmap/kube-proxy edited

# 使用命令直接修改并对比不同
$ kubectl get configmap kube-proxy -n kube-system -o yaml | sed -e "s/strictARP: false/strictARP: true/" | kubectl diff -f - -n kube-system

# 确认无误后使用命令直接修改并生效
$ kubectl get configmap kube-proxy -n kube-system -o yaml | sed -e "s/strictARP: false/strictARP: true/" | kubectl apply -f - -n kube-system

# 重启kube-proxy确保配置生效
$ kubectl rollout restart ds kube-proxy -n kube-system

# 确认配置生效
$ kubectl get configmap -n kube-system kube-proxy -o yaml | grep strictARP
      strictARP: true

开启之后k8s集群中的kube-proxy会停止响应kube-ipvs0网卡之外的其他网卡的arp请求，而由MetalLB接手处理。
strict ARP开启之后相当于把将arp_ignore设置为1;并将arp_announce设置为2启用严格的ARP，这个原理和LVS中的DR模式对RS的配置一样，可以参考之前的文章中的解释。

网卡配置

在这里插入图片描述

#多个网卡，需要指定master节点IP,一个网卡不需要
# kubectl annotate nodes k8s-master01 layer2.openelb.kubesphere.io/v1alpha1="masterip"

创建EIP

接下来我们需要配置loadbalancerIP所在的网段资源，这里我们创建一个Eip对象来进行定义，后面对IP段的管理也是在这里进行。

部署eip

apiVersion: network.kubesphere.io/v1alpha2
kind: Eip
metadata:
  # Eip 对象的名称。
  name: layer2-eip
spec:
  # Eip 对象的地址池
  address: 10.0.0.122-10.0.0.123
  # openELB的运行模式，默认为bgp
  protocol: layer2
  # OpenELB 在其上侦听 ARP/NDP 请求的网卡。该字段仅在protocol设置为时有效layer2。
  interface: ens160
  # 指定是否禁用 Eip 对象
  # false表示可以继续分配
  # true表示不再继续分配
  disable: false
status:
  # 指定 Eip 对象中的IP地址是否已用完。
  occupied: false
  # 指定 Eip 对象中有多少个 IP 地址已分配给服务。
  # 直接留空，系统会自动生成
  usage:
  # Eip 对象中的 IP 地址总数。
  poolSize: 2
  # 指定使用的 IP 地址和使用 IP 地址的服务。服务以Namespace/Service name格式显示（例如，default/test-svc）。
  # 直接留空，系统会自动生成
  used:
  # Eip 对象中的第一个 IP 地址。
  firstIP: 10.0.0.122
  # Eip 对象中的最后一个 IP 地址。
  lastIP: 10.0.0.123
  ready: true
  # 指定IP协议栈是否为 IPv4。目前，OpenELB 仅支持 IPv4，其值只能是true.
  v4: true

检查eip状态

 kubectl apply -f openelb/openelb-eip.yaml
 #部署完成后检查eip的状态
 kubectl get eip

部署openelb

这里我们还是使用yaml进行部署，官方把所有部署的资源整合到了一个文件中，我们还是老规矩先下载到本地再进行部署

apiVersion: v1
kind: Namespace
metadata:
  name: openelb-system
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.4.0
  creationTimestamp: null
  name: bgpconfs.network.kubesphere.io
spec:
  group: network.kubesphere.io
  names:
    kind: BgpConf
    listKind: BgpConfList
    plural: bgpconfs
    singular: bgpconf
  scope: Cluster
  versions:
    - name: v1alpha1
      schema:
        openAPIV3Schema:
          description: BgpConf is the Schema for the bgpconfs API
          properties:
            apiVersion:
              description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
              type: string
            kind:
              description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
              type: string
            metadata:
              type: object
            spec:
              description: struct for container bgp:config. Configuration parameters
                relating to the global BGP router.
              properties:
                as:
                  description: original -> bgp:as bgp:as's original type is inet:as-number.
                    Local autonomous system number of the router.  Uses the 32-bit as-number
                    type from the model in RFC 6991.
                  format: int32
                  type: integer
                port:
                  description: original -> gobgp:port
                  format: int32
                  maximum: 65535
                  minimum: 1
                  type: integer
                routerID:
                  description: original -> bgp:router-id bgp:router-id's original type
                    is inet:ipv4-address. Router id of the router, expressed as an 32-bit
                    value, IPv4 address.
                  pattern: ^([0-9]{
   1,3}\.){
   3}[0-9]{
   1,3}$
                  type: string
              required:
                - as
                - port
                - routerID
              type: object
            status:
              description: BgpConfStatus defines the observed state of BgpConf
              type: object
          type: object
      served: true
      storage: false
    - name: v1alpha2
      schema:
        openAPIV3Schema:
          description: BgpConf is the Schema for the bgpconfs API
          properties:
            apiVersion:
              description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
              type: string
            kind:
              description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
              type: string
            metadata:
              type: object
            spec:
              description: Configuration parameters relating to the global BGP router.
              properties:
                as:
                  format: int32
                  type: integer
                asPerRack:
                  additionalProperties:
                    format: int32
                    type: integer
                  type: object
                families:
                  items:
                    format: int32
                    type: integer
                  type: array
                gracefulRestart:
                  properties:
                    deferralTime:
                      format: int32
                      type: integer
                    enabled:
                      type: boolean
                    helperOnly:
                      type: boolean
                    localRestarting:
                      type: boolean
                    longlivedEnabled:
                      type: boolean
                    mode:
                      type: string
                    notificationEnabled:
                      type: boolean
                    peerRestartTime:
                      format: int32
                      type: integer
                    peerRestarting:
                      type: boolean
                    restartTime:
                      format: int32
                      type: integer
                    staleRoutesTime:
                      format: int32
                      type: integer
                  type: object
                listenAddresses:
                  items:
                    type: string
                  type: array
                listenPort:
                  format: int32
                  type: integer
                policy:
                  type: string
                routerId:
                  type: string
                useMultiplePaths:
                  type: boolean
              type: object
            status:
              description: BgpConfStatus defines the observed state of BgpConf
              properties:
                nodesConfStatus:
                  additionalProperties:
                    properties:
                      as:
                        format: int32
                        type: integer
                      routerId:
                        type: string
                    type: object
                  type: object
              type: object
          type: object
      served: true
      storage: true
      subresources:
        status: {
   }
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.4.0
  creationTimestamp: null
  name: bgppeers.network.kubesphere.io
spec:
  group: network.kubesphere.io
  names:
    kind: BgpPeer
    listKind: BgpPeerList
    plural: bgppeers
    singular: bgppeer
  scope: Cluster
  versions:
    - name: v1alpha1
      schema:
        openAPIV3Schema:
          description: BgpPeer is the Schema for the bgppeers API
          properties:
            apiVersion:
              description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
              type: string
            kind:
              description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
              type: string
            metadata:
              type: object
            spec:
              properties:
                addPaths:
                  description: original -> bgp:add-paths Parameters relating to the
                    advertisement and receipt of multiple paths for a single NLRI (add-paths).
                  properties:
                    sendMax:
                      description: original -> bgp:send-max The maximum number of paths
                        to advertise to neighbors for a single NLRI.
                      type: integer
                  type: object
                config:
                  description: original -> bgp:neighbor-address original -> bgp: