千云物流 - 使用k8s负载均衡openelb

openelb的介绍

具体根据官方文档进行安装官方文档,这里作为测试环境的安装使用.
OpenELB 是一个开源的云原生负载均衡器实现，可以在基于裸金属服务器、边缘以及虚拟化的 Kubernetes 环境中使用 LoadBalancer 类型的 Service 对外暴露服务。OpenELB 项目最初由 KubeSphere 社区 发起，目前已作为 CNCF 沙箱项目 加入 CNCF 基金会，由 OpenELB 开源社区维护与支持。
与MetalLB类似，OpenELB也拥有两种主要工作模式：Layer2模式和BGP模式。OpenELB的BGP模式目前暂不支持IPv6。

• layer2 Mode
  
  

• BGP Mode
  

准备k8s的环境

千云物流测试环境部署使用openelb部署.

所需要的软件&版本	对应依赖软件版本
linux [CentOS]	7.9.2009
kubernetes	v1.22.12
docker [20.10.8]	20.10.8
openelb	kubesphere/openelb:v0.5.1

准备Layer2 Mode配置

• 配置ARP参数
  部署Layer2模式需要把k8s集群中的ipvs配置打开strictARP，

strict ARP configure arp_ignore and arp_announce to avoid answering ARP queries from kube-ipvs0 interface

# 查看kube-proxy中的strictARP配置
$ kubectl get configmap -n kube-system kube-proxy -o yaml | grep strictARP
#strictARP: false# 手动修改strictARP配置为true
$ kubectl edit configmap -n kube-system kube-proxy
configmap/kube-proxy edited# 使用命令直接修改并对比不同
$ kubectl get configmap kube-proxy -n kube-system -o yaml | sed -e "s/strictARP: false/strictARP: true/" | kubectl diff -f - -n kube-system# 确认无误后使用命令直接修改并生效
$ kubectl get configmap kube-proxy -n kube-system -o yaml | sed -e "s/strictARP: false/strictARP: true/" | kubectl apply -f - -n kube-system# 重启kube-proxy确保配置生效
$ kubectl rollout restart ds kube-proxy -n kube-system# 确认配置生效
$ kubectl get configmap -n kube-system kube-proxy -o yaml | grep strictARPstrictARP: true

开启之后k8s集群中的kube-proxy会停止响应kube-ipvs0网卡之外的其他网卡的arp请求，而由MetalLB接手处理。
strict ARP开启之后相当于把将arp_ignore设置为1;并将arp_announce设置为2启用严格的ARP，这个原理和LVS中的DR模式对RS的配置一样，可以参考之前的文章中的解释。

网卡配置

#多个网卡，需要指定master节点IP,一个网卡不需要
# kubectl annotate nodes k8s-master01 layer2.openelb.kubesphere.io/v1alpha1="masterip"

创建EIP

接下来我们需要配置loadbalancerIP所在的网段资源，这里我们创建一个Eip对象来进行定义，后面对IP段的管理也是在这里进行。

• 部署eip

apiVersion: network.kubesphere.io/v1alpha2
kind: Eip
metadata:# Eip 对象的名称。name: layer2-eip
spec:# Eip 对象的地址池address: 10.0.0.122-10.0.0.123# openELB的运行模式，默认为bgpprotocol: layer2# OpenELB 在其上侦听 ARP/NDP 请求的网卡。该字段仅在protocol设置为时有效layer2。interface: ens160# 指定是否禁用 Eip 对象# false表示可以继续分配# true表示不再继续分配disable: false
status:# 指定 Eip 对象中的IP地址是否已用完。occupied: false# 指定 Eip 对象中有多少个 IP 地址已分配给服务。# 直接留空，系统会自动生成usage:# Eip 对象中的 IP 地址总数。poolSize: 2# 指定使用的 IP 地址和使用 IP 地址的服务。服务以Namespace/Service name格式显示（例如，default/test-svc）。# 直接留空，系统会自动生成used:# Eip 对象中的第一个 IP 地址。firstIP: 10.0.0.122# Eip 对象中的最后一个 IP 地址。lastIP: 10.0.0.123ready: true# 指定IP协议栈是否为 IPv4。目前，OpenELB 仅支持 IPv4，其值只能是true.v4: true

• 检查eip状态

 kubectl apply -f openelb/openelb-eip.yaml#部署完成后检查eip的状态kubectl get eip

部署openelb

这里我们还是使用yaml进行部署，官方把所有部署的资源整合到了一个文件中，我们还是老规矩先下载到本地再进行部署

apiVersion: v1
kind: Namespace
metadata:name: openelb-system
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:annotations:controller-gen.kubebuilder.io/version: v0.4.0creationTimestamp: nullname: bgpconfs.network.kubesphere.io
spec:group: network.kubesphere.ionames:kind: BgpConflistKind: BgpConfListplural: bgpconfssingular: bgpconfscope: Clusterversions:- name: v1alpha1schema:openAPIV3Schema:description: BgpConf is the Schema for the bgpconfs APIproperties:apiVersion:description: 'APIVersion defines the versioned schema of this representationof an object. Servers should convert recognized schemas to the latestinternal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'type: stringkind:description: 'Kind is a string value representing the REST resource thisobject represents. Servers may infer this from the endpoint the clientsubmits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'type: stringmetadata:type: objectspec:description: struct for container bgp:config. Configuration parametersrelating to the global BGP router.properties:as:description: original -> bgp:as bgp:as's original type is inet:as-number.Local autonomous system number of the router.  Uses the 32-bit as-numbertype from the model in RFC 6991.format: int32type: integerport:description: original -> gobgp:portformat: int32maximum: 65535minimum: 1type: integerrouterID:description: original -> bgp:router-id bgp:router-id's original typeis inet:ipv4-address. Router id of the router, expressed as an 32-bitvalue, IPv4 address.pattern: ^([0-9]{1,3}\.){3}[0-9]{1,3}$type: stringrequired:- as- port- routerIDtype: objectstatus:description: BgpConfStatus defines the observed state of BgpConftype: objecttype: objectserved: truestorage: false- name: v1alpha2schema:openAPIV3Schema:description: BgpConf is the Schema for the bgpconfs APIproperties:apiVersion:description: 'APIVersion defines the versioned schema of this representationof an object. Servers should convert recognized schemas to the latestinternal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'type: stringkind:description: 'Kind is a string value representing the REST resource thisobject represents. Servers may infer this from the endpoint the clientsubmits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'type: stringmetadata:type: objectspec:description: Configuration parameters relating to the global BGP router.properties:as:format: int32type: integerasPerRack:additionalProperties:format: int32type: integertype: objectfamilies:items:format: int32type: integertype: arraygracefulRestart:properties:deferralTime:format: int32type: integerenabled:type: booleanhelperOnly:type: booleanlocalRestarting:type: booleanlonglivedEnabled:type: booleanmode:type: stringnotificationEnabled:type: booleanpeerRestartTime:format: int32type: integerpeerRestarting:type: booleanrestartTime:format: int32type: integerstaleRoutesTime:format: int32type: integertype: objectlistenAddresses:items:type: stringtype: arraylistenPort:format: int32type: integerpolicy:type: stringrouterId:type: stringuseMultiplePaths:type: booleantype: objectstatus:description: BgpConfStatus defines the observed state of BgpConfproperties:nodesConfStatus:additionalProperties:properties:as:format: int32type: integerrouterId:type: stringtype: objecttype: objecttype: objecttype: objectserved: truestorage: truesubresources:status: {}
status:acceptedNames:kind: ""plural: ""conditions: []storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:annotations:controller-gen.kubebuilder.io/version: v0.4.0creationTimestamp: nullname: bgppeers.network.kubesphere.io
spec:group: network.kubesphere.ionames:kind: BgpPeerlistKind: BgpPeerListplural: bgppeerssingular: bgppeerscope: Clusterversions:- name: v1alpha1schema:openAPIV3Schema:description: BgpPeer is the Schema for the bgppeers APIproperties:apiVersion:description: 'APIVersion defines the versioned schema of this representationof an object. Servers should convert recognized schemas to the latestinternal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'type: stringkind:description: 'Kind is a string value representing the REST resource thisobject represents. Servers may infer this from the endpoint the clientsubmits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'type: stringmetadata:type: objectspec:properties:addPaths:description: original -> bgp:add-paths Parameters relating to theadvertisement and receipt of multiple paths for a single NLRI (add-paths).properties:sendMax:description: original -> bgp:send-max The maximum number of pathsto advertise to neighbors for a single NLRI.type: integertype: objectconfig:description: original -> bgp:neighbor-address original -> bgp: