文章目录
- 简介
- 一.条件及环境说明
- 二.需求说明
- 三.实现原理及说明
- 四.详细步骤
- 4.1.规划节点标签
- 4.2.创建三个statefulset和service headless配置
- 4.3.创建service配置
- 五.安装kibana
- 六.调整索引分区
- 七.安装说明
简介
k8s集群中搭建有elasticsearch服务一般都会用到pvc,但是考虑到有些自建k8s环境下,搭建的共享存储可能会存在稳定性及性能问题,所以这次是通过采用节点亲和性和hostpath来实现,目前的operator的基本都是采用共享存储的方法。本文将根据现有环境及不同需求将elasticsearch集群的搭建采用hostpath+亲和性的权重+多个副本分区的方式来实现数据持久化和高可用。
一.条件及环境说明
k8s版本k8s-1.29.4,环境搭建在电信机房,六个worker节点,每个节点有一块非系统盘的ssd盘挂载到/data/路径,不采用ceph或nas之类的共享存储,也未采用operator。
二.需求说明
- 搭建三个节点的elasticsearch集群,并配置三个主分区,2个副本分区。
- 高可用:有一个k8s节点长期故障后或重启pod之后也不影响使用。
- 高性能:读写本地磁盘实现高性能的io,共享存储如果资源及硬件性能不够的话,io将会是性能瓶颈。
- 数据安全:在发生切换,节点故障以后,数据要尽可能的保证完整
- 安装简单、管理维护容易
三.实现原理及说明
四.详细步骤
4.1.规划节点标签
节点名称 | 角色 | 标签 |
---|---|---|
k8s-worker-120-81 | 主 | storage-selector=node-a-master |
k8s-worker-120-82 | 主 | storage-selector=node-b-master |
k8s-worker-120-83 | 主 | storage-selector=node-c-master |
k8s-worker-120-84 | 备 | storage-selector=node-a-slve |
k8s-worker-120-85 | 备 | storage-selector=node-a-slve |
k8s-worker-120-86 | 备 | storage-selector=node-a-slve |
注:这里的角色划分是指:redis由于是一主两从,每个redis的pod分别落在三个主的节点上,当主节点发生故障是,就切换到备节点,例如:名称为autopocket-es-a-0的pod,默认落在标签为storage-selector=node-a-master的节点上,当该节点发生故障是,就会根据权重匹配罗在storage-selector=node-a-slve的节点上。
4.2.创建三个statefulset和service headless配置
1.elasticsearch_a.yaml 节点a的statuflset配置和无头服务配置
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: autopocket-es-a
namespace: zx-app
labels:
app: autopocket-es-a
appname: pcauto-pocket
appgroup: autopocket-es
spec:
serviceName: "autopocket-es-a"
replicas: 1
selector:
matchLabels:
app: autopocket-es-a
appgroup: autopocket-es
template:
metadata:
labels:
app: autopocket-es-a
appgroup: autopocket-es
spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 80
preference:
matchExpressions:
- key: storage-selector
operator: In
values:
- node-a-master
- weight: 20
preference:
matchExpressions:
- key: storage-selector
operator: In
values:
- node-a-slave
containers:
- name: elasticsearch
image: "xxx.cn-shenzhen.cr.aliyuncs.com/public/elasticsearch:7.17.22"
command: ["/bin/sh","-c"]
args:
- mkdir -p /home/elastic/${POD_NAME}/data /home/elastic/${POD_NAME}/logs;chown elasticsearch.elasticsearch -R /home/elastic;su elasticsearch -c "/usr/share/elasticsearch/bin/elasticsearch"
env:
- name: cluster.name
value: autopocket-es
- name: K8S_SERVICE_NAME
value: autopocket-es-a
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: discovery.seed_hosts
value: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-b-0.autopocket-es-b.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-c-0.autopocket-es-c.$(POD_NAMESPACE).svc.cluster.local"
- name: cluster.initial_master_nodes
value: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local"
- name: node.name
value: $(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.local
- name: path.data
value: /home/elastic/$(POD_NAME)/data
- name: path.logs
value: /home/elastic/$(POD_NAME)/logs
- name: xpack.security.enabled
value: "false"
ports:
- name: rest
containerPort: 9200
- name: inter-node
containerPort: 9300
volumeMounts:
- name: elasticsearch-data
mountPath: /home/elastic
restartPolicy: Always
terminationGracePeriodSeconds: 30
volumes:
- name: elasticsearch-data
hostPath:
path: /data/es_data
type: ""
tolerations:
- key: node.kubernetes.io/not-ready
operator: Exists
effect: NoExecute
tolerationSeconds: 3600
- key: node.kubernetes.io/unreachable
operator: Exists
effect: NoExecute
tolerationSeconds: 3600
---
apiVersion: v1
kind: Service
metadata:
name: autopocket-es-a
namespace: zx-app
labels:
appname: pcauto-pocket
app: autopocket-es-a
spec:
ports:
- port: 9200
name: rest
- port: 9300
name: inter-node
clusterIP: None
selector:
app: autopocket-es-a
2.elasticsearch_b.yaml 节点b的statuflset配置和无头服务配置
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: autopocket-es-b
namespace: zx-app
labels:
app: autopocket-es-b
appname: pcauto-pocket
appgroup: autopocket-es
spec:
serviceName: "autopocket-es-b"
replicas: 1
selector:
matchLabels:
app: autopocket-es-b
appgroup: autopocket-es
template:
metadata:
labels:
app: autopocket-es-b
appgroup: autopocket-es
spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 80
preference:
matchExpressions:
- key: storage-selector
operator: In
values:
- node-b-master
- weight: 20
preference:
matchExpressions:
- key: storage-selector
operator: In
values:
- node-b-slave
containers:
- name: elasticsearch
image: "xxx.cn-shenzhen.cr.aliyuncs.com/public/elasticsearch:7.17.22"
command: ["/bin/sh","-c"]
args:
- mkdir -p /home/elastic/${POD_NAME}/data /home/elastic/${POD_NAME}/logs;chown elasticsearch.elasticsearch -R /home/elastic;su elasticsearch -c "/usr/share/elasticsearch/bin/elasticsearch"
env:
- name: cluster.name
value: autopocket-es
- name: K8S_SERVICE_NAME
value: autopocket-es-b
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: discovery.seed_hosts
value: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-b-0.autopocket-es-b.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-c-0.autopocket-es-c.$(POD_NAMESPACE).svc.cluster.local"
- name: cluster.initial_master_nodes
value: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local"
- name: node.name
value: $(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.local
- name: path.data
value: /home/elastic/$(POD_NAME)/data
- name: path.logs
value: /home/elastic/$(POD_NAME)/logs
- name: xpack.security.enabled
value: "false"
ports:
- name: rest
containerPort: 9200
- name: inter-node
containerPort: 9300
volumeMounts:
- name: elasticsearch-data
mountPath: /home/elastic
restartPolicy: Always
terminationGracePeriodSeconds: 30
volumes:
- name: elasticsearch-data
hostPath:
path: /data/es_data
type: ""
tolerations:
- key: node.kubernetes.io/not-ready
operator: Exists
effect: NoExecute
tolerationSeconds: 3600
- key: node.kubernetes.io/unreachable
operator: Exists
effect: NoExecute
tolerationSeconds: 3600
---
apiVersion: v1
kind: Service
metadata:
name: autopocket-es-b
namespace: zx-app
labels:
appname: pcauto-pocket
app: autopocket-es-b
spec:
ports:
- port: 9200
name: rest
- port: 9300
name: inter-node
clusterIP: None
selector:
app: autopocket-es-b
3.elasticsearch_c.yaml 节点c的statuflset配置和无头服务配置
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: autopocket-es-c
namespace: zx-app
labels:
app: autopocket-es-c
appname: pcauto-pocket
appgroup: autopocket-es
spec:
serviceName: "autopocket-es-c"
replicas: 1
selector:
matchLabels:
app: autopocket-es-c
appgroup: autopocket-es
template:
metadata:
labels:
app: autopocket-es-c
appgroup: autopocket-es
spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 80
preference:
matchExpressions:
- key: storage-selector
operator: In
values:
- node-c-master
- weight: 20
preference:
matchExpressions:
- key: storage-selector
operator: In
values:
- node-c-slave
containers:
- name: elasticsearch
image: "xxx.cn-shenzhen.cr.aliyuncs.com/public/elasticsearch:7.17.22"
command: ["/bin/sh","-c"]
args:
- mkdir -p /home/elastic/${POD_NAME}/data /home/elastic/${POD_NAME}/logs;chown elasticsearch.elasticsearch -R /home/elastic;su elasticsearch -c "/usr/share/elasticsearch/bin/elasticsearch"
env:
- name: cluster.name
value: autopocket-es
- name: K8S_SERVICE_NAME
value: autopocket-es-c
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: discovery.seed_hosts
value: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-b-0.autopocket-es-b.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-c-0.autopocket-es-c.$(POD_NAMESPACE).svc.cluster.local"
- name: cluster.initial_master_nodes
value: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local"
- name: node.name
value: $(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.local
- name: path.data
value: /home/elastic/$(POD_NAME)/data
- name: path.logs
value: /home/elastic/$(POD_NAME)/logs
- name: xpack.security.enabled
value: "false"
ports:
- name: rest
containerPort: 9200
- name: inter-node
containerPort: 9300
volumeMounts:
- name: elasticsearch-data
mountPath: /home/elastic
restartPolicy: Always
terminationGracePeriodSeconds: 30
volumes:
- name: elasticsearch-data
hostPath:
path: /data/es_data
type: ""
tolerations:
- key: node.kubernetes.io/not-ready
operator: Exists
effect: NoExecute
tolerationSeconds: 3600
- key: node.kubernetes.io/unreachable
operator: Exists
effect: NoExecute
tolerationSeconds: 3600
---
apiVersion: v1
kind: Service
metadata:
name: autopocket-es-c
namespace: zx-app
labels:
appname: pcauto-pocket
app: autopocket-es-c
spec:
ports:
- port: 9200
name: rest
- port: 9300
name: inter-node
clusterIP: None
selector:
app: autopocket-es-c
4.3.创建service配置
给9200 9300配置一个service的代理服务,注意其中的selector是指向到appgroup,这是将三个statefulset定义了一个appgroup的标签。
apiVersion: v1
kind: Service
metadata:
name: autopocket-es
namespace: zx-app
labels:
app: autopocket-es
appname: pcauto-zx
spec:
ports:
- port: 9200
name: rest
- port: 9300
name: inter-node
selector:
appgroup: autopocket-es
type: LoadBalancer
五.安装kibana
apiVersion: apps/v1
kind: Deployment
metadata:
name: autopocket-es-kibana
namespace: zx-app
labels:
app: autopocket-es-kibana
appname: pcauto-pocket
spec:
replicas: 1
selector:
matchLabels:
app: autopocket-es-kibana
template:
metadata:
labels:
app: autopocket-es-kibana
spec:
containers:
- name: kibana
image: "xxx.cn-shenzhen.cr.aliyuncs.com/public/kibana:7.17.22"
ports:
- containerPort: 5601
env:
- name: ELASTICSEARCH_HOSTS
value: '["http://autopocket-es-a-0.autopocket-es-a:9200","http://autopocket-es-b-0.autopocket-es-b:9200","http://autopocket-es-c-0.autopocket-es-c:9200"]'
---
apiVersion: v1
kind: Service
metadata:
name: autopocket-es-kibana
namespace: zx-app
labels:
app: autopocket-es-kibana
appname: pcauto-pocket
spec:
selector:
app: autopocket-es-kibana
ports:
- port: 5601
targetPort: 5601
type: LoadBalancer
查看集群健康情况:
http://10.16.xx.xx:9200/_cat/health?v
elasticsearch集群状态查看:
http://10.16.1xx.xx:9200/_cat/nodes?v
索引状态查看:
http://10.16.1xx.xx:9200/_cat/indices?v
kibana地址登录:
http://10.16.1xx.xx:5601/app/home#/
六.调整索引分区
调整已存在索引的副本分区:
curl -XPUT -H "Content-Type:application/json" -d '{"settings":{"number_of_replicas":2}}' 'http://10.16.xx.xx:9200/testindex/_settings'
调整集群最大分片数:
curl -XPUT -H "Content-Type:application/json" http://10.16.xx.xx:9200/_cluster/settings -d '{"transient":{"cluster":{"max_shards_per_node":10000}}}'
调整默认主分片和副本分片数:
curl -X PUT -H "Content-Type:application/json" -d '{"index_patterns":["*"],"settings":{"number_of_shards":3,"number_of_replicas":2}}' http://10.16.xx.xx:9200/_template/template_http_request_record
七.安装说明
- 需要先在k8s的节点创建/data/es_data的路径来存放数据。
- 使用配置前将配置中的autopocket替换成自己需要的命名,配置的命名空间为:zx-app,修改成自己的命名空间,appname为pcauto-pocket,修改成自己的应用名称,该标签也可以删除。
- 配置中的镜像地址采用了私有的镜像地址,镜像是dockerhub上下载的elasticsearch:7.17.22 镜像到私有镜像仓库的,如果能直通外网的k8s可以直接用elasticsearch:7.17.22,kibana的镜像是kibana:7.17.22。否则配置一个可以通的镜像地址。
- 默认没有配置用户名和密码。
- 节点标签的命名规则若有更改需要在每个statefulset中的affinity: 中的标签名称进行修改。