前言
- 业务需要将 Elasticsearch 快照到 AWS S3，再将快照拷贝到 Windows 系统，并恢复到 Elasticsearch。如下图所示：
-
环境
Elasticsearch 7.10.1 Windows Server 2019 Ubuntu 20.04 （ES 宿主）
ES 集群1 安装 S3 插件
- 官方文档： https://www.elastic.co/guide/...
- 应在 ES 集群的所有节点上安装插件
- 下载 S3 插件和 SHA hash，并上传到 ES 服务器
-
查看并校验
# ll -ah total 4.6M drwxr-xr-x 2 qhy qhy 4.0K Mar 9 01:55 ./ drwxr-xr-x 7 qhy qhy 4.0K Mar 9 01:50 ../ -rw-rw-r-- 1 qhy qhy 4.6M Mar 9 01:55 repository-s3-7.10.1.zip -rw-rw-r-- 1 qhy qhy 154 Mar 9 01:55 repository-s3-7.10.1.zip.sha512 # shasum -a 512 -c repository-s3-7.10.1.zip.sha512 repository-s3-7.10.1.zip: OK
-
安装
# /usr/share/elasticsearch/bin/elasticsearch-plugin install file:///home/qhy/es_down/repository-s3-7.10.1.zip -> Installing file:///home/qhy/es_down/repository-s3-7.10.1.zip -> Downloading file:///home/qhy/es_down/repository-s3-7.10.1.zip [=================================================] 100% @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ java.lang.RuntimePermission accessDeclaredMembers java.lang.RuntimePermission getClassLoader java.lang.reflect.ReflectPermission suppressAccessChecks java.net.SocketPermission * connect,resolve java.util.PropertyPermission es.allow_insecure_settings read,write See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. Continue with installation? [y/N]y -> Installed repository-s3
- 重启
ES服务
ES 集群1 添加 KEY
- 官方文档： https://www.elastic.co/guide/...
- 应在 ES 集群的所有节点上添加 key
-
查看有哪些 key
/usr/share/elasticsearch/bin/elasticsearch-keystore list
-
添加
access_key/usr/share/elasticsearch/bin/elasticsearch-keystore add s3.client.default.access_key
-
添加
secret_key/usr/share/elasticsearch/bin/elasticsearch-keystore add s3.client.default.secret_key
-
重载配置
POST _nodes/reload_secure_settings
创建 S3 快照仓库
- 官方文档： https://www.elastic.co/guide/...
- 注意中国区不支持
region，用endpoint替代
PUT _snapshot/my_s3_repository
{
"type": "s3",
"settings": {
"endpoint": "s3.cn-northwest-1.amazonaws.com.cn",
"bucket": "zt-hadoop-cn-northwest-1",
"base_path": "es_snapshot",
"max_snapshot_bytes_per_sec": "200mb", # 调整快照创建的速度，默认 40mb
"max_restore_bytes_per_sec": "200mb" # 调整快照恢复的速度，默认无限制
}
}
- 查看所有注册的快照仓库
GET _snapshot/_all
- 删除快照仓库
DELETE _snapshot/my_s3_repository
- 调整集群恢复分片速度和并发数
PUT _cluster/settings
{
"transient": {
"indices.recovery.max_bytes_per_sec": "200mb", # 默认 40mb
"cluster.routing.allocation.node_concurrent_recoveries": "5" # 默认 2
}
}
- 查看集群配置（包括默认配置）
GET _cluster/settings?flat_settings&include_defaults
创建快照
- 官方文档： https://www.elastic.co/guide/...
PUT /_snapshot/my_s3_repository/snapshot_zt
{
"indices": "zt_product_doc_index_20210223_3",
"ignore_unavailable": true,
"include_global_state": false
}
- 查看一个
my_s3_repository仓库下的所有快照
GET _snapshot/my_s3_repository/_all
- 查看
snapshot_zt快照的概要状态
GET _snapshot/my_s3_repository/snapshot_zt
- 查看
snapshot_zt快照的详细状态
GET _snapshot/my_s3_repository/snapshot_zt/_status
- 删除快照
DELETE _snapshot/my_s3_repository/snapshot_zt
从 S3 恢复快照
-
官方文档： https://www.elastic.co/guide/...
POST /_snapshot/my_s3_repository/snapshot_zt/_restore { "indices": "zt_product_doc_index_20210223_3", "index_settings": { "index.number_of_replicas": 0 }, "rename_pattern": "zt_product_doc_index_20210223_3", "rename_replacement": "zt_product_doc_index_20210223_3_restore" } -
增加副本
PUT zt_product_doc_index_20210223_3_restore/_settings { "index.number_of_replicas" : "1" }
从 Windows 共享目录恢复快照
-
已将
S3上的文件拷贝到Windows的共享目录，并挂载到ES 集群2服务器的/mnt/winshare目录# tree -d /mnt/winshare/ /mnt/winshare/ └── es_snapshot └── indices └── kM6SVcCrTUGjP-634aDUYg ├── 0 ├── 1 └── 2 - Linux 挂载 Windows 共享目录参见： Ubuntu 20.04 读写 Windows 10 共享目录
-
在
elasticsearch.yml文件中添加如下配置，并重启 ES 服务path.repo: - /mnt/winshare
-
创建 Windows 系统的快照仓库
PUT /_snapshot/my_backup { "type": "fs", "settings": { "location": "/mnt/winshare/es_snapshot" } } -
官方文档
POST /_snapshot/my_backup/snapshot_zt/_restore { "indices": "zt_product_doc_index_20210223_3", "index_settings": { "index.number_of_replicas": 0 }, "rename_pattern": "zt_product_doc_index_20210223_3", "rename_replacement": "zt_product_doc_index_20210223_3_smb" } -
增加副本
PUT zt_product_doc_index_20210223_3_smb/_settings { "index.number_of_replicas" : "1" }
本文出自 qbit snap
