相关内容
bclinux aarch64 ceph 14.2.10 文件存储 Ceph File System, 需要部署mds： ceph-deploy mds-CSDN博客
ceph-deploy bclinux aarch64 ceph 14.2.10【3】vdbench fsd 文件系统测试-CSDN博客
ceph-deploy bclinux aarch64 ceph 14.2.10【2】vdbench rbd 块设备rbd 测试失败-CSDN博客
ceph-deploy bclinux aarch64 ceph 14.2.10-CSDN博客
部署对象存储网关
cd /etc/ceph
ceph-deploy rgw create ceph-0 ceph-1 ceph-2 ceph-3成功日志
[root@ceph-0 ceph]# ceph-deploy rgw create ceph-0 ceph-1 ceph-2 ceph-3
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (2.0.1): /usr/bin/ceph-deploy rgw create ceph-0 ceph-1 ceph-2 ceph-3
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] rgw : [('ceph-0', 'rgw.ceph-0'), ('ceph-1', 'rgw.ceph-1'), ('ceph-2', 'rgw.ceph-2'), ('ceph-3', 'rgw.ceph-3')]
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : create
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0xffffa1b8faa0>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] func : <function rgw at 0xffffa1d554d0>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.rgw][DEBUG ] Deploying rgw, cluster ceph hosts ceph-0:rgw.ceph-0 ceph-1:rgw.ceph-1 ceph-2:rgw.ceph-2 ceph-3:rgw.ceph-3
[ceph-0][DEBUG ] connected to host: ceph-0
[ceph-0][DEBUG ] detect platform information from remote host
21.10U3 LTS
bclinux
[ceph-0][DEBUG ] detect machine type
[ceph_deploy.rgw][INFO ] Distro info: bclinux 21.10U3 21.10U3 LTS
[ceph_deploy.rgw][DEBUG ] remote host will use systemd
[ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to ceph-0
[ceph-0][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-0][WARNIN] rgw keyring does not exist yet, creating one
[ceph-0][DEBUG ] create a keyring file
[ceph-0][DEBUG ] create path recursively if it doesn't exist
[ceph-0][INFO ] Running command: ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.ceph-0 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.ceph-0/keyring
[ceph-0][INFO ] Running command: systemctl enable ceph-radosgw@rgw.ceph-0
[ceph-0][WARNIN] Created symlink /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw@rgw.ceph-0.service → /usr/lib/systemd/system/ceph-radosgw@.service.
[ceph-0][INFO ] Running command: systemctl start ceph-radosgw@rgw.ceph-0
[ceph-0][INFO ] Running command: systemctl enable ceph.target
[ceph_deploy.rgw][INFO ] The Ceph Object Gateway (RGW) is now running on host ceph-0 and default port 7480
dhclient(1787) is already running - exiting.
This version of ISC DHCP is based on the release available
on ftp.isc.org. Features have been added and other changes
have been made to the base software release in order to make
it work better with this distribution.
Please report issues with this software via:
https://gitee.com/src-openeuler/dhcp/issues
exiting.
dhclient(1787) is already running - exiting.
This version of ISC DHCP is based on the release available
on ftp.isc.org. Features have been added and other changes
have been made to the base software release in order to make
it work better with this distribution.
Please report issues with this software via:
https://gitee.com/src-openeuler/dhcp/issues
exiting.
[ceph-1][DEBUG ] connected to host: ceph-1
[ceph-1][DEBUG ] detect platform information from remote host
21.10U3 LTS
bclinux
[ceph-1][DEBUG ] detect machine type
[ceph_deploy.rgw][INFO ] Distro info: bclinux 21.10U3 21.10U3 LTS
[ceph_deploy.rgw][DEBUG ] remote host will use systemd
[ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to ceph-1
[ceph-1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-1][WARNIN] rgw keyring does not exist yet, creating one
[ceph-1][DEBUG ] create a keyring file
[ceph-1][DEBUG ] create path recursively if it doesn't exist
[ceph-1][INFO ] Running command: ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.ceph-1 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.ceph-1/keyring
[ceph-1][INFO ] Running command: systemctl enable ceph-radosgw@rgw.ceph-1
[ceph-1][WARNIN] Created symlink /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw@rgw.ceph-1.service → /usr/lib/systemd/system/ceph-radosgw@.service.
[ceph-1][INFO ] Running command: systemctl start ceph-radosgw@rgw.ceph-1
[ceph-1][INFO ] Running command: systemctl enable ceph.target
[ceph_deploy.rgw][INFO ] The Ceph Object Gateway (RGW) is now running on host ceph-1 and default port 7480
dhclient(1742) is already running - exiting.
This version of ISC DHCP is based on the release available
on ftp.isc.org. Features have been added and other changes
have been made to the base software release in order to make
it work better with this distribution.
Please report issues with this software via:
https://gitee.com/src-openeuler/dhcp/issues
exiting.
dhclient(1742) is already running - exiting.
This version of ISC DHCP is based on the release available
on ftp.isc.org. Features have been added and other changes
have been made to the base software release in order to make
it work better with this distribution.
Please report issues with this software via:
https://gitee.com/src-openeuler/dhcp/issues
exiting.
[ceph-2][DEBUG ] connected to host: ceph-2
[ceph-2][DEBUG ] detect platform information from remote host
21.10U3 LTS
bclinux
[ceph-2][DEBUG ] detect machine type
[ceph_deploy.rgw][INFO ] Distro info: bclinux 21.10U3 21.10U3 LTS
[ceph_deploy.rgw][DEBUG ] remote host will use systemd
[ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to ceph-2
[ceph-2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-2][WARNIN] rgw keyring does not exist yet, creating one
[ceph-2][DEBUG ] create a keyring file
[ceph-2][DEBUG ] create path recursively if it doesn't exist
[ceph-2][INFO ] Running command: ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.ceph-2 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.ceph-2/keyring
[ceph-2][INFO ] Running command: systemctl enable ceph-radosgw@rgw.ceph-2
[ceph-2][WARNIN] Created symlink /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw@rgw.ceph-2.service → /usr/lib/systemd/system/ceph-radosgw@.service.
[ceph-2][INFO ] Running command: systemctl start ceph-radosgw@rgw.ceph-2
[ceph-2][INFO ] Running command: systemctl enable ceph.target
[ceph_deploy.rgw][INFO ] The Ceph Object Gateway (RGW) is now running on host ceph-2 and default port 7480
dhclient(1722) is already running - exiting.
This version of ISC DHCP is based on the release available
on ftp.isc.org. Features have been added and other changes
have been made to the base software release in order to make
it work better with this distribution.
Please report issues with this software via:
https://gitee.com/src-openeuler/dhcp/issues
exiting.
dhclient(1722) is already running - exiting.
This version of ISC DHCP is based on the release available
on ftp.isc.org. Features have been added and other changes
have been made to the base software release in order to make
it work better with this distribution.
Please report issues with this software via:
https://gitee.com/src-openeuler/dhcp/issues
exiting.
[ceph-3][DEBUG ] connected to host: ceph-3
[ceph-3][DEBUG ] detect platform information from remote host
21.10U3 LTS
bclinux
[ceph-3][DEBUG ] detect machine type
[ceph_deploy.rgw][INFO ] Distro info: bclinux 21.10U3 21.10U3 LTS
[ceph_deploy.rgw][DEBUG ] remote host will use systemd
[ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to ceph-3
[ceph-3][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-3][WARNIN] rgw keyring does not exist yet, creating one
[ceph-3][DEBUG ] create a keyring file
[ceph-3][DEBUG ] create path recursively if it doesn't exist
[ceph-3][INFO ] Running command: ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.ceph-3 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.ceph-3/keyring
[ceph-3][INFO ] Running command: systemctl enable ceph-radosgw@rgw.ceph-3
[ceph-3][WARNIN] Created symlink /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw@rgw.ceph-3.service → /usr/lib/systemd/system/ceph-radosgw@.service.
[ceph-3][INFO ] Running command: systemctl start ceph-radosgw@rgw.ceph-3
[ceph-3][INFO ] Running command: systemctl enable ceph.target
[ceph_deploy.rgw][INFO ] The Ceph Object Gateway (RGW) is now running on host ceph-3 and default port 7480
[root@ceph-0 ceph]# cp: cannot stat '/tmp/security_updates': No such file or directory
cp: cannot stat '/tmp/security_updates': No such file or directory
cp: cannot stat '/tmp/security_updates': No such file or directory
创建成功
验证服务状态（失败）
日志目录
/var/log/ceph
日志文件
ceph-client.rgw.ceph-0.log
清空日志文件，重启服务
cd /var/log/ceph
echo "" > ceph-client.rgw.ceph-0.log
systemctl start ceph-radosgw@rgw-ceph-0先看服务日志
journalctl -xeu ceph-radosgw@rgw-ceph-0日志
Nov 14 13:38:08 ceph-0 radosgw[58375]: 2023-11-14 13:38:08.450 ffff7ab37010 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-rgw-ceph-0/keyring: (2) No such file or directory
Nov 14 13:38:08 ceph-0 radosgw[58375]: 2023-11-14 13:38:08.450 ffff7ab37010 -1 AuthRegistry(0xffff7a330148) no keyring found at /var/lib/ceph/radosgw/ceph-rgw-ceph-0/keyring, disabling cephx
Nov 14 13:38:08 ceph-0 radosgw[58375]: 2023-11-14 13:38:08.450 ffff7ab37010 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-rgw-ceph-0/keyring: (2) No such file or directory
Nov 14 13:38:08 ceph-0 radosgw[58375]: 2023-11-14 13:38:08.450 ffff7ab37010 -1 AuthRegistry(0xffffdc7f98c8) no keyring found at /var/lib/ceph/radosgw/ceph-rgw-ceph-0/keyring, disabling cephx
Nov 14 13:38:08 ceph-0 radosgw[58375]: failed to fetch mon config (--no-mon-config to skip)
Nov 14 13:38:08 ceph-0 systemd[1]: ceph-radosgw@rgw-ceph-0.service: Main process exited, code=exited, status=1/FAILURE
没有按照官方文档执行网关安装有关？ceph-deploy install --rgw node..
但是这个版本的ceph-deploy指定的ceph版本太低，且是el6
解决 /var/lib/ceph/radosgw/ceph-rgw-ceph-0/keyring: (2) No such file or directory
查看/etc/ceph目录下的keyring文件，发现已生成了ceph.bootstrap-rgw.keyring
错误的路径/var/lib/ceph/radosgw/ceph-rgw-ceph-0/keyring
正确的路径/var/lib/ceph/radosgw/ceph-rgw.ceph-0/keyring
不知道为什么路径有一个符号错误
做个软连接
cd /var/lib/ceph/radosgw
ln -s ceph-rgw.ceph-0 ceph-rgw-ceph-0
错误：
Nov 14 14:11:43 ceph-0 radosgw[59264]: 2023-11-14 14:11:43.076 ffff99055730 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
Nov 14 14:11:43 ceph-0 radosgw[59264]: 2023-11-14 14:11:43.080 ffff99856730 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
Nov 14 14:11:43 ceph-0 radosgw[59264]: failed to fetch mon config (--no-mon-config to skip)
Nov 14 14:11:43 ceph-0 systemd[1]: ceph-radosgw@rgw-ceph-0.service: Main process exited, code=exited, status=1/FAILURE
重新执行部署
cd /etc/ceph
ceph-deploy rgw create ceph-0 ceph-1 ceph-2 ceph-3
重新部署后，会生成/var/log/ceph/ceph-client.rgw.ceph-0.log日志文件
其中有日志
Numerical result out of range
2023-11-14 14:32:24.461 ffff8abe8010 0 deferred set uid:gid to 1000:1000 (ceph:ceph)
2023-11-14 14:32:24.461 ffff8abe8010 0 ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable), process radosgw, pid 60035
2023-11-14 14:32:25.509 ffff8abe8010 0 rgw_init_ioctx ERROR: librados::Rados::pool_create returned (34) Numerical result out of range (this can be due to a pool or placement group misconfiguration, e.g. pg_num < pgp_num or mon_max_pg_per_osd exceeded)
2023-11-14 14:32:25.509 ffff8abe8010 0 failed reading realm info: ret -34 (34) Numerical result out of range
2023-11-14 14:32:25.509 ffff8abe8010 0 ERROR: failed to start notify service ((34) Numerical result out of range
2023-11-14 14:32:25.509 ffff8abe8010 0 ERROR: failed to init services (ret=(34) Numerical result out of range)
2023-11-14 14:32:25.513 ffff8abe8010 -1 Couldn't init storage provider (RADOS)
设置默认pg_num pgs_num
/etc/ceph/ceph.conf
osd pool default size = 3
osd pool default min_size = 1
osd pool default pg num = 128
osd pool default pgp num = 128
重新部署(--overwrite-conf)
cd /etc/ceph
ceph-deploy --overwrite-conf rgw create ceph-0 ceph-1 ceph-2 ceph-3
故障依旧
成功的方案
删掉所有pool，重新部署，参考以下历史记录
652 ceph osd pool ls
653 ceph osd pool rm vdbench
654 ceph osd pool rm vdbench vdbench --yes-i-really-really-mean-it
655 ls /dev/
656 mount
657 df -h
658 umount /mnt/rbd0
659 ceph osd ls
660 ceph osd pool ls
661 ceph osd volume ls
662 ceph osd dump
663 ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it
664 ceph fs ls
665 ceph fs rm cephfs
666 ceph fs fail
667 ceph fs fail cephfs
668 ceph -s
669 ceph fs status cephfs
670 ceph fs ls
671 ceph fs rm cephfs
672 ceph fs rm cephfs --yes-i-really-mean-it
673 ceph fs ls
674 ceph osd pool ls
675 ceph osd pool rm cephfs_data cephfs_data --yes-i-really-mean-it
676 ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it
677 ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it
678 ceph -
679 ceph -s
680 cd /etc/ceph
681 ceph-deploy rgw create ceph-0
682 cat /var/log/ceph/ceph-client.rgw.ceph-0.log
683 systemctl status ceph-radosgw@rgw.ceph-0
再来一次集群环境的部署
cd /etc/ceph
ceph-deploy --overwrite-conf rgw create ceph-0 ceph-1 ceph-2 ceph-3部分日志
ceph -s rgw已经起来了
======================
以下为参考官方文档的失败经理，留档参考
部署对象存储网关rgw（失败）
ceph-deploy install --rgw ceph-0 ceph-1 ceph-2 ceph-3故障 Unable to find a match: epel-release
dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm -y故障Status code: 404
坑爹的阿里云镜像，没有epel 8，bclinux又是el8版本！
采用huawei epel镜像
[epel-everything]
name=huawei epel mirror
baseurl=https://repo.huaweicloud.com/epel/8/Everything/aarch64/
enabled=1
gpgcheck=0
yum install epel-release --nobest --skip-broken
ceph-deploy 会强制执行yum install epel-release(失败)
注释掉epel-release的安装
修改/usr/lib/python2.7/site-packages/ceph_deploy/hosts/centos/install.py 60行，注释掉epel-release的安装（前面已经手动--skip-broken安装好了）
生效了
故障 Unable to find a match: yum-plugin-priorities
参考上面思路，直接注释掉/usr/lib/python2.7/site-packages/ceph_deploy/hosts/centos/install.py 61,62行
自动找到老版本ceph-release-1-0.el6 Status code: 404
手动安装
修改/usr/lib/python2.7/site-packages/ceph_deploy/install.py
安装版本参数args.release = 'mimic' 改为 args.release = 'nautilus
关于python3版本
bclinux python3.7.9
cenos7 python3.6.8
openeuler 20.03 lts sp3 python 3.7.9
参考
Ceph Object Gateway — Ceph Documentation
如何在 CentOS 8 和 RHEL 8 服务器上启用 EPEL 仓库 - 知乎 (zhihu.com)
ceph-deploy部署ceph-nautilus_ceph nautilus-CSDN博客
