k8s分布式存储-ceph

文章目录

  • Ceph
    • deploy-ceph部署
      • 1.系统环境初始化
        • 1.1 修改主机名,DNS解析
        • 1.2 时间同步
        • 1.3 配置apt基础源与ceph源
        • 1.4关闭selinux与防火墙
        • 1.5 **创建** ceph **集群部署用户** cephadmin
        • 1.6分发密钥
      • 2. ceph部署
        • 2.1 **安装** ceph 部署工具
        • 2.2 **初始化** mon **节点**
        • 2.3 安装ceph-mon服务
          • 2.3.1 ceph-mon节点安装ceph-mon
          • 2.3.2 ceph 集群添加 ceph-mon 服务
          • 2.3.2 验证mon节点
        • 2.4 分发admin 秘钥
        • 2.5 部署manager
          • 2.5.1 部署 ceph-mgr 节点
          • 2.5.2 验证ceph-mgr节点
        • 2.6 部署osd
          • 2.6.1 初始化存储节点
          • 2.6.2 OSD与磁盘部署关系
          • 2.6.3 添加OSD
          • 2.6.4 验证ceph集群
        • 2.7 测试上传与下载数据
          • 创建 pool:
          • 上传数据
          • 列出数据
          • 文件信息
          • 下载文件
          • 修改文件
          • 删除文件
      • 3. Ceph RBD 使用详解
        • 3.1 RBD架构图
        • 3.2 创建存储池
        • 3.3 创建img镜像
          • 3.3.1 创建镜像
          • 3.3.2 列出镜像详细信息
          • 3.3.3 :镜像的特性
        • 3.4 客户端使用RBD
          • 3.4.1 客户端安装 ceph-common
          • 3.4.2 同步账户认证文件
          • 3.4.3 客户端映射镜像
          • 3.4.4 客户端挂载使用
      • 4.CephFS使用详解
        • 4.1 部署MDS服务
        • 4.2 创建CephFS mdetadata和data存储池
        • 4.3 创建 cephFS 并验证
        • 4.4 创建cephFS客户端账户
        • 4.5 安装ceph客户端
        • 4.6 同步认证文件
        • 4.7 客户端安装ceph-common
        • 4.8 cephfs挂载使用
          • 4.8.1 内核空间挂载使用ceph-fs
          • 4.8.2 开机自动挂载
        • 4.9用户空间挂载 ceph-fs
          • 4.9.1**安装** ceph-fuse
          • 4.9.2 ceph-fuse 挂载 ceph
          • 4.9.3 开机自动挂载
      • 5.k8s使用ceph案例
        • 5.1 RBD静态存储
          • 5.1.1 使用pv/pvc挂载RBD
          • 5.1.2 直接使用pod挂载RBD
        • 5.1 RBD动态存储类
          • 5.1.1 创建rbd pool
          • 5.1.2 创建 admin 用户 secret:
          • 5.1.3 创建普通用户的 secret
          • 5.1.4 创建存储类
          • 5.1.5 创建基于存储类的PVC
          • 5.1.6 运行单机mysql pod验证
        • 5.2 cephFS静态存储
          • 5.2.1 使用pv/pvc挂载cephFS
          • 5.2 直接使用pod挂载cephFS
        • 5.4 cephFS动态存储类
        • 5.5 ceph-csi 动态存储
          • Ceph-CSI RBD
          • Ceph-CSI CephFS
    • 理解
          • 1.存储数据, object, pg,pgp, pool, osd, 存储磁盘的关系
          • 2.Filestore、BlueStore、journal理解
    • 需求
          • 1.[删除OSD的正确方式](https://www.cnblogs.com/varden/p/16274669.html)
          • 2.[ceph 旧OSD 节点 格式化 数据,加入新 ceph集群](https://www.cnblogs.com/breg/p/16517598.html)
          • 3.ceph如何修改配置文件
    • 报错记录
          • 1. bash: python2: command not found
          • 2.[ceph_deploy][ERROR ] RuntimeError: AttributeError: module 'platform' has no attribute 'linux_distribution'
          • 3.apt-cache madison ceph-deploy 为1.5.38的低版本
          • 4. RuntimeError: Failed to execute command: /usr/sbin/ceph-volume lvm zap /dev/sdd
          • 5. 使用zap擦除格式化磁盘时,报错[ERROR ] RuntimeError: Failed to execute command: /usr/sbin/ceph-volume lvm zap /dev/sdd
          • 6. mons are allowing insecure global_id reclaim
          • 7. 1 pool(s) do not have an application enabled
          • 8.clock skew detected on mon.ceph-master02
          • 9. 低内核无法挂载ceph rbd块存储
          • 10. 1 filesystem is online with fewer MDS than max_mds

Ceph

deploy-ceph部署

投入使用ceph前,要知道一个很现实得问题,ceph对低版本内核得客户端使用非常不友好,低内核是指小于等于3.10.0-862,默认的centos7.5及以下的系统都是小于此类内核,无法正常使用ceph的文件存储(cephFS)块存储(RBD)。

ceph部署一定要考虑好版本问题,经测试如果想使用ceph16版本,那你的客户端操作系统内核小于3.10.0-862根本用不了,常见的centos7.5以下默认没升级的内核都是小于3.10.0-862,所以这一大批服务器使用ceph提供的rdb存储都会有问题,而且ceph已经不提供centos的16版本的ceph-common组件,也就是说ceph集群部署16版本,常见的客户端centos7系统只能使用15版本的ceph-common,虽说也可以使用,但也存在一定隐患毕竟不是同一版本客户端软件,目前推荐使用ceph15的最高版本,15版本的安装与16相同,只是ceph源不同。

以上说法不正确,ceph版本选择和客户端内核没有关系,是所有版本的ceph都不友好支持内核小于等于3.10.0-862(CentOS7.5)

环境

ubuntu 18.04b版本

ceph 16.10版本

主机名IP部署 内容
ceph-master01public IP:172.26.156.217 内部通讯IP: 10.0.0.217mon,mgr,osd,ceph-deploy
ceph-master02public IP:172.26.156.218 内部通讯IP:10.0.0.218mon,mgr,osd
ceph-master03public IP:172.26.156.219 内部通讯IP:10.0.0.219mon,mgr,osd

服务器配置:服务器型号,cpu,内存,磁盘大小及磁盘类型必须完全一致,磁盘最好全ssd盘

1.系统环境初始化

1.1 修改主机名,DNS解析
master01:
hostnamectl  set-hostname  ceph-master01
vi /etc/hostname
ceph-master01master02:
hostnamectl  set-hostname  ceph-master02
vi /etc/hostname
ceph-master02master03:
hostnamectl  set-hostname  ceph-master03
vi /etc/hostname
ceph-master03vi /etc/hosts
10.0.0.217 ceph-master01.example.local ceph-master01
10.0.0.218 ceph-master02.example.local ceph-master02
10.0.0.219 ceph-master03.example.local ceph-master03
1.2 时间同步

所有服务器执行

#修改时区
timedatectl set-timezone Asia/Shanghai#时间同步
root@ubuntu:~# apt install ntpdate
root@ubuntu:~# ntpdate  ntp.aliyun.com1 Sep 20:54:39 ntpdate[9120]: adjust time server 203.107.6.88 offset 0.003441 sec
root@ubuntu:~# crontab  -e 
crontab: installing new crontab
root@ubuntu:~# crontab  -l 
* * * * * ntpdate  ntp.aliyun.com
1.3 配置apt基础源与ceph源

所有服务器执行如下命令自动替换

#基础源
sed -i "s@http://.*archive.ubuntu.com@http://mirrors.tuna.tsinghua.edu.cn@g" /etc/apt/sources.list
sed -i "s@http://.*security.ubuntu.com@http://mirrors.tuna.tsinghua.edu.cn@g" /etc/apt/sources.list
#ceph源
echo "deb https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic main" >> /etc/apt/sources.list.d/ceph.list
echo "deb https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-nautilus  bionic main" >> /etc/apt/sources.list.d/ceph.list#导入ceph源key,不然不能使用ceph源
wget -q -O- 'https://mirrors.tuna.tsinghua.edu.cn/ceph/keys/release.asc' | sudo apt-key add -# ceph仓库为https的话需要安装下面,不然无法使用https源
apt install -y apt-transport-https ca-certificates curl software-properties-commonapt update
1.4关闭selinux与防火墙
# ufw disable
1.5 创建 ceph 集群部署用户 cephadmin

推荐使用指定的普通用户部署和运行 ceph 集群,普通用户只要能以非交互方式执行 sudo 命令执行一些特权命令即可,新版的 ceph-deploy 可以指定包含 root 的在内只要可以执 行 sudo 命令的用户,不过仍然推荐使用普通用户,ceph 集群安装完成后会自动创建 ceph 用户(ceph 集群默认会使用 ceph 用户运行各服务进程, ceph-osd ),因此推荐 使用除了 ceph 用户之外的比如 cephuser**、**cephadmin 这样的普通用户去部署和 管理ceph 集群。

在包含 ceph-deploy 节点的存储节点、mon 节点和 mgr 节点等创建 cephadmin 用户.

groupadd -r -g 2088 cephadmin && useradd -r -m -s /bin/bash -u 2088 -g 2088 cephadmin && echo cephadmin:chinadci888. | chpasswd

各服务器允许 cephadmin 用户以 sudo 执行特权命令:

~# echo "cephadmin ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
1.6分发密钥

deploy节点要与所有服务器mon,mgr,osd节点免密,本文这里只有三台服务器,mon,mgr,osd都混合一起部署,所以只免密了三台服务器

master01(deploy节点):

su - cephadmin
ssh-keygen
ssh-copy-id   cephadmin@ceph-master01
ssh-copy-id   cephadmin@ceph-master02
ssh-copy-id   cephadmin@ceph-master03

2. ceph部署

2.1 安装 ceph 部署工具
cephadmin@ceph-master01:~$ apt-cache madison ceph-deploy
ceph-deploy |      2.0.1 | https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-nautilus bionic/main amd64 Packages
ceph-deploy |      2.0.1 | https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-nautilus bionic/main i386 Packages
ceph-deploy | 1.5.38-0ubuntu1 | http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic/universe amd64 Packages
ceph-deploy | 1.5.38-0ubuntu1 | http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic/universe i386 Packages
cephadmin@ceph-master01:~$ sudo apt install ceph-deploy
2.2 初始化 mon 节点

Ubuntu 各服务器需要单独安装 Python2(mon,mgr,osd节点所有服务器必须做):

cephadmin@ceph-master01:~$ sudo apt install python2.7 -y
cephadmin@ceph-master01:~$ sudo ln -sv /usr/bin/python2.7 /usr/bin/python2

ceph-master01:

ceph-deploy new --cluster-network 10.0.0.0/24 --public-network 172.26.0.0/16 ceph-master01 ceph-master02 ceph-master03

–cluster-network: 集群内部之间通讯网络

–public-network:业务客户端使用网络,单独使用网络,规避

~$ mkdir /etc/ceph-cluster
~$ sudo chown  cephadmin:cephadmin /etc/ceph-cluster
~$ cd /etc/ceph-cluster/
cephadmin@ceph-master01:/etc/ceph-cluster$ ceph-deploy new --cluster-network 10.0.0.0/24 --public-network 172.26.0.0/16 ceph-master01  ceph-master02 ceph-master03
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadmin/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy new --cluster-network 10.0.0.0/24 --public-network 172.26.0.0/16 ceph-master01 ceph-master02 ceph-master03
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7efd0a772e10>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  ssh_copykey                   : True
[ceph_deploy.cli][INFO  ]  mon                           : ['ceph-master01', 'ceph-master02', 'ceph-master03']
[ceph_deploy.cli][INFO  ]  func                          : <function new at 0x7efd07a2bbd0>
[ceph_deploy.cli][INFO  ]  public_network                : 172.26.0.0/16
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  cluster_network               : 10.0.0.0/24
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  fsid                          : None
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
[ceph-master01][DEBUG ] connection detected need for sudo
[ceph-master01][DEBUG ] connected to host: ceph-master01 
[ceph-master01][DEBUG ] detect platform information from remote host
[ceph-master01][DEBUG ] detect machine type
[ceph-master01][DEBUG ] find the location of an executable
[ceph-master01][INFO  ] Running command: sudo /bin/ip link show
[ceph-master01][INFO  ] Running command: sudo /bin/ip addr show
[ceph-master01][DEBUG ] IP addresses found: [u'172.26.156.217', u'10.0.0.217']
[ceph_deploy.new][DEBUG ] Resolving host ceph-master01
[ceph_deploy.new][DEBUG ] Monitor ceph-master01 at 172.26.156.217
[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
[ceph-master02][DEBUG ] connected to host: ceph-master01 
[ceph-master02][INFO  ] Running command: ssh -CT -o BatchMode=yes ceph-master02
[ceph-master02][DEBUG ] connection detected need for sudo
[ceph-master02][DEBUG ] connected to host: ceph-master02 
[ceph-master02][DEBUG ] detect platform information from remote host
[ceph-master02][DEBUG ] detect machine type
[ceph-master02][DEBUG ] find the location of an executable
[ceph-master02][INFO  ] Running command: sudo /bin/ip link show
[ceph-master02][INFO  ] Running command: sudo /bin/ip addr show
[ceph-master02][DEBUG ] IP addresses found: [u'10.0.0.218', u'172.26.156.218']
[ceph_deploy.new][DEBUG ] Resolving host ceph-master02
[ceph_deploy.new][DEBUG ] Monitor ceph-master02 at 172.26.156.218
[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
[ceph-master03][DEBUG ] connected to host: ceph-master01 
[ceph-master03][INFO  ] Running command: ssh -CT -o BatchMode=yes ceph-master03
[ceph-master03][DEBUG ] connection detected need for sudo
[ceph-master03][DEBUG ] connected to host: ceph-master03 
[ceph-master03][DEBUG ] detect platform information from remote host
[ceph-master03][DEBUG ] detect machine type
[ceph-master03][DEBUG ] find the location of an executable
[ceph-master03][INFO  ] Running command: sudo /bin/ip link show
[ceph-master03][INFO  ] Running command: sudo /bin/ip addr show
[ceph-master03][DEBUG ] IP addresses found: [u'172.26.156.219', u'10.0.0.219']
[ceph_deploy.new][DEBUG ] Resolving host ceph-master03
[ceph_deploy.new][DEBUG ] Monitor ceph-master03 at 172.26.156.219
[ceph_deploy.new][DEBUG ] Monitor initial members are ['ceph-master01', 'ceph-master02', 'ceph-master03']
[ceph_deploy.new][DEBUG ] Monitor addrs are [u'172.26.156.217', u'172.26.156.218', u'172.26.156.219']
[ceph_deploy.new][DEBUG ] Creating a random mon key...
[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...cephadmin@ceph-master01:/etc/ceph-cluster$ ll
total 36
drwxr-xr-x  2 cephadmin cephadmin  4096 Sep  2 16:50 ./
drwxr-xr-x 91 root      root       4096 Sep  2 16:22 ../
-rw-rw-r--  1 cephadmin cephadmin   326 Sep  2 16:50 ceph.conf
-rw-rw-r--  1 cephadmin cephadmin 17603 Sep  2 16:50 ceph-deploy-ceph.log
-rw-------  1 cephadmin cephadmin    73 Sep  2 16:50 ceph.mon.keyring

此步骤必须执行,否 ceph 集群的后续安装步骤会报错。

cephadmin@ceph-master01:/etc/ceph-cluster$ ceph-deploy install --no-adjust-repos --nogpgcheck ceph-master01 ceph-master02 ceph-master03

--no-adjust-repos #不修改已有的 apt 仓库源(默认会使用官方仓库) 
--nogpgcheck #不进行校验 
cephadmin@ceph-master01:/etc/ceph-cluster$ ceph-deploy install --no-adjust-repos --nogpgcheck ceph-master01 ceph-master02 ceph-master03
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadmin/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy install --no-adjust-repos --nogpgcheck ceph-master01 ceph-master02 ceph-master03
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  testing                       : None
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f59e4913e60>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  dev_commit                    : None
[ceph_deploy.cli][INFO  ]  install_mds                   : False
[ceph_deploy.cli][INFO  ]  stable                        : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  adjust_repos                  : False
[ceph_deploy.cli][INFO  ]  func                          : <function install at 0x7f59e51c5b50>
[ceph_deploy.cli][INFO  ]  install_mgr                   : False
[ceph_deploy.cli][INFO  ]  install_all                   : False
[ceph_deploy.cli][INFO  ]  repo                          : False
[ceph_deploy.cli][INFO  ]  host                          : ['ceph-master01', 'ceph-master02', 'ceph-master03']
[ceph_deploy.cli][INFO  ]  install_rgw                   : False
[ceph_deploy.cli][INFO  ]  install_tests                 : False
[ceph_deploy.cli][INFO  ]  repo_url                      : None
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  install_osd                   : False
[ceph_deploy.cli][INFO  ]  version_kind                  : stable
[ceph_deploy.cli][INFO  ]  install_common                : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  dev                           : master
[ceph_deploy.cli][INFO  ]  nogpgcheck                    : True
[ceph_deploy.cli][INFO  ]  local_mirror                  : None
[ceph_deploy.cli][INFO  ]  release                       : None
[ceph_deploy.cli][INFO  ]  install_mon                   : False
[ceph_deploy.cli][INFO  ]  gpg_url                       : None
[ceph_deploy.install][DEBUG ] Installing stable version mimic on cluster ceph hosts ceph-master01 ceph-master02 ceph-master03
[ceph_deploy.install][DEBUG ] Detecting platform for host ceph-master01 ...
[ceph-master01][DEBUG ] connection detected need for sudo
[ceph-master01][DEBUG ] connected to host: ceph-master01 
[ceph-master01][DEBUG ] detect platform information from remote host
[ceph-master01][DEBUG ] detect machine type
[ceph_deploy.install][INFO  ] Distro info: Ubuntu 18.04 bionic
[ceph-master01][INFO  ] installing Ceph on ceph-master01
[ceph-master01][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q update
[ceph-master01][DEBUG ] Hit:1 https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic InRelease
[ceph-master01][DEBUG ] Hit:2 https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-nautilus bionic InRelease
[ceph-master01][DEBUG ] Hit:3 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic InRelease
[ceph-master01][DEBUG ] Hit:4 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates InRelease
[ceph-master01][DEBUG ] Hit:5 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-backports InRelease
[ceph-master01][DEBUG ] Hit:6 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-security InRelease
[ceph-master01][DEBUG ] Reading package lists...
[ceph-master01][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install ca-certificates apt-transport-https
[ceph-master01][DEBUG ] Reading package lists...
[ceph-master01][DEBUG ] Building dependency tree...
[ceph-master01][DEBUG ] Reading state information...
[ceph-master01][DEBUG ] ca-certificates is already the newest version (20211016~18.04.1).
[ceph-master01][DEBUG ] apt-transport-https is already the newest version (1.6.14).
[ceph-master01][DEBUG ] 0 upgraded, 0 newly installed, 0 to remove and 202 not upgraded.
[ceph-master01][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q update
[ceph-master01][DEBUG ] Hit:1 https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic InRelease
[ceph-master01][DEBUG ] Hit:2 https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-nautilus bionic InRelease
[ceph-master01][DEBUG ] Hit:3 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic InRelease
[ceph-master01][DEBUG ] Hit:4 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates InRelease
[ceph-master01][DEBUG ] Hit:5 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-backports InRelease
[ceph-master01][DEBUG ] Hit:6 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-security InRelease
[ceph-master01][DEBUG ] Reading package lists...
[ceph-master01][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install ceph ceph-osd ceph-mds ceph-mon radosgw
[ceph-master01][DEBUG ] Reading package lists...
[ceph-master01][DEBUG ] Building dependency tree...
[ceph-master01][DEBUG ] Reading state information...
[ceph-master01][DEBUG ] ceph is already the newest version (16.2.10-1bionic).
[ceph-master01][DEBUG ] ceph-mds is already the newest version (16.2.10-1bionic).
[ceph-master01][DEBUG ] ceph-mon is already the newest version (16.2.10-1bionic).
[ceph-master01][DEBUG ] ceph-osd is already the newest version (16.2.10-1bionic).
[ceph-master01][DEBUG ] radosgw is already the newest version (16.2.10-1bionic).
[ceph-master01][DEBUG ] 0 upgraded, 0 newly installed, 0 to remove and 202 not upgraded.
[ceph-master01][INFO  ] Running command: sudo ceph --version
[ceph-master01][DEBUG ] ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable)
[ceph_deploy.install][DEBUG ] Detecting platform for host ceph-master02 ...
[ceph-master02][DEBUG ] connection detected need for sudo
[ceph-master02][DEBUG ] connected to host: ceph-master02 
[ceph-master02][DEBUG ] detect platform information from remote host
[ceph-master02][DEBUG ] detect machine type
[ceph_deploy.install][INFO  ] Distro info: Ubuntu 18.04 bionic
[ceph-master02][INFO  ] installing Ceph on ceph-master02
[ceph-master02][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q update
[ceph-master02][DEBUG ] Hit:1 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic InRelease
[ceph-master02][DEBUG ] Hit:2 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates InRelease
[ceph-master02][DEBUG ] Get:3 https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic InRelease [8,572 B]
[ceph-master02][DEBUG ] Hit:4 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-backports InRelease
[ceph-master02][DEBUG ] Get:5 https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-nautilus bionic InRelease [8,560 B]
[ceph-master02][DEBUG ] Hit:6 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-security InRelease
[ceph-master02][DEBUG ] Fetched 17.1 kB in 1s (13.1 kB/s)
[ceph-master02][DEBUG ] Reading package lists...
[ceph-master02][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install ca-certificates apt-transport-https
[ceph-master02][DEBUG ] Reading package lists...
[ceph-master02][DEBUG ] Building dependency tree...
[ceph-master02][DEBUG ] Reading state information...
[ceph-master02][DEBUG ] ca-certificates is already the newest version (20211016~18.04.1).
[ceph-master02][DEBUG ] apt-transport-https is already the newest version (1.6.14).
[ceph-master02][DEBUG ] 0 upgraded, 0 newly installed, 0 to remove and 202 not upgraded.
[ceph-master02][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q update
[ceph-master02][DEBUG ] Hit:1 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic InRelease
[ceph-master02][DEBUG ] Hit:2 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates InRelease
[ceph-master02][DEBUG ] Hit:3 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-backports InRelease
[ceph-master02][DEBUG ] Hit:4 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-security InRelease
[ceph-master02][DEBUG ] Get:5 https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic InRelease [8,572 B]
[ceph-master02][DEBUG ] Get:6 https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-nautilus bionic InRelease [8,560 B]
[ceph-master02][DEBUG ] Fetched 17.1 kB in 1s (12.5 kB/s)
[ceph-master02][DEBUG ] Reading package lists...
[ceph-master02][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install ceph ceph-osd ceph-mds ceph-mon radosgw
[ceph-master02][DEBUG ] Reading package lists...
[ceph-master02][DEBUG ] Building dependency tree...
[ceph-master02][DEBUG ] Reading state information...
[ceph-master02][DEBUG ] ceph is already the newest version (16.2.10-1bionic).
[ceph-master02][DEBUG ] ceph-mds is already the newest version (16.2.10-1bionic).
[ceph-master02][DEBUG ] ceph-mon is already the newest version (16.2.10-1bionic).
[ceph-master02][DEBUG ] ceph-osd is already the newest version (16.2.10-1bionic).
[ceph-master02][DEBUG ] radosgw is already the newest version (16.2.10-1bionic).
[ceph-master02][DEBUG ] 0 upgraded, 0 newly installed, 0 to remove and 202 not upgraded.
[ceph-master02][INFO  ] Running command: sudo ceph --version
[ceph-master02][DEBUG ] ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable)
[ceph_deploy.install][DEBUG ] Detecting platform for host ceph-master03 ...
[ceph-master03][DEBUG ] connection detected need for sudo
[ceph-master03][DEBUG ] connected to host: ceph-master03 
[ceph-master03][DEBUG ] detect platform information from remote host
[ceph-master03][DEBUG ] detect machine type
[ceph_deploy.install][INFO  ] Distro info: Ubuntu 18.04 bionic
[ceph-master03][INFO  ] installing Ceph on ceph-master03
[ceph-master03][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q update
[ceph-master03][DEBUG ] Hit:1 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic InRelease
[ceph-master03][DEBUG ] Hit:2 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates InRelease
[ceph-master03][DEBUG ] Hit:3 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-backports InRelease
[ceph-master03][DEBUG ] Hit:4 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-security InRelease
[ceph-master03][DEBUG ] Get:5 https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic InRelease [8,572 B]
[ceph-master03][DEBUG ] Get:6 https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-nautilus bionic InRelease [8,560 B]
[ceph-master03][DEBUG ] Fetched 17.1 kB in 2s (8,636 B/s)
[ceph-master03][DEBUG ] Reading package lists...
[ceph-master03][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install ca-certificates apt-transport-https
[ceph-master03][DEBUG ] Reading package lists...
[ceph-master03][DEBUG ] Building dependency tree...
[ceph-master03][DEBUG ] Reading state information...
[ceph-master03][DEBUG ] ca-certificates is already the newest version (20211016~18.04.1).
[ceph-master03][DEBUG ] apt-transport-https is already the newest version (1.6.14).
[ceph-master03][DEBUG ] 0 upgraded, 0 newly installed, 0 to remove and 202 not upgraded.
[ceph-master03][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q update
[ceph-master03][DEBUG ] Hit:1 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic InRelease
[ceph-master03][DEBUG ] Hit:2 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates InRelease
[ceph-master03][DEBUG ] Hit:3 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-backports InRelease
[ceph-master03][DEBUG ] Hit:4 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-security InRelease
[ceph-master03][DEBUG ] Get:5 https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic InRelease [8,572 B]
[ceph-master03][DEBUG ] Get:6 https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-nautilus bionic InRelease [8,560 B]
[ceph-master03][DEBUG ] Fetched 17.1 kB in 1s (14.3 kB/s)
[ceph-master03][DEBUG ] Reading package lists...
[ceph-master03][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install ceph ceph-osd ceph-mds ceph-mon radosgw
[ceph-master03][DEBUG ] Reading package lists...
[ceph-master03][DEBUG ] Building dependency tree...
[ceph-master03][DEBUG ] Reading state information...
[ceph-master03][DEBUG ] ceph is already the newest version (16.2.10-1bionic).
[ceph-master03][DEBUG ] ceph-mds is already the newest version (16.2.10-1bionic).
[ceph-master03][DEBUG ] ceph-mon is already the newest version (16.2.10-1bionic).
[ceph-master03][DEBUG ] ceph-osd is already the newest version (16.2.10-1bionic).
[ceph-master03][DEBUG ] radosgw is already the newest version (16.2.10-1bionic).
[ceph-master03][DEBUG ] 0 upgraded, 0 newly installed, 0 to remove and 202 not upgraded.
[ceph-master03][INFO  ] Running command: sudo ceph --version
[ceph-master03][DEBUG ] ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable)

此 过 程 会 在 指 定 的 ceph node 节 点 按 照 串 行 的 方 式 逐 个 服 务 器 安 装 ceph-base

ceph-common 等组件包:

在这里插入图片描述

2.3 安装ceph-mon服务
2.3.1 ceph-mon节点安装ceph-mon
cephadmin@ceph-master01:/etc/ceph-cluster# apt-cache madison ceph-monceph-mon | 16.2.10-1bionic | https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic/main amd64 Packagesceph-mon | 14.2.22-1bionic | https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-nautilus bionic/main amd64 Packagesceph-mon | 12.2.13-0ubuntu0.18.04.10 | http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates/main amd64 Packagesceph-mon | 12.2.13-0ubuntu0.18.04.10 | http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-security/main amd64 Packagesceph-mon | 12.2.4-0ubuntu1 | http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic/main amd64 Packages
cephadmin@ceph-master01:/etc/ceph-cluster$ 
root@ceph-master01:~# apt install ceph-mon
root@ceph-master02:~# apt install ceph-mon
root@ceph-master03:~# apt install ceph-mon

#可能已经安装完毕

image-20220905104407863.png

2.3.2 ceph 集群添加 ceph-mon 服务
cephadmin@ceph-master01:/etc/ceph-cluster# pwd
/etc/ceph-cluster
cephadmin@ceph-master01:/etc/ceph-cluster# cat ceph.conf 
[global]
fsid = f69afe6f-e559-4df7-998a-c5dc3e300209
public_network = 172.26.0.0/16
cluster_network = 10.0.0.0/24
mon_initial_members = ceph-master01, ceph-master02, ceph-master03
mon_host = 172.26.156.217,172.26.156.218,172.26.156.219     #通过配置文件将mon服务加入节点
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
cephadmin@ceph-master01:/etc/ceph-cluster# ceph-deploy mon create-initial
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadmin/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy mon create-initial
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create-initial
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fe450df12d0>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mon at 0x7fe450dcebd0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  keyrings                      : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-master01 ceph-master02 ceph-master03
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-master01 ...
[ceph-master01][DEBUG ] connection detected need for sudo
[ceph-master01][DEBUG ] connected to host: ceph-master01 
[ceph-master01][DEBUG ] detect platform information from remote host
[ceph-master01][DEBUG ] detect machine type
[ceph-master01][DEBUG ] find the location of an executable
[ceph_deploy.mon][INFO  ] distro info: Ubuntu 18.04 bionic
[ceph-master01][DEBUG ] determining if provided host has same hostname in remote
[ceph-master01][DEBUG ] get remote short hostname
[ceph-master01][DEBUG ] deploying mon to ceph-master01
[ceph-master01][DEBUG ] get remote short hostname
[ceph-master01][DEBUG ] remote hostname: ceph-master01
[ceph-master01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-master01][DEBUG ] create the mon path if it does not exist
[ceph-master01][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-ceph-master01/done
[ceph-master01][DEBUG ] done path does not exist: /var/lib/ceph/mon/ceph-ceph-master01/done
[ceph-master01][INFO  ] creating keyring file: /var/lib/ceph/tmp/ceph-ceph-master01.mon.keyring
[ceph-master01][DEBUG ] create the monitor keyring file
[ceph-master01][INFO  ] Running command: sudo ceph-mon --cluster ceph --mkfs -i ceph-master01 --keyring /var/lib/ceph/tmp/ceph-ceph-master01.mon.keyring --setuser 64045 --setgroup 64045
[ceph-master01][INFO  ] unlinking keyring file /var/lib/ceph/tmp/ceph-ceph-master01.mon.keyring
[ceph-master01][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-master01][DEBUG ] create the init path if it does not exist
[ceph-master01][INFO  ] Running command: sudo systemctl enable ceph.target
[ceph-master01][INFO  ] Running command: sudo systemctl enable ceph-mon@ceph-master01
[ceph-master01][WARNIN] Created symlink /etc/systemd/system/ceph-mon.target.wants/ceph-mon@ceph-master01.service → /lib/systemd/system/ceph-mon@.service.
[ceph-master01][INFO  ] Running command: sudo systemctl start ceph-mon@ceph-master01
[ceph-master01][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-master01.asok mon_status
[ceph-master01][DEBUG ] ********************************************************************************
[ceph-master01][DEBUG ] status for monitor: mon.ceph-master01
[ceph-master01][DEBUG ] {
[ceph-master01][DEBUG ]   "election_epoch": 0, 
[ceph-master01][DEBUG ]   "extra_probe_peers": [
[ceph-master01][DEBUG ]     {
[ceph-master01][DEBUG ]       "addrvec": [
[ceph-master01][DEBUG ]         {
[ceph-master01][DEBUG ]           "addr": "172.26.156.218:3300", 
[ceph-master01][DEBUG ]           "nonce": 0, 
[ceph-master01][DEBUG ]           "type": "v2"
[ceph-master01][DEBUG ]         }, 
[ceph-master01][DEBUG ]         {
[ceph-master01][DEBUG ]           "addr": "172.26.156.218:6789", 
[ceph-master01][DEBUG ]           "nonce": 0, 
[ceph-master01][DEBUG ]           "type": "v1"
[ceph-master01][DEBUG ]         }
[ceph-master01][DEBUG ]       ]
[ceph-master01][DEBUG ]     }, 
[ceph-master01][DEBUG ]     {
[ceph-master01][DEBUG ]       "addrvec": [
[ceph-master01][DEBUG ]         {
[ceph-master01][DEBUG ]           "addr": "172.26.156.219:3300", 
[ceph-master01][DEBUG ]           "nonce": 0, 
[ceph-master01][DEBUG ]           "type": "v2"
[ceph-master01][DEBUG ]         }, 
[ceph-master01][DEBUG ]         {
[ceph-master01][DEBUG ]           "addr": "172.26.156.219:6789", 
[ceph-master01][DEBUG ]           "nonce": 0, 
[ceph-master01][DEBUG ]           "type": "v1"
[ceph-master01][DEBUG ]         }
[ceph-master01][DEBUG ]       ]
[ceph-master01][DEBUG ]     }
[ceph-master01][DEBUG ]   ], 
[ceph-master01][DEBUG ]   "feature_map": {
[ceph-master01][DEBUG ]     "mon": [
[ceph-master01][DEBUG ]       {
[ceph-master01][DEBUG ]         "features": "0x3f01cfb9fffdffff", 
[ceph-master01][DEBUG ]         "num": 1, 
[ceph-master01][DEBUG ]         "release": "luminous"
[ceph-master01][DEBUG ]       }
[ceph-master01][DEBUG ]     ]
[ceph-master01][DEBUG ]   }, 
[ceph-master01][DEBUG ]   "features": {
[ceph-master01][DEBUG ]     "quorum_con": "0", 
[ceph-master01][DEBUG ]     "quorum_mon": [], 
[ceph-master01][DEBUG ]     "required_con": "0", 
[ceph-master01][DEBUG ]     "required_mon": []
[ceph-master01][DEBUG ]   }, 
[ceph-master01][DEBUG ]   "monmap": {
[ceph-master01][DEBUG ]     "created": "2022-09-05T02:52:15.915768Z", 
[ceph-master01][DEBUG ]     "disallowed_leaders: ": "", 
[ceph-master01][DEBUG ]     "election_strategy": 1, 
[ceph-master01][DEBUG ]     "epoch": 0, 
[ceph-master01][DEBUG ]     "features": {
[ceph-master01][DEBUG ]       "optional": [], 
[ceph-master01][DEBUG ]       "persistent": []
[ceph-master01][DEBUG ]     }, 
[ceph-master01][DEBUG ]     "fsid": "f69afe6f-e559-4df7-998a-c5dc3e300209", 
[ceph-master01][DEBUG ]     "min_mon_release": 0, 
[ceph-master01][DEBUG ]     "min_mon_release_name": "unknown", 
[ceph-master01][DEBUG ]     "modified": "2022-09-05T02:52:15.915768Z", 
[ceph-master01][DEBUG ]     "mons": [
[ceph-master01][DEBUG ]       {
[ceph-master01][DEBUG ]         "addr": "172.26.156.217:6789/0", 
[ceph-master01][DEBUG ]         "crush_location": "{}", 
[ceph-master01][DEBUG ]         "name": "ceph-master01", 
[ceph-master01][DEBUG ]         "priority": 0, 
[ceph-master01][DEBUG ]         "public_addr": "172.26.156.217:6789/0", 
[ceph-master01][DEBUG ]         "public_addrs": {
[ceph-master01][DEBUG ]           "addrvec": [
[ceph-master01][DEBUG ]             {
[ceph-master01][DEBUG ]               "addr": "172.26.156.217:3300", 
[ceph-master01][DEBUG ]               "nonce": 0, 
[ceph-master01][DEBUG ]               "type": "v2"
[ceph-master01][DEBUG ]             }, 
[ceph-master01][DEBUG ]             {
[ceph-master01][DEBUG ]               "addr": "172.26.156.217:6789", 
[ceph-master01][DEBUG ]               "nonce": 0, 
[ceph-master01][DEBUG ]               "type": "v1"
[ceph-master01][DEBUG ]             }
[ceph-master01][DEBUG ]           ]
[ceph-master01][DEBUG ]         }, 
[ceph-master01][DEBUG ]         "rank": 0, 
[ceph-master01][DEBUG ]         "weight": 0
[ceph-master01][DEBUG ]       }, 
[ceph-master01][DEBUG ]       {
[ceph-master01][DEBUG ]         "addr": "0.0.0.0:0/1", 
[ceph-master01][DEBUG ]         "crush_location": "{}", 
[ceph-master01][DEBUG ]         "name": "ceph-master02", 
[ceph-master01][DEBUG ]         "priority": 0, 
[ceph-master01][DEBUG ]         "public_addr": "0.0.0.0:0/1", 
[ceph-master01][DEBUG ]         "public_addrs": {
[ceph-master01][DEBUG ]           "addrvec": [
[ceph-master01][DEBUG ]             {
[ceph-master01][DEBUG ]               "addr": "0.0.0.0:0", 
[ceph-master01][DEBUG ]               "nonce": 1, 
[ceph-master01][DEBUG ]               "type": "v1"
[ceph-master01][DEBUG ]             }
[ceph-master01][DEBUG ]           ]
[ceph-master01][DEBUG ]         }, 
[ceph-master01][DEBUG ]         "rank": 1, 
[ceph-master01][DEBUG ]         "weight": 0
[ceph-master01][DEBUG ]       }, 
[ceph-master01][DEBUG ]       {
[ceph-master01][DEBUG ]         "addr": "0.0.0.0:0/2", 
[ceph-master01][DEBUG ]         "crush_location": "{}", 
[ceph-master01][DEBUG ]         "name": "ceph-master03", 
[ceph-master01][DEBUG ]         "priority": 0, 
[ceph-master01][DEBUG ]         "public_addr": "0.0.0.0:0/2", 
[ceph-master01][DEBUG ]         "public_addrs": {
[ceph-master01][DEBUG ]           "addrvec": [
[ceph-master01][DEBUG ]             {
[ceph-master01][DEBUG ]               "addr": "0.0.0.0:0", 
[ceph-master01][DEBUG ]               "nonce": 2, 
[ceph-master01][DEBUG ]               "type": "v1"
[ceph-master01][DEBUG ]             }
[ceph-master01][DEBUG ]           ]
[ceph-master01][DEBUG ]         }, 
[ceph-master01][DEBUG ]         "rank": 2, 
[ceph-master01][DEBUG ]         "weight": 0
[ceph-master01][DEBUG ]       }
[ceph-master01][DEBUG ]     ], 
[ceph-master01][DEBUG ]     "stretch_mode": false, 
[ceph-master01][DEBUG ]     "tiebreaker_mon": ""
[ceph-master01][DEBUG ]   }, 
[ceph-master01][DEBUG ]   "name": "ceph-master01", 
[ceph-master01][DEBUG ]   "outside_quorum": [
[ceph-master01][DEBUG ]     "ceph-master01"
[ceph-master01][DEBUG ]   ], 
[ceph-master01][DEBUG ]   "quorum": [], 
[ceph-master01][DEBUG ]   "rank": 0, 
[ceph-master01][DEBUG ]   "state": "probing", 
[ceph-master01][DEBUG ]   "stretch_mode": false, 
[ceph-master01][DEBUG ]   "sync_provider": []
[ceph-master01][DEBUG ] }
[ceph-master01][DEBUG ] ********************************************************************************
[ceph-master01][INFO  ] monitor: mon.ceph-master01 is running
[ceph-master01][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-master01.asok mon_status
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-master02 ...
[ceph-master02][DEBUG ] connection detected need for sudo
[ceph-master02][DEBUG ] connected to host: ceph-master02 
[ceph-master02][DEBUG ] detect platform information from remote host
[ceph-master02][DEBUG ] detect machine type
[ceph-master02][DEBUG ] find the location of an executable
[ceph_deploy.mon][INFO  ] distro info: Ubuntu 18.04 bionic
[ceph-master02][DEBUG ] determining if provided host has same hostname in remote
[ceph-master02][DEBUG ] get remote short hostname
[ceph-master02][DEBUG ] deploying mon to ceph-master02
[ceph-master02][DEBUG ] get remote short hostname
[ceph-master02][DEBUG ] remote hostname: ceph-master02
[ceph-master02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-master02][DEBUG ] create the mon path if it does not exist
[ceph-master02][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-ceph-master02/done
[ceph-master02][DEBUG ] done path does not exist: /var/lib/ceph/mon/ceph-ceph-master02/done
[ceph-master02][INFO  ] creating keyring file: /var/lib/ceph/tmp/ceph-ceph-master02.mon.keyring
[ceph-master02][DEBUG ] create the monitor keyring file
[ceph-master02][INFO  ] Running command: sudo ceph-mon --cluster ceph --mkfs -i ceph-master02 --keyring /var/lib/ceph/tmp/ceph-ceph-master02.mon.keyring --setuser 64045 --setgroup 64045
[ceph-master02][INFO  ] unlinking keyring file /var/lib/ceph/tmp/ceph-ceph-master02.mon.keyring
[ceph-master02][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-master02][DEBUG ] create the init path if it does not exist
[ceph-master02][INFO  ] Running command: sudo systemctl enable ceph.target
[ceph-master02][INFO  ] Running command: sudo systemctl enable ceph-mon@ceph-master02
[ceph-master02][WARNIN] Created symlink /etc/systemd/system/ceph-mon.target.wants/ceph-mon@ceph-master02.service → /lib/systemd/system/ceph-mon@.service.
[ceph-master02][INFO  ] Running command: sudo systemctl start ceph-mon@ceph-master02
[ceph-master02][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-master02.asok mon_status
[ceph-master02][DEBUG ] ********************************************************************************
[ceph-master02][DEBUG ] status for monitor: mon.ceph-master02
[ceph-master02][DEBUG ] {
[ceph-master02][DEBUG ]   "election_epoch": 1, 
[ceph-master02][DEBUG ]   "extra_probe_peers": [
[ceph-master02][DEBUG ]     {
[ceph-master02][DEBUG ]       "addrvec": [
[ceph-master02][DEBUG ]         {
[ceph-master02][DEBUG ]           "addr": "172.26.156.217:3300", 
[ceph-master02][DEBUG ]           "nonce": 0, 
[ceph-master02][DEBUG ]           "type": "v2"
[ceph-master02][DEBUG ]         }, 
[ceph-master02][DEBUG ]         {
[ceph-master02][DEBUG ]           "addr": "172.26.156.217:6789", 
[ceph-master02][DEBUG ]           "nonce": 0, 
[ceph-master02][DEBUG ]           "type": "v1"
[ceph-master02][DEBUG ]         }
[ceph-master02][DEBUG ]       ]
[ceph-master02][DEBUG ]     }, 
[ceph-master02][DEBUG ]     {
[ceph-master02][DEBUG ]       "addrvec": [
[ceph-master02][DEBUG ]         {
[ceph-master02][DEBUG ]           "addr": "172.26.156.219:3300", 
[ceph-master02][DEBUG ]           "nonce": 0, 
[ceph-master02][DEBUG ]           "type": "v2"
[ceph-master02][DEBUG ]         }, 
[ceph-master02][DEBUG ]         {
[ceph-master02][DEBUG ]           "addr": "172.26.156.219:6789", 
[ceph-master02][DEBUG ]           "nonce": 0, 
[ceph-master02][DEBUG ]           "type": "v1"
[ceph-master02][DEBUG ]         }
[ceph-master02][DEBUG ]       ]
[ceph-master02][DEBUG ]     }
[ceph-master02][DEBUG ]   ], 
[ceph-master02][DEBUG ]   "feature_map": {
[ceph-master02][DEBUG ]     "mon": [
[ceph-master02][DEBUG ]       {
[ceph-master02][DEBUG ]         "features": "0x3f01cfb9fffdffff", 
[ceph-master02][DEBUG ]         "num": 1, 
[ceph-master02][DEBUG ]         "release": "luminous"
[ceph-master02][DEBUG ]       }
[ceph-master02][DEBUG ]     ]
[ceph-master02][DEBUG ]   }, 
[ceph-master02][DEBUG ]   "features": {
[ceph-master02][DEBUG ]     "quorum_con": "0", 
[ceph-master02][DEBUG ]     "quorum_mon": [], 
[ceph-master02][DEBUG ]     "required_con": "0", 
[ceph-master02][DEBUG ]     "required_mon": []
[ceph-master02][DEBUG ]   }, 
[ceph-master02][DEBUG ]   "monmap": {
[ceph-master02][DEBUG ]     "created": "2022-09-05T02:52:20.691459Z", 
[ceph-master02][DEBUG ]     "disallowed_leaders: ": "", 
[ceph-master02][DEBUG ]     "election_strategy": 1, 
[ceph-master02][DEBUG ]     "epoch": 0, 
[ceph-master02][DEBUG ]     "features": {
[ceph-master02][DEBUG ]       "optional": [], 
[ceph-master02][DEBUG ]       "persistent": []
[ceph-master02][DEBUG ]     }, 
[ceph-master02][DEBUG ]     "fsid": "f69afe6f-e559-4df7-998a-c5dc3e300209", 
[ceph-master02][DEBUG ]     "min_mon_release": 0, 
[ceph-master02][DEBUG ]     "min_mon_release_name": "unknown", 
[ceph-master02][DEBUG ]     "modified": "2022-09-05T02:52:20.691459Z", 
[ceph-master02][DEBUG ]     "mons": [
[ceph-master02][DEBUG ]       {
[ceph-master02][DEBUG ]         "addr": "172.26.156.217:6789/0", 
[ceph-master02][DEBUG ]         "crush_location": "{}", 
[ceph-master02][DEBUG ]         "name": "ceph-master01", 
[ceph-master02][DEBUG ]         "priority": 0, 
[ceph-master02][DEBUG ]         "public_addr": "172.26.156.217:6789/0", 
[ceph-master02][DEBUG ]         "public_addrs": {
[ceph-master02][DEBUG ]           "addrvec": [
[ceph-master02][DEBUG ]             {
[ceph-master02][DEBUG ]               "addr": "172.26.156.217:3300", 
[ceph-master02][DEBUG ]               "nonce": 0, 
[ceph-master02][DEBUG ]               "type": "v2"
[ceph-master02][DEBUG ]             }, 
[ceph-master02][DEBUG ]             {
[ceph-master02][DEBUG ]               "addr": "172.26.156.217:6789", 
[ceph-master02][DEBUG ]               "nonce": 0, 
[ceph-master02][DEBUG ]               "type": "v1"
[ceph-master02][DEBUG ]             }
[ceph-master02][DEBUG ]           ]
[ceph-master02][DEBUG ]         }, 
[ceph-master02][DEBUG ]         "rank": 0, 
[ceph-master02][DEBUG ]         "weight": 0
[ceph-master02][DEBUG ]       }, 
[ceph-master02][DEBUG ]       {
[ceph-master02][DEBUG ]         "addr": "172.26.156.218:6789/0", 
[ceph-master02][DEBUG ]         "crush_location": "{}", 
[ceph-master02][DEBUG ]         "name": "ceph-master02", 
[ceph-master02][DEBUG ]         "priority": 0, 
[ceph-master02][DEBUG ]         "public_addr": "172.26.156.218:6789/0", 
[ceph-master02][DEBUG ]         "public_addrs": {
[ceph-master02][DEBUG ]           "addrvec": [
[ceph-master02][DEBUG ]             {
[ceph-master02][DEBUG ]               "addr": "172.26.156.218:3300", 
[ceph-master02][DEBUG ]               "nonce": 0, 
[ceph-master02][DEBUG ]               "type": "v2"
[ceph-master02][DEBUG ]             }, 
[ceph-master02][DEBUG ]             {
[ceph-master02][DEBUG ]               "addr": "172.26.156.218:6789", 
[ceph-master02][DEBUG ]               "nonce": 0, 
[ceph-master02][DEBUG ]               "type": "v1"
[ceph-master02][DEBUG ]             }
[ceph-master02][DEBUG ]           ]
[ceph-master02][DEBUG ]         }, 
[ceph-master02][DEBUG ]         "rank": 1, 
[ceph-master02][DEBUG ]         "weight": 0
[ceph-master02][DEBUG ]       }, 
[ceph-master02][DEBUG ]       {
[ceph-master02][DEBUG ]         "addr": "0.0.0.0:0/2", 
[ceph-master02][DEBUG ]         "crush_location": "{}", 
[ceph-master02][DEBUG ]         "name": "ceph-master03", 
[ceph-master02][DEBUG ]         "priority": 0, 
[ceph-master02][DEBUG ]         "public_addr": "0.0.0.0:0/2", 
[ceph-master02][DEBUG ]         "public_addrs": {
[ceph-master02][DEBUG ]           "addrvec": [
[ceph-master02][DEBUG ]             {
[ceph-master02][DEBUG ]               "addr": "0.0.0.0:0", 
[ceph-master02][DEBUG ]               "nonce": 2, 
[ceph-master02][DEBUG ]               "type": "v1"
[ceph-master02][DEBUG ]             }
[ceph-master02][DEBUG ]           ]
[ceph-master02][DEBUG ]         }, 
[ceph-master02][DEBUG ]         "rank": 2, 
[ceph-master02][DEBUG ]         "weight": 0
[ceph-master02][DEBUG ]       }
[ceph-master02][DEBUG ]     ], 
[ceph-master02][DEBUG ]     "stretch_mode": false, 
[ceph-master02][DEBUG ]     "tiebreaker_mon": ""
[ceph-master02][DEBUG ]   }, 
[ceph-master02][DEBUG ]   "name": "ceph-master02", 
[ceph-master02][DEBUG ]   "outside_quorum": [], 
[ceph-master02][DEBUG ]   "quorum": [], 
[ceph-master02][DEBUG ]   "rank": 1, 
[ceph-master02][DEBUG ]   "state": "electing", 
[ceph-master02][DEBUG ]   "stretch_mode": false, 
[ceph-master02][DEBUG ]   "sync_provider": []
[ceph-master02][DEBUG ] }
[ceph-master02][DEBUG ] ********************************************************************************
[ceph-master02][INFO  ] monitor: mon.ceph-master02 is running
[ceph-master02][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-master02.asok mon_status
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-master03 ...
[ceph-master03][DEBUG ] connection detected need for sudo
[ceph-master03][DEBUG ] connected to host: ceph-master03 
[ceph-master03][DEBUG ] detect platform information from remote host
[ceph-master03][DEBUG ] detect machine type
[ceph-master03][DEBUG ] find the location of an executable
[ceph_deploy.mon][INFO  ] distro info: Ubuntu 18.04 bionic
[ceph-master03][DEBUG ] determining if provided host has same hostname in remote
[ceph-master03][DEBUG ] get remote short hostname
[ceph-master03][DEBUG ] deploying mon to ceph-master03
[ceph-master03][DEBUG ] get remote short hostname
[ceph-master03][DEBUG ] remote hostname: ceph-master03
[ceph-master03][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-master03][DEBUG ] create the mon path if it does not exist
[ceph-master03][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-ceph-master03/done
[ceph-master03][DEBUG ] done path does not exist: /var/lib/ceph/mon/ceph-ceph-master03/done
[ceph-master03][INFO  ] creating keyring file: /var/lib/ceph/tmp/ceph-ceph-master03.mon.keyring
[ceph-master03][DEBUG ] create the monitor keyring file
[ceph-master03][INFO  ] Running command: sudo ceph-mon --cluster ceph --mkfs -i ceph-master03 --keyring /var/lib/ceph/tmp/ceph-ceph-master03.mon.keyring --setuser 64045 --setgroup 64045
[ceph-master03][INFO  ] unlinking keyring file /var/lib/ceph/tmp/ceph-ceph-master03.mon.keyring
[ceph-master03][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-master03][DEBUG ] create the init path if it does not exist
[ceph-master03][INFO  ] Running command: sudo systemctl enable ceph.target
[ceph-master03][INFO  ] Running command: sudo systemctl enable ceph-mon@ceph-master03
[ceph-master03][WARNIN] Created symlink /etc/systemd/system/ceph-mon.target.wants/ceph-mon@ceph-master03.service → /lib/systemd/system/ceph-mon@.service.
[ceph-master03][INFO  ] Running command: sudo systemctl start ceph-mon@ceph-master03
[ceph-master03][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-master03.asok mon_status
[ceph-master03][DEBUG ] ********************************************************************************
[ceph-master03][DEBUG ] status for monitor: mon.ceph-master03
[ceph-master03][DEBUG ] {
[ceph-master03][DEBUG ]   "election_epoch": 0, 
[ceph-master03][DEBUG ]   "extra_probe_peers": [
[ceph-master03][DEBUG ]     {
[ceph-master03][DEBUG ]       "addrvec": [
[ceph-master03][DEBUG ]         {
[ceph-master03][DEBUG ]           "addr": "172.26.156.217:3300", 
[ceph-master03][DEBUG ]           "nonce": 0, 
[ceph-master03][DEBUG ]           "type": "v2"
[ceph-master03][DEBUG ]         }, 
[ceph-master03][DEBUG ]         {
[ceph-master03][DEBUG ]           "addr": "172.26.156.217:6789", 
[ceph-master03][DEBUG ]           "nonce": 0, 
[ceph-master03][DEBUG ]           "type": "v1"
[ceph-master03][DEBUG ]         }
[ceph-master03][DEBUG ]       ]
[ceph-master03][DEBUG ]     }, 
[ceph-master03][DEBUG ]     {
[ceph-master03][DEBUG ]       "addrvec": [
[ceph-master03][DEBUG ]         {
[ceph-master03][DEBUG ]           "addr": "172.26.156.218:3300", 
[ceph-master03][DEBUG ]           "nonce": 0, 
[ceph-master03][DEBUG ]           "type": "v2"
[ceph-master03][DEBUG ]         }, 
[ceph-master03][DEBUG ]         {
[ceph-master03][DEBUG ]           "addr": "172.26.156.218:6789", 
[ceph-master03][DEBUG ]           "nonce": 0, 
[ceph-master03][DEBUG ]           "type": "v1"
[ceph-master03][DEBUG ]         }
[ceph-master03][DEBUG ]       ]
[ceph-master03][DEBUG ]     }
[ceph-master03][DEBUG ]   ], 
[ceph-master03][DEBUG ]   "feature_map": {
[ceph-master03][DEBUG ]     "mon": [
[ceph-master03][DEBUG ]       {
[ceph-master03][DEBUG ]         "features": "0x3f01cfb9fffdffff", 
[ceph-master03][DEBUG ]         "num": 1, 
[ceph-master03][DEBUG ]         "release": "luminous"
[ceph-master03][DEBUG ]       }
[ceph-master03][DEBUG ]     ]
[ceph-master03][DEBUG ]   }, 
[ceph-master03][DEBUG ]   "features": {
[ceph-master03][DEBUG ]     "quorum_con": "0", 
[ceph-master03][DEBUG ]     "quorum_mon": [], 
[ceph-master03][DEBUG ]     "required_con": "0", 
[ceph-master03][DEBUG ]     "required_mon": []
[ceph-master03][DEBUG ]   }, 
[ceph-master03][DEBUG ]   "monmap": {
[ceph-master03][DEBUG ]     "created": "2022-09-05T02:52:25.483539Z", 
[ceph-master03][DEBUG ]     "disallowed_leaders: ": "", 
[ceph-master03][DEBUG ]     "election_strategy": 1, 
[ceph-master03][DEBUG ]     "epoch": 0, 
[ceph-master03][DEBUG ]     "features": {
[ceph-master03][DEBUG ]       "optional": [], 
[ceph-master03][DEBUG ]       "persistent": []
[ceph-master03][DEBUG ]     }, 
[ceph-master03][DEBUG ]     "fsid": "f69afe6f-e559-4df7-998a-c5dc3e300209", 
[ceph-master03][DEBUG ]     "min_mon_release": 0, 
[ceph-master03][DEBUG ]     "min_mon_release_name": "unknown", 
[ceph-master03][DEBUG ]     "modified": "2022-09-05T02:52:25.483539Z", 
[ceph-master03][DEBUG ]     "mons": [
[ceph-master03][DEBUG ]       {
[ceph-master03][DEBUG ]         "addr": "172.26.156.219:6789/0", 
[ceph-master03][DEBUG ]         "crush_location": "{}", 
[ceph-master03][DEBUG ]         "name": "ceph-master03", 
[ceph-master03][DEBUG ]         "priority": 0, 
[ceph-master03][DEBUG ]         "public_addr": "172.26.156.219:6789/0", 
[ceph-master03][DEBUG ]         "public_addrs": {
[ceph-master03][DEBUG ]           "addrvec": [
[ceph-master03][DEBUG ]             {
[ceph-master03][DEBUG ]               "addr": "172.26.156.219:3300", 
[ceph-master03][DEBUG ]               "nonce": 0, 
[ceph-master03][DEBUG ]               "type": "v2"
[ceph-master03][DEBUG ]             }, 
[ceph-master03][DEBUG ]             {
[ceph-master03][DEBUG ]               "addr": "172.26.156.219:6789", 
[ceph-master03][DEBUG ]               "nonce": 0, 
[ceph-master03][DEBUG ]               "type": "v1"
[ceph-master03][DEBUG ]             }
[ceph-master03][DEBUG ]           ]
[ceph-master03][DEBUG ]         }, 
[ceph-master03][DEBUG ]         "rank": 0, 
[ceph-master03][DEBUG ]         "weight": 0
[ceph-master03][DEBUG ]       }, 
[ceph-master03][DEBUG ]       {
[ceph-master03][DEBUG ]         "addr": "0.0.0.0:0/1", 
[ceph-master03][DEBUG ]         "crush_location": "{}", 
[ceph-master03][DEBUG ]         "name": "ceph-master01", 
[ceph-master03][DEBUG ]         "priority": 0, 
[ceph-master03][DEBUG ]         "public_addr": "0.0.0.0:0/1", 
[ceph-master03][DEBUG ]         "public_addrs": {
[ceph-master03][DEBUG ]           "addrvec": [
[ceph-master03][DEBUG ]             {
[ceph-master03][DEBUG ]               "addr": "0.0.0.0:0", 
[ceph-master03][DEBUG ]               "nonce": 1, 
[ceph-master03][DEBUG ]               "type": "v1"
[ceph-master03][DEBUG ]             }
[ceph-master03][DEBUG ]           ]
[ceph-master03][DEBUG ]         }, 
[ceph-master03][DEBUG ]         "rank": 1, 
[ceph-master03][DEBUG ]         "weight": 0
[ceph-master03][DEBUG ]       }, 
[ceph-master03][DEBUG ]       {
[ceph-master03][DEBUG ]         "addr": "0.0.0.0:0/2", 
[ceph-master03][DEBUG ]         "crush_location": "{}", 
[ceph-master03][DEBUG ]         "name": "ceph-master02", 
[ceph-master03][DEBUG ]         "priority": 0, 
[ceph-master03][DEBUG ]         "public_addr": "0.0.0.0:0/2", 
[ceph-master03][DEBUG ]         "public_addrs": {
[ceph-master03][DEBUG ]           "addrvec": [
[ceph-master03][DEBUG ]             {
[ceph-master03][DEBUG ]               "addr": "0.0.0.0:0", 
[ceph-master03][DEBUG ]               "nonce": 2, 
[ceph-master03][DEBUG ]               "type": "v1"
[ceph-master03][DEBUG ]             }
[ceph-master03][DEBUG ]           ]
[ceph-master03][DEBUG ]         }, 
[ceph-master03][DEBUG ]         "rank": 2, 
[ceph-master03][DEBUG ]         "weight": 0
[ceph-master03][DEBUG ]       }
[ceph-master03][DEBUG ]     ], 
[ceph-master03][DEBUG ]     "stretch_mode": false, 
[ceph-master03][DEBUG ]     "tiebreaker_mon": ""
[ceph-master03][DEBUG ]   }, 
[ceph-master03][DEBUG ]   "name": "ceph-master03", 
[ceph-master03][DEBUG ]   "outside_quorum": [
[ceph-master03][DEBUG ]     "ceph-master03"
[ceph-master03][DEBUG ]   ], 
[ceph-master03][DEBUG ]   "quorum": [], 
[ceph-master03][DEBUG ]   "rank": 0, 
[ceph-master03][DEBUG ]   "state": "probing", 
[ceph-master03][DEBUG ]   "stretch_mode": false, 
[ceph-master03][DEBUG ]   "sync_provider": []
[ceph-master03][DEBUG ] }
[ceph-master03][DEBUG ] ********************************************************************************
[ceph-master03][INFO  ] monitor: mon.ceph-master03 is running
[ceph-master03][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-master03.asok mon_status
[ceph_deploy.mon][INFO  ] processing monitor mon.ceph-master01
[ceph-master01][DEBUG ] connection detected need for sudo
[ceph-master01][DEBUG ] connected to host: ceph-master01 
[ceph-master01][DEBUG ] detect platform information from remote host
[ceph-master01][DEBUG ] detect machine type
[ceph-master01][DEBUG ] find the location of an executable
[ceph-master01][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-master01.asok mon_status
[ceph_deploy.mon][WARNIN] mon.ceph-master01 monitor is not yet in quorum, tries left: 5
[ceph_deploy.mon][WARNIN] waiting 5 seconds before retrying
[ceph-master01][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-master01.asok mon_status
[ceph_deploy.mon][WARNIN] mon.ceph-master01 monitor is not yet in quorum, tries left: 4
[ceph_deploy.mon][WARNIN] waiting 10 seconds before retrying
[ceph-master01][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-master01.asok mon_status
[ceph_deploy.mon][INFO  ] mon.ceph-master01 monitor has reached quorum!
[ceph_deploy.mon][INFO  ] processing monitor mon.ceph-master02
[ceph-master02][DEBUG ] connection detected need for sudo
[ceph-master02][DEBUG ] connected to host: ceph-master02 
[ceph-master02][DEBUG ] detect platform information from remote host
[ceph-master02][DEBUG ] detect machine type
[ceph-master02][DEBUG ] find the location of an executable
[ceph-master02][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-master02.asok mon_status
[ceph_deploy.mon][INFO  ] mon.ceph-master02 monitor has reached quorum!
[ceph_deploy.mon][INFO  ] processing monitor mon.ceph-master03
[ceph-master03][DEBUG ] connection detected need for sudo
[ceph-master03][DEBUG ] connected to host: ceph-master03 
[ceph-master03][DEBUG ] detect platform information from remote host
[ceph-master03][DEBUG ] detect machine type
[ceph-master03][DEBUG ] find the location of an executable
[ceph-master03][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-master03.asok mon_status
[ceph_deploy.mon][INFO  ] mon.ceph-master03 monitor has reached quorum!
[ceph_deploy.mon][INFO  ] all initial monitors are running and have formed quorum
[ceph_deploy.mon][INFO  ] Running gatherkeys...
[ceph_deploy.gatherkeys][INFO  ] Storing keys in temp directory /tmp/tmpP6crY0
[ceph-master01][DEBUG ] connection detected need for sudo
[ceph-master01][DEBUG ] connected to host: ceph-master01 
[ceph-master01][DEBUG ] detect platform information from remote host
[ceph-master01][DEBUG ] detect machine type
[ceph-master01][DEBUG ] get remote short hostname
[ceph-master01][DEBUG ] fetch remote file
[ceph-master01][INFO  ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --admin-daemon=/var/run/ceph/ceph-mon.ceph-master01.asok mon_status
[ceph-master01][INFO  ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-master01/keyring auth get client.admin
[ceph-master01][INFO  ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-master01/keyring auth get client.bootstrap-mds
[ceph-master01][INFO  ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-master01/keyring auth get client.bootstrap-mgr
[ceph-master01][INFO  ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-master01/keyring auth get client.bootstrap-osd
[ceph-master01][INFO  ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-master01/keyring auth get client.bootstrap-rgw
[ceph_deploy.gatherkeys][INFO  ] Storing ceph.client.admin.keyring
[ceph_deploy.gatherkeys][INFO  ] Storing ceph.bootstrap-mds.keyring
[ceph_deploy.gatherkeys][INFO  ] Storing ceph.bootstrap-mgr.keyring
[ceph_deploy.gatherkeys][INFO  ] keyring 'ceph.mon.keyring' already exists
[ceph_deploy.gatherkeys][INFO  ] Storing ceph.bootstrap-osd.keyring
[ceph_deploy.gatherkeys][INFO  ] Storing ceph.bootstrap-rgw.keyring
[ceph_deploy.gatherkeys][INFO  ] Destroy temp directory /tmp/tmpP6crY0
2.3.2 验证mon节点

验证在 mon 定节点已经自动安装并启动了 ceph-mon 服务,ceph-mon服务的作用之一就是验证权限,会在ceph-deploy 节点初始化目录会生成 ceph.bootstrap-mds/mgr/osd/rgw 服务的 keyring 认证文件,这

些初始化文件拥有对 ceph 集群的最高权限,所以一定要保存好,后续需要发送给各个服务节点。

cephadmin@ceph-master01:/etc/ceph-cluster# ps -ef | grep ceph-mon
ceph       28179       1  0 10:52 ?        00:00:05 /usr/bin/ceph-mon -f --cluster ceph --id ceph-master01 --setuser ceph --setgroup ceph
cephadm+   28519   28038  0 11:10 pts/0    00:00:00 grep --color=auto ceph-mon
cephadmin@ceph-master01:/etc/ceph-cluster# systemctl  status ceph-mon.target 
● ceph-mon.target - ceph target allowing to start/stop all ceph-mon@.service instances at onceLoaded: loaded (/lib/systemd/system/ceph-mon.target; enabled; vendor preset: enabled)Active: active since Mon 2022-09-05 09:46:11 CST; 1h 24min ago
cephadmin@ceph-master01:/etc/ceph-cluster# ll  
total 248
drwxr-xr-x  2 cephadmin cephadmin   4096 Sep  5 10:52 ./
drwxr-xr-x 92 root      root        4096 Sep  5 09:46 ../
-rw-------  1 cephadmin cephadmin    113 Sep  5 10:52 ceph.bootstrap-mds.keyring
-rw-------  1 cephadmin cephadmin    113 Sep  5 10:52 ceph.bootstrap-mgr.keyring
-rw-------  1 cephadmin cephadmin    113 Sep  5 10:52 ceph.bootstrap-osd.keyring
-rw-------  1 cephadmin cephadmin    113 Sep  5 10:52 ceph.bootstrap-rgw.keyring
-rw-------  1 cephadmin cephadmin    151 Sep  5 10:52 ceph.client.admin.keyring
-rw-rw-r--  1 cephadmin cephadmin    326 Sep  2 16:50 ceph.conf
-rw-rw-r--  1 cephadmin cephadmin 209993 Sep  5 10:52 ceph-deploy-ceph.log
-rw-------  1 cephadmin cephadmin     73 Sep  2 16:50 ceph.mon.keyring

在这里插入图片描述

执行ceph -s 发现有健康告警

在这里插入图片描述

在其中一个mon节点执行:

ceph config set mon auth_allow_insecure_global_id_reclaim false
2.4 分发admin 秘钥

在 ceph-deploy 节点把配置文件和 admin 密钥拷贝至 Ceph 集群需要执行 ceph 管理命令的节点,从而不需要后期通过 ceph 命令对 ceph 集群进行管理配置的时候每次都需要指定ceph-mon 节点地址和 ceph.client.admin.keyring 文件,另外各 ceph-mon 节点也需要同步ceph 的集群配置文件与认证文件。

cephadmin@ceph-master01:~# sudo apt install ceph-common -y #node 节点在初始化时已经安装

发送admin密钥到deploy节点,默认分发到/etc/ceph/下, ceph.client.admin.keyring只需要存放在要执行ceph客户端命令下即可,同k8s kubeconfig文件,传到日常管理的ceph-deploy下

cephadmin@ceph-master01:/etc/ceph-cluster# ceph-deploy admin ceph-master01 

在这里插入图片描述

一般情况下ceph.client.admin.keyring文件的权限为600,属主和属组为root,如果在集群内节点使用cephadmin用户直接直接ceph命令,将会提示无法找到/etc/ceph/ceph.client.admin.keyring文件,因为权限不足

cephadmin@ceph-master01:~# sudo setfacl -m u:cephadmin:rw /etc/ceph/ceph.client.admin.keyring
cephadmin@ceph-master02:~# sudo setfacl -m u:cephadmin:rw /etc/ceph/ceph.client.admin.keyring
cephadmin@ceph-master03:~# sudo setfacl -m u:cephadmin:rw /etc/ceph/ceph.client.admin.keyring
2.5 部署manager

ceph 的 Luminious(12) 及以上版本有 manager 节点,早期的版本没有。

2.5.1 部署 ceph-mgr 节点

因为此节点是monitor节点,所有的ceph包已经安装了,如果mgr节点与monitor节点不是一台服务器就会安装

cephadmin@ceph-master01:~# sudo apt install ceph-mgr
Reading package lists... Done
Building dependency tree       
Reading state information... Done
ceph-mgr is already the newest version (16.2.10-1bionic).
0 upgraded, 0 newly installed, 0 to remove and 202 not upgraded.cephadmin@ceph-master02:~# sudo apt install ceph-mgr
Reading package lists... Done
Building dependency tree       
Reading state information... Done
ceph-mgr is already the newest version (16.2.10-1bionic).
0 upgraded, 0 newly installed, 0 to remove and 202 not upgraded.cephadmin@ceph-master03:~# sudo apt install ceph-mgr
Reading package lists... Done
Building dependency tree       
Reading state information... Done
ceph-mgr is already the newest version (16.2.10-1bionic).
0 upgraded, 0 newly installed, 0 to remove and 202 not upgraded.

创建mgr节点

cephadmin@ceph-master01:/etc/ceph-cluster# ceph-deploy mgr create ceph-master01 ceph-master02 ceph-master03
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadmin/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy mgr create ceph-master01 ceph-master02 ceph-master03
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  mgr                           : [('ceph-master01', 'ceph-master01'), ('ceph-master02', 'ceph-master02'), ('ceph-master03', 'ceph-master03')]
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f97e641fe60>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mgr at 0x7f97e687f250>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mgr][DEBUG ] Deploying mgr, cluster ceph hosts ceph-master01:ceph-master01 ceph-master02:ceph-master02 ceph-master03:ceph-master03
[ceph-master01][DEBUG ] connection detected need for sudo
[ceph-master01][DEBUG ] connected to host: ceph-master01 
[ceph-master01][DEBUG ] detect platform information from remote host
[ceph-master01][DEBUG ] detect machine type
[ceph_deploy.mgr][INFO  ] Distro info: Ubuntu 18.04 bionic
[ceph_deploy.mgr][DEBUG ] remote host will use systemd
[ceph_deploy.mgr][DEBUG ] deploying mgr bootstrap to ceph-master01
[ceph-master01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-master01][WARNIN] mgr keyring does not exist yet, creating one
[ceph-master01][DEBUG ] create a keyring file
[ceph-master01][DEBUG ] create path recursively if it doesn't exist
[ceph-master01][INFO  ] Running command: sudo ceph --cluster ceph --name client.bootstrap-mgr --keyring /var/lib/ceph/bootstrap-mgr/ceph.keyring auth get-or-create mgr.ceph-master01 mon allow profile mgr osd allow * mds allow * -o /var/lib/ceph/mgr/ceph-ceph-master01/keyring
[ceph-master01][INFO  ] Running command: sudo systemctl enable ceph-mgr@ceph-master01
[ceph-master01][WARNIN] Created symlink /etc/systemd/system/ceph-mgr.target.wants/ceph-mgr@ceph-master01.service → /lib/systemd/system/ceph-mgr@.service.
[ceph-master01][INFO  ] Running command: sudo systemctl start ceph-mgr@ceph-master01
[ceph-master01][INFO  ] Running command: sudo systemctl enable ceph.target
[ceph-master02][DEBUG ] connection detected need for sudo
[ceph-master02][DEBUG ] connected to host: ceph-master02 
[ceph-master02][DEBUG ] detect platform information from remote host
[ceph-master02][DEBUG ] detect machine type
[ceph_deploy.mgr][INFO  ] Distro info: Ubuntu 18.04 bionic
[ceph_deploy.mgr][DEBUG ] remote host will use systemd
[ceph_deploy.mgr][DEBUG ] deploying mgr bootstrap to ceph-master02
[ceph-master02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-master02][WARNIN] mgr keyring does not exist yet, creating one
[ceph-master02][DEBUG ] create a keyring file
[ceph-master02][DEBUG ] create path recursively if it doesn't exist
[ceph-master02][INFO  ] Running command: sudo ceph --cluster ceph --name client.bootstrap-mgr --keyring /var/lib/ceph/bootstrap-mgr/ceph.keyring auth get-or-create mgr.ceph-master02 mon allow profile mgr osd allow * mds allow * -o /var/lib/ceph/mgr/ceph-ceph-master02/keyring
[ceph-master02][INFO  ] Running command: sudo systemctl enable ceph-mgr@ceph-master02
[ceph-master02][WARNIN] Created symlink /etc/systemd/system/ceph-mgr.target.wants/ceph-mgr@ceph-master02.service → /lib/systemd/system/ceph-mgr@.service.
[ceph-master02][INFO  ] Running command: sudo systemctl start ceph-mgr@ceph-master02
[ceph-master02][INFO  ] Running command: sudo systemctl enable ceph.target
[ceph-master03][DEBUG ] connection detected need for sudo
[ceph-master03][DEBUG ] connected to host: ceph-master03 
[ceph-master03][DEBUG ] detect platform information from remote host
[ceph-master03][DEBUG ] detect machine type
[ceph_deploy.mgr][INFO  ] Distro info: Ubuntu 18.04 bionic
[ceph_deploy.mgr][DEBUG ] remote host will use systemd
[ceph_deploy.mgr][DEBUG ] deploying mgr bootstrap to ceph-master03
[ceph-master03][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-master03][WARNIN] mgr keyring does not exist yet, creating one
[ceph-master03][DEBUG ] create a keyring file
[ceph-master03][DEBUG ] create path recursively if it doesn't exist
[ceph-master03][INFO  ] Running command: sudo ceph --cluster ceph --name client.bootstrap-mgr --keyring /var/lib/ceph/bootstrap-mgr/ceph.keyring auth get-or-create mgr.ceph-master03 mon allow profile mgr osd allow * mds allow * -o /var/lib/ceph/mgr/ceph-ceph-master03/keyring
[ceph-master03][INFO  ] Running command: sudo systemctl enable ceph-mgr@ceph-master03
[ceph-master03][WARNIN] Created symlink /etc/systemd/system/ceph-mgr.target.wants/ceph-mgr@ceph-master03.service → /lib/systemd/system/ceph-mgr@.service.
[ceph-master03][INFO  ] Running command: sudo systemctl start ceph-mgr@ceph-master03
[ceph-master03][INFO  ] Running command: sudo systemctl enable ceph.target
2.5.2 验证ceph-mgr节点
cephadmin@ceph-master01:/etc/ceph-cluster# ps -ef | grep ceph-mgr
cephadmin@ceph-master01:/etc/ceph-cluster# systemctl  status  ceph-mgr@ceph-master01cephadmin@ceph-master02:/etc/ceph-cluster# ps -ef | grep ceph-mgr
cephadmin@ceph-master02:/etc/ceph-cluster# systemctl  status  ceph-mgr@ceph-master02cephadmin@ceph-master03:/etc/ceph-cluster# ps -ef | grep ceph-mgr
cephadmin@ceph-master03:/etc/ceph-cluster# systemctl  status  ceph-mgr@ceph-master03

在这里插入图片描述

2.6 部署osd
2.6.1 初始化存储节点

deploy节点操作,安装指定版本的ceph包,本文这里由于node节点与master节点部署在一起,已经安装过了,新node节点接入可以执行

cephadmin@ceph-master01:~# ceph-deploy install --release pacific  ceph-master01 ceph-master02 ceph-master03

在这里插入图片描述

列出 ceph node 节点各个磁盘:

cephadmin@ceph-master01:~# ceph-deploy disk list ceph-master01  ceph-master02 ceph-master03
#也可以使用fdisk -l 查看node节点所有未分区使用的磁盘

在这里插入图片描述

使用 ceph-deploy disk zap 擦除各 ceph node 的 ceph 数据磁盘

ceph-master01 ceph-master02 ceph-master03的存储节点磁盘擦除过程如下,可以反复擦除执行

ceph-deploy disk zap ceph-master01 /dev/sdb
ceph-deploy disk zap ceph-master01 /dev/sdc
ceph-deploy disk zap ceph-master01 /dev/sdd
ceph-deploy disk zap ceph-master02 /dev/sdb
ceph-deploy disk zap ceph-master02 /dev/sdc
ceph-deploy disk zap ceph-master02 /dev/sdd
ceph-deploy disk zap ceph-master03 /dev/sdb
ceph-deploy disk zap ceph-master03 /dev/sdc
ceph-deploy disk zap ceph-master03 /dev/sdd

在这里插入图片描述

2.6.2 OSD与磁盘部署关系

在这里插入图片描述

#服务器上有两块ssd盘时,可以分别把block-db,block-wal存放到ssd盘中
ceph-deploy osd create {node} --data /dev/sdc --block-db /dev/sda --block-wal /dev/sdb
#服务器上只有一块硬盘时,只指定db的话存放ssd盘,没有指定waf存放位置,waf也会自动写到更快速的ssd盘上,和db共用
ceph-deploy osd create {node} --data /path/to/data --block-db /dev/sda 
#第三种无意义
ceph-deploy osd create {node} --data /path/to/data --block-wal /dev/sda 

这里采用最简单的第一种方案 单块磁盘,高性能的ceph集群可以使用第二种方案,ssd存放元数据与waf日志

2.6.3 添加OSD
cephadmin@ceph-master01:/etc/ceph-cluster# ceph-deploy osd create ceph-master01 --data /dev/sdb
cephadmin@ceph-master01:/etc/ceph-cluster# ceph-deploy osd create ceph-master01 --data /dev/sdc
cephadmin@ceph-master01:/etc/ceph-cluster# ceph-deploy osd create ceph-master01 --data /dev/sdd
cephadmin@ceph-master01:/etc/ceph-cluster# ceph-deploy osd create ceph-master02 --data /dev/sdb
cephadmin@ceph-master01:/etc/ceph-cluster# ceph-deploy osd create ceph-master02 --data /dev/sdc
cephadmin@ceph-master01:/etc/ceph-cluster# ceph-deploy osd create ceph-master02 --data /dev/sdd
cephadmin@ceph-master01:/etc/ceph-cluster# ceph-deploy osd create ceph-master03 --data /dev/sdb
cephadmin@ceph-master01:/etc/ceph-cluster# ceph-deploy osd create ceph-master03 --data /dev/sdc
cephadmin@ceph-master01:/etc/ceph-cluster# ceph-deploy osd create ceph-master03 --data /dev/sdd

在这里插入图片描述

2.6.4 验证ceph集群
cephadmin@ceph-master01:/etc/ceph-cluster# ceph -s cluster:id:     f69afe6f-e559-4df7-998a-c5dc3e300209health: HEALTH_OKservices:mon: 3 daemons, quorum ceph-master01,ceph-master02,ceph-master03 (age 31m)mgr: ceph-master03(active, since 27h), standbys: ceph-master01, ceph-master02osd: 9 osds: 9 up (since 27h), 9 in (since 28h)data:pools:   2 pools, 33 pgsobjects: 1 objects, 100 MiBusage:   370 MiB used, 450 GiB / 450 GiB availpgs:     33 active+clean
2.7 测试上传与下载数据
存取数据时,客户端必须首先连接至 RADOS 集群上某存储池,然后根据对象名称由相关的CRUSH 规则完成数据对象寻址。于是,为了测试集群的数据存取功能,这里首先创建一个用于测试的存储池 mypool,并设定其 PG 数量为 32 个。
$ ceph -h #一个更底层的客户端命令 
$ rados -h #客户端命令
创建 pool:
cephadmin@ceph-master01:~# ceph osd pool create mypool 32 32 
pool 'mypool' createdcephadmin@ceph-master01:/etc/ceph-cluster# sudo ceph osd pool ls  
device_health_metrics
mypool
或者: 
cephadmin@ceph-master01:/etc/ceph-cluster# rados lspools mypool
device_health_metrics
mypool
或者:
cephadmin@ceph-master01:/etc/ceph-cluster# ceph osd lspools
1 device_health_metrics
2 mypool
上传数据

当前的 ceph 环境还没还没有部署使用块存储和文件系统使用 ceph,也没有使用对象存储的客户端,但是 ceph 的 rados 命令可以实现访问 ceph 对象存储的功能:

cephadmin@ceph-master01:~# sudo rados put msg1 /var/log/syslog  --pool=mypool
列出数据
cephadmin@ceph-master01:/etc/ceph-cluster# rados ls --pool=mypool
msg1
文件信息
cephadmin@ceph-master01:/etc/ceph-cluster# ceph osd map mypool msg1 
osdmap e114 pool 'mypool' (2) object 'msg1' -> pg 2.c833d430 (2.10) -> up ([15,13,0], p15) acting ([15,13,0], p15)

表示文件放在了存储池 id 为 2 的 c833d430 的 PG 上,10 为当前 PG 的 id, 2.10 表示数据是在 id 为 2 的存储池当中 id 为 10 的 PG 中存储,在线的 OSD 编号 15,13,10,主 OSD 为 5,活动的 OSD 15,13,10,三个 OSD 表示数据放一共 3 个副本,PG 中的 OSD 是 ceph 的 crush算法计算出三份数据保存在哪些 OSD。

下载文件
cephadmin@ceph-master01:/etc/ceph-cluster# sudo rados get msg1 --pool=mypool /opt/my.txt 
cephadmin@ceph-master01:/etc/ceph-cluster# ll /opt/my.txt 
-rw-r--r-- 1 root root 155733 Sep  7 20:51 /opt/my.txt
cephadmin@ceph-master01:/etc/ceph-cluster# head  /opt/my.txt
Sep  7 06:25:06 ceph-master01 rsyslogd:  [origin software="rsyslogd" swVersion="8.32.0" x-pid="998" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Sep  7 06:26:01 ceph-master01 CRON[10792]: (root) CMD (ntpdate  ntp.aliyun.com)
Sep  7 06:26:01 ceph-master01 CRON[10791]: (CRON) info (No MTA installed, discarding output)
Sep  7 06:27:01 ceph-master01 CRON[10794]: (root) CMD (ntpdate  ntp.aliyun.com)
Sep  7 06:27:01 ceph-master01 CRON[10793]: (CRON) info (No MTA installed, discarding output)
Sep  7 06:28:01 ceph-master01 CRON[10797]: (root) CMD (ntpdate  ntp.aliyun.com)
Sep  7 06:28:01 ceph-master01 CRON[10796]: (CRON) info (No MTA installed, discarding output)
Sep  7 06:29:01 ceph-master01 CRON[10799]: (root) CMD (ntpdate  ntp.aliyun.com)
Sep  7 06:29:01 ceph-master01 CRON[10798]: (CRON) info (No MTA installed, discarding output)
Sep  7 06:30:01 ceph-master01 CRON[10801]: (root) CMD (ntpdate  ntp.aliyun.com)
修改文件

修改文件只能下载后修改再上传覆盖

cephadmin@ceph-master01:/etc/ceph-cluster# sudo rados put msg1 /etc/passwd --pool=mypoo
删除文件
cephadmin@ceph-master01:/etc/ceph-cluster# sudo rados rm msg1 --pool=mypool
cephadmin@ceph-master01:/etc/ceph-cluster# rados ls --pool=mypool

3. Ceph RBD 使用详解

3.1 RBD架构图

Ceph 可以同时提供 RADOSGW(对象存储网关)、RBD(块存储)、Ceph FS(文件系统存储), RBD 即 RADOS Block Device 的简称,RBD 块存储是常用的存储类型之一,RBD 块设备类似磁盘可以被挂载,RBD 块设备具有快照、多副本、克隆和一致性等特性,数据以条带化的方式存储在 Ceph 集群的多个 OSD 中。

条带化技术就是一种自动的将 I/O 的负载均衡到多个物理磁盘上的技术,条带化技术就是 将一块连续的数据分成很多小部分并把他们分别存储到不同磁盘上去。这就能使多个进程同 时访问数据的多个不同部分而不会造成磁盘冲突,而且在需要对这种数据进行顺序访问的时 候可以获得最大程度上的 I/O 并行能力,从而获得非常好的性能。

在这里插入图片描述

3.2 创建存储池
#创建存储池
root@ceph-master01:~# ceph osd pool create rbd-data1 32 32
pool 'rbd-data1' created
#存储池启用 rbd
root@ceph-master01:~# ceph osd pool application enable rbd-data1 rbd 
enabled application 'rbd' on pool 'rbd-data1'
#初始化 rbd
root@ceph-master01:~# rbd pool init -p rbd-data1
3.3 创建img镜像

rbd 存储池并不能直接用于块设备,而是需要事先在其中按需创建映像(image),并把映像文件作为块设备使用。rbd 命令可用于创建、查看及删除块设备相在的映像(image),以及克隆映像、创建快照、将映像回滚到快照和查看快照等管理操作。例如,下面的命令能 够在指定的 RBD 即 rbd-data1 创建一个名为 myimg1 的映像.

3.3.1 创建镜像
root@ceph-master01:~# rbd create data-img1 --size 3G --pool rbd-data1 --image-format 2 --image-feature layering
#列出镜像
root@ceph-master01:~# rbd ls --pool rbd-data1 -l
NAME       SIZE   PARENT  FMT  PROT  LOCK
data-img1  3 GiB            2            
3.3.2 列出镜像详细信息
root@ceph-master01:~# rbd --image data-img1 --pool rbd-data1 info 
rbd image 'data-img1':size 3 GiB in 768 objects   order 22 (4 MiB objects)  #3G 768个objects,每个objects为4M snapshot_count: 0  id: 284d64e8f879d  # 镜像idblock_name_prefix: rbd_data.284d64e8f879dformat: 2  features: layering  #镜像特性op_features: flags: create_timestamp: Fri Sep 16 20:34:47 2022  access_timestamp: Fri Sep 16 20:34:47 2022modify_timestamp: Fri Sep 16 20:34:47 2022#已json显示详细信息
root@ceph-master01:~# rbd ls --pool rbd-data1 -l --format json --pretty-format
[{"image": "data-img1","id": "284d64e8f879d","size": 3221225472,"format": 2}
]
3.3.3 :镜像的特性

RBD默认开启的特性包括: layering/exlcusive lock/object map/fast diff/deep flatten

#启用指定存储池中的指定镜像的特性
$ rbd feature enable exclusive-lock --pool rbd-data1 --image data-img1 
$ rbd feature enable object-map --pool rbd-data1 --image data-img1 
$ rbd feature enable fast-diff --pool rbd-data1 --image data-img1
#关闭指定存储池中的指定镜像的特性
$ rbd feature disable fast-diff --pool rbd-data1 --image data-img1
#验证镜像特性
$ rbd --image data-img1 --pool rbd-data1 info
3.4 客户端使用RBD

客户端使用RBD需要两个条件:

一.安装ceph 客户端组件 ceph-common

二.ceph用户

3.4.1 客户端安装 ceph-common

客户端要想挂载使用 ceph RBD,需要安装 ceph 客户端组件 ceph-common,但是 ceph-common 不在 cenos 的 yum 仓库,因此需要单独配置 yum 源,并且centos只能安装最高的版本为Octopus版(15版本)

#配置 yum 源: 
$ yum install epel-release 
$ yum install https://mirrors.aliyun.com/ceph/rpm-octopus/el7/noarch/ceph-release-1-1.el7.noarch.rpm -y
#下载ceph-common
$ yum install -y ceph-common
#验证ceph-common
[root@zd_spring_156_101 ~]# rpm -qa | grep ceph-common
python3-ceph-common-15.2.17-0.el7.x86_64
ceph-common-15.2.17-0.el7.x86_64
3.4.2 同步账户认证文件

#scp至客户端服务器的/etc/ceph目录下,客户端默认会读取

[cephadmin@ceph-deploy ceph-cluster]$ scp ceph.conf ceph.client.admin.keyring root@172.26.156.17:/etc/ceph/
3.4.3 客户端映射镜像
#映射rbd 
[root@xianchaonode1 ~]# rbd -p rbd-data1 map data-img1 
/dev/rbd0#客户端验证映射镜像
[root@xianchaonode1 ~]# lsblk  
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
rbd0   253:0    0     3G  0 disk 
sr0     11:0    1   4.2G  0 rom  
sda      8:0    0   200G  0 disk 
├─sda2   8:2    0 199.8G  0 part /
└─sda1   8:1    0   200M  0 part /boot
3.4.4 客户端挂载使用
#初始化磁盘
[root@xianchaonode1 ~]# mkfs.xfs /dev/rbd0
Discarding blocks...Done.
meta-data=/dev/rbd0              isize=512    agcount=8, agsize=98304 blks=                       sectsz=512   attr=2, projid32bit=1=                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=786432, imaxpct=25=                       sunit=16     swidth=16 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2=                       sectsz=512   sunit=16 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
[root@xianchaonode1 ~]# mount /dev/rbd0  /mnt/
[root@xianchaonode1 ~]# echo 111 >> /mnt/test.txt
[root@xianchaonode1 ~]# cat /mnt/test.txt
111
[root@xianchaonode1 ~]# df -h 
Filesystem                                                                       Size  Used Avail Use% Mounted on
devtmpfs                                                                         7.9G     0  7.9G   0% /dev
tmpfs                                                                            7.9G     0  7.9G   0% /dev/shm
tmpfs                                                                            7.9G  795M  7.1G  10% /run
tmpfs                                                                            7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sda2                                                                        200G   62G  138G  31% /             
tmpfs                                                                            1.6G     0  1.6G   0% /run/user/0
/dev/rbd0                                                                        3.0G   33M  3.0G   2% /mnt
[root@xianchaonode1 ~]# 

4.CephFS使用详解

ceph FS 即 ceph filesystem,可以实现文件系统共享功能(POSIX 标准), 客户端通过 ceph协议挂载并使用 ceph 集群作为数据存储服务器,http://docs.ceph.org.cn/cephfs/。 Ceph FS 需要运行 Meta Data Services(MDS)服务,其守护进程为 ceph-mds,ceph-mds 进程管理与 cephFS 上存储的文件相关的元数据,并协调对 ceph 存储集群的访问。

   在linux系统使用 ls 等操作查看某个目录下的文件的时候,会有保存在磁盘上的分区表 记录文件的名称、创建日期、大小、inode 及存储位置等元数据信息,在 cephfs 由于数据 是被打散为若干个离散的 object 进行分布式存储,因此并没有统一保存文件的元数据,而且将文件的元数据保存到一个单独的存储出 matedata pool,但是客户端并不能直接访问 matedata pool 中的元数据信息,而是在读写数的时候有 MDS(matadata server)进行处理, 读数据的时候有 MDS从 matedata pool加载元数据然后缓存在内存(用于后期快速响应其它 客户端的请求)并返回给客户端,写数据的时候有MDS 缓存在内存并同步到matedata pool。

在这里插入图片描述

4.1 部署MDS服务

如果要使用 cephFS,需要部署 MDS 服务,可以部署在mon节点,

root@ceph-master01:~# apt-cache madison ceph-mds
root@ceph-master01:~# apt install ceph-mds
root@ceph-master01:~# ceph-deploy mds create  ceph-master01 ceph-master02 ceph-master03

在这里插入图片描述

#检查主从状态
ceph -s
ceph fs status

在这里插入图片描述

在这里插入图片描述

4.2 创建CephFS mdetadata和data存储池

使用 CephFS 之前需要事先于集群中创建一个文件系统,并为其分别指定元数据和数据相关的存储池。下面创建一个名为 cephfs 的文件系统用于测试,它使用 cephfs-metadata 为 元数据存储池,使用 cephfs-data 为数据存储池.

root@ceph-master01:~# ceph osd pool create cephfs-metadata 32 32
pool 'cephfs-metadata' created
root@ceph-master01:~# ceph osd pool create cephfs-data 64 64
pool 'cephfs-data' created
4.3 创建 cephFS 并验证
root@ceph-master01:~# ceph fs new mycephfs cephfs-metadata cephfs-data
new fs with metadata pool 5 and data pool 6
root@ceph-master01:~# ceph fs ls
name: mycephfs, metadata pool: cephfs-metadata, data pools: [cephfs-data ]
root@ceph-master01:~# ceph fs status mycephfs
mycephfs - 0 clients
========POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata     0    142G  cephfs-data      data       0    142G  
4.4 创建cephFS客户端账户
#创建账户
root@ceph-master01:/etc/ceph-cluster# ceph auth add client.yanyan mon 'allow r' mds 'allow rw' osd 'allow rwx pool=cephfs-data'
added key for client.yanyan#验证账户
root@ceph-master01:/etc/ceph-cluster# ceph auth get client.yanyan
[client.yanyan]key = AQDnhSljvlhoLxAAWrV9uY1kXq5/C0jAziaB9Q==caps mds = "allow rw"caps mon = "allow r"caps osd = "allow rwx pool=cephfs-data"
exported keyring for client.yanyan
root@ceph-master01:/etc/ceph-cluster# ceph auth get client.yanyan -o ceph.client.yanyan.keyring
exported keyring for client.yanyanroot@ceph-master01:/etc/ceph-cluster# ll
total 416
drwxr-xr-x  2 cephadmin cephadmin   4096 Sep 20 17:21 ./
drwxr-xr-x 92 root      root        4096 Sep  5 09:46 ../
-rw-------  1 cephadmin cephadmin    113 Sep  5 10:52 ceph.bootstrap-mds.keyring
-rw-------  1 cephadmin cephadmin    113 Sep  5 10:52 ceph.bootstrap-mgr.keyring
-rw-------  1 cephadmin cephadmin    113 Sep  5 10:52 ceph.bootstrap-osd.keyring
-rw-------  1 cephadmin cephadmin    113 Sep  5 10:52 ceph.bootstrap-rgw.keyring
-rw-------  1 cephadmin cephadmin    151 Sep  5 10:52 ceph.client.admin.keyring
-rw-r--r--  1 root      root         150 Sep 20 17:21 ceph.client.yanyan.keyring
-rw-rw-r--  1 cephadmin cephadmin    398 Sep  7 20:01 ceph.conf
-rw-rw-r--  1 cephadmin cephadmin 368945 Sep  7 20:02 ceph-deploy-ceph.log
-rw-------  1 cephadmin cephadmin     73 Sep  2 16:50 ceph.mon.keyring
-rw-r--r--  1 root      root           9 Sep 12 13:06 pass.txt
-rw-r--r--  1 root      root        1645 Oct 16  2015 release.asc
root@ceph-master01:/etc/ceph-cluster# ceph auth print-key client.yanyan > yanyan.key
root@ceph-master01:/etc/ceph-cluster# ll
total 420
drwxr-xr-x  2 cephadmin cephadmin   4096 Sep 20 17:21 ./
drwxr-xr-x 92 root      root        4096 Sep  5 09:46 ../
-rw-------  1 cephadmin cephadmin    113 Sep  5 10:52 ceph.bootstrap-mds.keyring
-rw-------  1 cephadmin cephadmin    113 Sep  5 10:52 ceph.bootstrap-mgr.keyring
-rw-------  1 cephadmin cephadmin    113 Sep  5 10:52 ceph.bootstrap-osd.keyring
-rw-------  1 cephadmin cephadmin    113 Sep  5 10:52 ceph.bootstrap-rgw.keyring
-rw-------  1 cephadmin cephadmin    151 Sep  5 10:52 ceph.client.admin.keyring
-rw-r--r--  1 root      root         150 Sep 20 17:21 ceph.client.yanyan.keyring
-rw-rw-r--  1 cephadmin cephadmin    398 Sep  7 20:01 ceph.conf
-rw-rw-r--  1 cephadmin cephadmin 368945 Sep  7 20:02 ceph-deploy-ceph.log
-rw-------  1 cephadmin cephadmin     73 Sep  2 16:50 ceph.mon.keyring
-rw-r--r--  1 root      root           9 Sep 12 13:06 pass.txt
-rw-r--r--  1 root      root        1645 Oct 16  2015 release.asc
-rw-r--r--  1 root      root          40 Sep 20 17:21 yanyan.keyroot@ceph-master01:/etc/ceph-cluster# cat ceph.client.yanyan.keyring
[client.yanyan]key = AQDnhSljvlhoLxAAWrV9uY1kXq5/C0jAziaB9Q==caps mds = "allow rw"caps mon = "allow r"caps osd = "allow rwx pool=cephfs-data"
root@ceph-master01:/etc/ceph-cluster# 
4.5 安装ceph客户端
#以centos客户端
yum install epel-release -y
yum install https://mirrors.aliyun.com/ceph/rpm-octopus/el7/noarch/ceph-release-1-1.el7.noarch.rpm
yum install ceph-common -y
4.6 同步认证文件
root@ceph-master01:~# cd /etc/ceph-cluster/
root@ceph-master01:/etc/ceph-cluster# scp ceph.conf ceph.client.yanyan.keyring yanyan.key root@172.26.156.165:/etc/ceph/

客户端权限认证

[root@zd_spring_156_101 ceph]# ceph  --user yanyan  -s 

在这里插入图片描述

4.7 客户端安装ceph-common
#配置 yum 源: 
$ yum install epel-release 
$ yum install https://mirrors.aliyun.com/ceph/rpm-octopus/el7/noarch/ceph-release-1-1.el7.noarch.rpm -y
#下载ceph-common
$ yum install -y ceph-common
#验证ceph-common
[root@zd_spring_156_101 ~]# rpm -qa | grep ceph-common
python3-ceph-common-15.2.17-0.el7.x86_64
ceph-common-15.2.17-0.el7.x86_64
4.8 cephfs挂载使用

客户端挂载有两种方式,一是内核空间一是用户空间,内核空间挂载需要内核支持 ceph模块(内核版本3.10.0-862以上,centos7.5默认内核),用户空间挂载需要安装 ceph-fuse,如果内核本较低而没有 ceph 模块(验证centos7.5及以上默认内核基本都有ceph模块,centos7.3以下默认内核未测试),那么可以安装 ceph-fuse 挂载,但是推荐使用内核模块挂载。

4.8.1 内核空间挂载使用ceph-fs
#客户端通过 key 挂载(不需要安装ceph-common)
[root@other165 ~]# cat /etc/ceph/yanyan.key 
AQDnhSljvlhoLxAAWrV9uY1kXq5/C0jAziaB9Q==
[root@other165 ~]# mount -t ceph 172.26.156.217:6789,172.26.156.218:6789,172.26.156.219:6789:/ /mnt  -o name=yanyan,secret=AQDnhSljvlhoLxAAWrV9uY1kXq5/C0jAziaB9Q==#客户端通过 key 文件挂载(需要安装ceph-common)
[root@other165 ~]# mount -t ceph 172.26.156.217:6789,172.26.156.218:6789,172.26.156.219:6789:/ /mnt  -o name=yanyan,secretfile=/etc/ceph/yanyan.key

在这里插入图片描述

4.8.2 开机自动挂载
# cat /etc/fstab
172.26.156.217:6789,172.26.156.218:6789,172.26.156.219:6789:/ /mnt ceph defaults,name=yanyan,secretfile=/etc/ceph/yanyan.key,_netdev 0 0

在这里插入图片描述

4.9用户空间挂载 ceph-fs

如果内核本较低而没有 ceph 模块,那么可以安装 ceph-fuse 挂载,但是推荐使用内核模块挂载。

4.9.1安装 ceph-fuse
#配置 yum 源: 
$ yum install epel-release 
$ yum install https://mirrors.aliyun.com/ceph/rpm-octopus/el7/noarch/ceph-release-1-1.el7.noarch.rpm -y
#下载ceph-common
$ yum install ceph-fuse  -y
4.9.2 ceph-fuse 挂载 ceph
#默认读取/etc/ceph/下
ceph-fuse --name client.yanyan -m 172.26.156.217:6789,172.26.156.218:6789,172.26.156.219:6789 /mnt

在这里插入图片描述

4.9.3 开机自动挂载

指定用户会自动根据用户名称加载授权文件及配置文件 ceph.conf

vim /etc/fstab
none /data fuse.ceph ceph.id=yanyan,ceph.conf=/etc/ceph/ceph.conf,_netdev,defaults 0 0

5.k8s使用ceph案例

5.1 RBD静态存储
5.1.1 使用pv/pvc挂载RBD
apiVersion: v1 
kind: PersistentVolume 
metadata: name: ceph-pv 
spec:   capacity:     storage: 1Gi   accessModes:     - ReadWriteOnce   rbd:     monitors:      - '172.26.156.217:6789'- '172.26.156.218:6789'- '172.26.156.219:6789' pool: k8stest     #需要创建image: rbda       #需要创建user: admin       #需要创建secretRef:       name: ceph-secret   #需要创建  fsType: xfs     readOnly: false   persistentVolumeReclaimPolicy: Recycle
---
kind: PersistentVolumeClaim 
apiVersion: v1 
metadata:   name: ceph-pvc 
spec:   accessModes:     - ReadWriteOnce   resources:     requests:       storage: 1Gi
5.1.2 直接使用pod挂载RBD
apiVersion: apps/v1
kind: Deployment
metadata:name: nginx-deployment
spec:replicas: 1selector:matchLabels: #rs or deploymentapp: ng-deploy-80template:metadata:labels:app: ng-deploy-80spec:nodeName: xianchaonode1containers:- name: ng-deploy-80image: nginxports:- containerPort: 80volumeMounts:- name: rbd-data1mountPath: /usr/share/nginx/html/rbdvolumes:- name: rbd-data1rbd:monitors:- '172.26.156.217:6789'- '172.26.156.218:6789'- '172.26.156.219:6789' pool: shijie-rbd-pool1image: shijie-img-img1fsType: xfsreadOnly: falseuser: magedu-shijiesecretRef:name: ceph-secret-magedu-shijie
5.1 RBD动态存储类

存储卷可以通过 kube-controller-manager 组件动态创建,适用于有状态服务需要多个存储卷的场合。 将 ceph admin 用户 key 文件定义为 k8s secret,用于 k8s 调用 ceph admin 权限动态创建存储卷,即不再需要提前创建好 image 而是 k8s 在需要使用的时候再调用 ceph 创建。

5.1.1 创建rbd pool
root@ceph-master01:/etc/ceph# ceph osd pool create k8s-rbd 32 32 
pool 'k8s-rbd' created
root@ceph-master01:/etc/ceph# ceph osd pool application enable  k8s-rbd rbd 
enabled application 'rbd' on pool 'k8s-rbd'
root@ceph-master01:/etc/ceph# rbd pool init -p k8s-rbd 
5.1.2 创建 admin 用户 secret:

用于k8s有权限创建rbd

#查看ceph adminbase64密钥
root@ceph-master01:/etc/ceph# ceph auth print-key client.admin | base64
QVFCM1pCVmpMOE4wRUJBQVJlRzBxM3JwVkYvOERkbk11cnlaTkE9PQ==#ceph admin 用户 secret 文件内容
[root@xianchaomaster1 pod-rbd]# vi case1-secret-admin.yaml 
apiVersion: v1
kind: Secret
metadata:name: ceph-secret-adminnamespace: default
type: "kubernetes.io/rbd"
data:key: QVFCM1pCVmpMOE4wRUJBQVJlRzBxM3JwVkYvOERkbk11cnlaTkE9PQ==
5.1.3 创建普通用户的 secret

用于访问存储卷进行数据读写

root@ceph-master01:/etc/ceph# ceph auth get-or-create client.k8s-rbd mon 'allow r' osd 'allow * pool=k8s-rbd'
[client.k8s-rbd]key = AQAMgkZjDyhsMhAAEH8F0Gwe3L+aiP/wAkqdyA==
root@ceph-master01:/etc/ceph# ceph auth print-key client.k8s-rbd | base64
QVFBTWdrWmpEeWhzTWhBQUVIOEYwR3dlM0wrYWlQL3dBa3FkeUE9PQ==vi case2-secret-client.yaml
apiVersion: v1
kind: Secret
metadata:name: k8s-rbd
type: "kubernetes.io/rbd"
data:key: QVFBTWdrWmpEeWhzTWhBQUVIOEYwR3dlM0wrYWlQL3dBa3FkeUE9PQ==
5.1.4 创建存储类

创建动态存储类,为pod提供动态pv

vi case3-ceph-storage-class.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:name: ceph-storage-classannotations:storageclass.kubernetes.io/is-default-class: "false" #设置为默认存储类
provisioner: kubernetes.io/rbd
reclaimPolicy: Retain      #默认是Delete,危险
parameters:monitors: 172.26.156.217:6789,172.26.156.218:6789,172.26.156.219:6789adminId: adminadminSecretName: ceph-secret-adminadminSecretNamespace: defaultpool: k8s-rbduserId: k8s-rbduserSecretName: k8s-rbd
5.1.5 创建基于存储类的PVC
vi case4-mysql-pvc.yaml  
apiVersion: v1
kind: PersistentVolumeClaim
metadata:name: mysql-data-pvc
spec:accessModes:- ReadWriteOncestorageClassName: ceph-storage-classresources:requests:storage: '5Gi'#验证 PV/PVC:
kubectl  get pvc
kubectl  get pv

在这里插入图片描述

#验证ceph是否自动创建image

rbd ls --pool  k8s-rbd

在这里插入图片描述

5.1.6 运行单机mysql pod验证
vi case5-mysql-deploy-svc.yaml
apiVersion: apps/v1
kind: Deployment
metadata:name: mysql
spec:selector:matchLabels:app: mysqlstrategy:type: Recreatetemplate:metadata:labels:app: mysqlspec:containers:- image: mysql:5.6.46name: mysqlenv:# Use secret in real usage- name: MYSQL_ROOT_PASSWORDvalue: 123456ports:- containerPort: 3306name: mysqlvolumeMounts:- name: mysql-persistent-storagemountPath: /var/lib/mysqlvolumes:- name: mysql-persistent-storagepersistentVolumeClaim:claimName: mysql-data-pvc
---
kind: Service
apiVersion: v1
metadata:labels:app: mysql-service-labelname: mysql-service
spec:type: NodePortports:- name: httpport: 3306protocol: TCPtargetPort: 3306nodePort: 33306selector:app: mysql

#连接验证,创建test库

在这里插入图片描述

#删除mysql pod 重新创建,验证rbd数据持久

 kubectl delete -f case5-mysql-deploy-svc.yamlkubectl apply -f case5-mysql-deploy-svc.yaml

在这里插入图片描述

#将pod调度到指定的其他node节点,验证能否挂载rbd

在这里插入图片描述

 kubectl delete -f case5-mysql-deploy-svc.yamlkubectl apply -f case5-mysql-deploy-svc.yaml

依然可以挂载成功

在这里插入图片描述

5.2 cephFS静态存储
5.2.1 使用pv/pvc挂载cephFS

注意的是,一个cephFS pool共享多个目录,需要在cephfs中提前创建好子目录分给各个deployment挂载,找一台linux主机提前挂载此cephfs,创建/data2目录,不然pod只能挂载cepfFS的/目录,mount -t ceph 172.26.156.217:6789,172.26.156.218:6789,172.26.156.219:6789:/ /mnt -o name=admin,secret=AQB3ZBVjL8N0EBAAReG0q3rpVF/8DdnMuryZNA==

#创建pv
apiVersion: v1
kind: PersistentVolume
metadata:name: cephfs-pvlabels:app: static-cephfs-pv
spec:capacity:storage: 1GiaccessModes:- ReadWriteManycephfs:monitors:- 172.26.156.217:6789- 172.26.156.218:6789- 172.26.156.219:6789path: /data2/    #需要提前在cephFS pool中创建好/data2user: adminsecretRef:name: ceph-secret-adminreadOnly: falsepersistentVolumeReclaimPolicy: RecyclestorageClassName: slow
---
#创建pvc
apiVersion: v1
kind: PersistentVolumeClaim
metadata:name: cephfs-pvc-claim
spec:selector:matchLabels:app: static-cephfs-pvstorageClassName: slowaccessModes:- ReadWriteManyresources:requests:storage: 1Gi
---
#deployment
apiVersion: apps/v1
kind: Deployment
metadata:name: nginx2
spec:selector:matchLabels:k8s-app: nginx2replicas: 2template:metadata:labels:k8s-app: nginx2spec:containers:- name: nginximage: nginximagePullPolicy: IfNotPresentports:- containerPort: 80protocol: TCPvolumeMounts:- name: pvc-recyclemountPath: /usr/share/nginx/html/nginx2volumes:- name: pvc-recyclepersistentVolumeClaim:claimName: cephfs-pvc-claim
---
kind: Service
apiVersion: v1
metadata:labels:k8s-app: nginx2name: ng-deploy-80-service
spec:type: NodePortports:- name: httpport: 80protocol: TCPtargetPort: 80nodePort: 23380selector:k8s-app: nginx2
5.2 直接使用pod挂载cephFS

不需要创建pv/pvc,直接创建deployment挂载cephFS

apiVersion: apps/v1
kind: Deployment
metadata:name: nginx-deployment
spec:replicas: 2selector:matchLabels: #rs or deploymentapp: ng-deploy-80template:metadata:labels:app: ng-deploy-80spec:containers:- name: ng-deploy-80image: nginxports:- containerPort: 80volumeMounts:- name: magedu-staticdata-cephfsmountPath: /usr/share/nginx/html/cephfsvolumes:- name: magedu-staticdata-cephfscephfs:monitors:- '172.26.156.217:6789'- '172.26.156.218:6789'- '172.26.156.219:6789'path: /user: adminsecretRef:name: ceph-secret-admin
---
kind: Service
apiVersion: v1
metadata:labels:app: ng-deploy-80-service-labelname: ng-deploy-80-service
spec:type: NodePortports:- name: httpport: 80protocol: TCPtargetPort: 80nodePort: 33380selector:app: ng-deploy-80
5.4 cephFS动态存储类

虽然官方并没有直接提供对Cephfs StorageClass的支持,但是社区给出了类似的解决方案 external-storage/ cephfs。

测试发现Cephfs StorageClass k8s1.20版本之后已经不能使用。按照这种方式会报错以下截图,网上的解决方案需要在kube-apiserver.yaml配置文件中添加–feature-gates=RemoveSelfLink=false,这个参数在k8s1.20版本之后已经移除,后续使用ceph-csi方式。

Cephfs StorageClass部署方案(不成功):

https://www.cnblogs.com/leffss/p/15630641.html

https://www.cnblogs.com/estarhaohao/p/15965785.html

github issues: https://github.com/kubernetes/kubernetes/issues/94660

在这里插入图片描述

5.5 ceph-csi 动态存储
Ceph-CSI RBD

https://www.modb.pro/db/137721

Ceph-CSI CephFS

最新版本3.7遇到问题,遂使用CSI-3.4版本

git clone https://github.com/ceph/ceph-csi.git -b release-v3.4
cd ceph-csi/deploy/cephfs/kubernetes

修改 ConfigMap 对象配置,clusterID 是 ceph fsid 。

vi csi-config-map.yaml 
---
apiVersion: v1
kind: ConfigMap
data:config.json: |-[{"clusterID": "f69afe6f-e559-4df7-998a-c5dc3e300209","monitors": ["172.26.156.217:6789","172.26.156.218:6789","172.26.156.219:6789"]}]
metadata:name: ceph-csi-config

ceph-csi 默认部署在 default 命名空间,这里改到 kube-system 。

sed -i "s/namespace: default/namespace: kube-system/g" $(grep -rl "namespace: default" ./)

部署 ceph-csi CephFS ,镜像的仓库是 k8s.gcr.io , 部分镜像拉取失败,可在dockerhub上search替换

kubectl get po -n kube-system  | grep csi-cephfs
csi-cephfsplugin-8xt97                          3/3     Running   0          6d10h
csi-cephfsplugin-bmxwr                          3/3     Running   0          6d10h
csi-cephfsplugin-n74cd                          3/3     Running   0          6d10h
csi-cephfsplugin-provisioner-79d84c9598-fb6bg   6/6     Running   0          6d10h
csi-cephfsplugin-provisioner-79d84c9598-g579j   6/6     Running   0          6d10h
csi-cephfsplugin-provisioner-79d84c9598-n8w2j   6/6     Running   0          6d10h

***创建 Ceph*FS storageClass

ceph-csi 需要 cephx 凭据才能与 Ceph 集群通信,这里使用的是 admin 用户。

vi secret.yaml
apiVersion: v1
kind: Secret
metadata:name: csi-cephfs-secretnamespace: default
stringData:adminID: adminadminKey: AQB3ZBVjL8N0EBAAReG0q3rpVF/8DdnMuryZNA==

创建 storageClass 对象,这里使用的 Ceph FS name 是 mycephfs(ceph中新建cephfs时的名字,他不是一个pool) 。

#ceph fs new mycephfs cephfs-metadata cephfs-data

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:name: ceph-csi-cephfs
provisioner: cephfs.csi.ceph.com
parameters:clusterID: f69afe6f-e559-4df7-998a-c5dc3e300209fsName: mycephfscsi.storage.k8s.io/provisioner-secret-name: csi-cephfs-secretcsi.storage.k8s.io/provisioner-secret-namespace: defaultcsi.storage.k8s.io/controller-expand-secret-name: csi-cephfs-secretcsi.storage.k8s.io/controller-expand-secret-namespace: defaultcsi.storage.k8s.io/node-stage-secret-name: csi-cephfs-secretcsi.storage.k8s.io/node-stage-secret-namespace: default
reclaimPolicy: Retain
allowVolumeExpansion: true
mountOptions:- discard

创建 PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:name: csi-cephfs-pvcnamespace: default
spec:accessModes:- ReadWriteManyresources:requests:storage: 1GistorageClassName: ceph-csi-cephfs

自动创建了pv,并且pv/pvc绑定

在这里插入图片描述

创建测试的 Deployment

vi Deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:name: cephfs-testlabels:component: cephfs-test
spec:replicas: 2strategy:type: Recreateselector:matchLabels:component: cephfs-testtemplate:metadata:labels:component: cephfs-testspec:containers:- name: nginximage: nginximagePullPolicy: IfNotPresentports:- containerPort: 80volumeMounts:- name: configmountPath: "/data"volumes:- name: configpersistentVolumeClaim:claimName: csi-cephfs-pvcreadOnly: false

csi-cephfs 默认会创建一个名为 csi 的子文件系统

# ceph fs subvolumegroup ls cephfs
[{"name": "_deleting"},{"name": "csi"}
]

所有使用 csi-cephfs 创建的 PV ,都是在子文件系统 csi 的目录下

 kubectl get pv | grep default/csi-cephfs-pvc
pvc-0f36fd44-40f1-4ac3-aebe-0264a2fb50ea   1Gi        RWX            Delete           Bound    default/csi-cephfs-pvc              ceph-csi-cephfs            6d11h# kubectl describe  pv pvc-0f36fd44-40f1-4ac3-aebe-0264a2fb50ea  | egrep 'subvolumeName|subvolumePath'subvolumeName=csi-vol-056e44c5-eddf-11eb-a990-a63fe71a40b6subvolumePath=/volumes/csi/csi-vol-056e44c5-eddf-11eb-a990-a63fe71a40b6/e423daf3-017b-4a7e-8713-bd05bab695ee# cd /mnt/cephfs-test/
# tree -L 4 ./
./
└── volumes├── csi│   ├── csi-vol-056e44c5-eddf-11eb-a990-a63fe71a40b6│   │   └── e423daf3-017b-4a7e-8713-bd05bab695ee│   └── csi-vol-1ac1f4c1-ef8a-11eb-a990-a63fe71a40b6│       └── 3773a567-a8cb-4bae-9181-38f4e3065436├── _csi:csi-vol-056e44c5-eddf-11eb-a990-a63fe71a40b6.meta├── _csi:csi-vol-1ac1f4c1-ef8a-11eb-a990-a63fe71a40b6.meta└── _deleting7 directories, 2 files

理解

1.存储数据, object, pg,pgp, pool, osd, 存储磁盘的关系

在这里插入图片描述

2.Filestore、BlueStore、journal理解

ceph后端支持多种存储引擎,我们常听到的其中两种存储引擎就是FileStore与BlueStore,在L版(包括L版)之前,默认使用的filestore作为默认的存储引擎,但是由于FileStore存在一些缺陷,重新设计开发了BlueStore,L版之后默认使用的存储引擎就是BlueStore了。

BlueStore 对于整块数据的写入,数据直接 AIO 的方式写入磁盘,避免了 filestore的先写日志,后 apply到实际磁盘的两次写盘。对于随机IO,直接 WAL 的形式,直接写入 RocksDB 高性能的 KV 存储中。

所以我们在网上看到的各种优化资源文档,部署ceph时需要给ceph单独设定一块ssd盘为Journal刷写日志,但这种只针对L版本之前使用,在L版本之后已经没有Journal预刷写日志了。L版本之后使用下面的优化方式

#服务器上有两块ssd盘时,可以分别把block-db,block-wal存放到ssd盘中
ceph-deploy osd create ceph-node1 --data /dev/sdc --block-db /dev/sda --block-wal /dev/sdb
#只有一块硬盘时,只指定db的话存放ssd盘,没有指定waf存放位置,waf也会自动写到更快速的ssd盘上,和db共用一块。
ceph-deploy osd create ceph-node1 --data /dev/sdb--block-db /dev/sda 

需求

1.删除OSD的正确方式

Luminous 之前版本

1.调整osd的crush weight

  • 调整osd的crush weight
ceph osd crush reweight osd.0 0.5
ceph osd crush reweight osd.0 0.2
ceph osd crush reweight osd.0 0

说明:这个地方如果想慢慢的调整就分几次将crush 的weight 减低到0 ,这个过程实际上是让数据不分布在这个节点上,让数据慢慢的分布到其他节点上,直到最终为没有分布在这个osd,并且迁移完成。这个地方不光调整了osd 的crush weight ,实际上同时调整了host 的 weight ,这样会调整集群的整体的crush 分布,在osd 的crush 为0 后, 再对这个osd的任何删除相关操作都不会影响到集群的数据的分布。

  • 停止osd进程
 systemctl  stop ceph-osd@0.service

停止osd的进程,这个是通知集群这个osd进程不在了,不提供服务了,因为本身没权重,就不会影响到整体的分布,也就没有迁移。

  • 将节点状态标记为out
ceph osd out osd.0

将osd退出集群,这个是通知集群这个osd不再映射数据了,不提供服务了,因为本身没权重,就不会影响到整体的分布,也就没有迁移。

  • 从crush中移除节点
ceph osd crush remove osd.0

这个是从crush中删除,因为OSD权重已经是0了 所以没影响主机的权重,也就没有迁移了。

  • 删除节点
ceph osd rm osd.0

这个是从集群里面删除这个OSD的记录。

  • 删除OSD认证(不删除编号会占住)
ceph auth del osd.0

这个是从认证当中去删除这个OSD的信息。

经过验证,此种方式只触发了一次迁移,虽然只是一个步骤先后上的调整,对于生产环境的的集群来说,迁移的量要少了一次,实际生产环境当中节点是有自动out的功能,这个可以考虑自己去控制,只是监控的密度需要加大,毕竟这个是一个需要监控的集群,完全让其自己处理数据的迁移是不可能的,带来的故障只会更多。

Luminous 之后版本

1.调整osd的crush weight

  • 调整osd的crush weight
ceph osd crush reweight osd.0 0.5
ceph osd crush reweight osd.0 0.2
ceph osd crush reweight osd.0 0

说明:这个地方如果想慢慢的调整就分几次将crush 的weight 减低到0 ,这个过程实际上是让数据不分布在这个节点上,让数据慢慢的分布到其他节点上,直到最终为没有分布在这个osd,并且迁移完成。这个地方不光调整了osd 的crush weight ,实际上同时调整了host 的 weight ,这样会调整集群的整体的crush 分布,在osd 的crush 为0 后, 再对这个osd的任何删除相关操作都不会影响到集群的数据的分布。

  • 停止osd进程
 systemctl  stop ceph-osd@0.service

停止osd的进程,这个是通知集群这个osd进程不在了,不提供服务了,因为本身没权重,就不会影响到整体的分布,也就没有迁移。

  • 将节点状态标记为out
ceph osd out osd.0

将osd退出集群,这个是通知集群这个osd不再映射数据了,不提供服务了,因为本身没权重,就不会影响到整体的分布,也就没有迁移。

移除设备

ceph osd purge {id} --yes-i-really-mean-it

若 OSD 的配置信息存在于 ceph.conf 配置文件中,管理员在删除 OSD 之后手

动将其删除。

2.ceph 旧OSD 节点 格式化 数据,加入新 ceph集群
旧有的ceph osd , 想要格式化之后加入新的ceph节点。
查询 osd 旧有 数据,后面的步骤会用到。[root@ceph-207 ~]# ceph-volume lvm list
====== osd.1 =======
[block] /dev/ceph-58ef1d0f-272b-4273-82b1-689946254645/osd-block-e0efe172-778e-46e1-baa2-cd56408aac34block device /dev/ceph-58ef1d0f-272b-4273-82b1-689946254645/osd-block-e0efe172-778e-46e1-baa2-cd56408aac34
block uuid hCx4XW-OjKC-OC8Y-jEg2-NKYo-Pb6f-y9Nfl3
cephx lockbox secret
cluster fsid b7e4cb56-9cc8-4e44-ab87-24d4253d0951
cluster name ceph
crush device class None
encrypted 0
osd fsid e0efe172-778e-46e1-baa2-cd56408aac34
osd id 1
osdspec affinity
type block
vdo 0
devices /dev/sdb
直接加入集群报错:ceph-volume lvm activate 1 e0efe172-778e-46e1-baa2-cd56408aac34
目前遇到了两类报错:osd.1 21 heartbeat_check: no reply from 192.168.8.206:6804 osd.0 ever on either front or back, first ping sent 2020-11-26T16:00:04.842947+0800 (oldest deadline 2020-11-26T16:00:24.842947+0800)stderr: Calculated size of logical volume is 0 extents. Needs to be larger.--> Was unable to complete a new OSD, will rollback changes格式化数据,重新加入新的ceph集群:1、停止osd 服务 , @ 后面的 1 为 ceph-volume lvm list 命令查询出的 osd idsystemctl stop ceph-osd@1
2、重处理 osd lvm, 1 还是 osd idceph-volume lvm zap --osd-id 1
3、查询 lvs 信息, 删除 lv、pg 等信息[root@ceph-207 ~]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
osd-block-e0efe172-778e-46e1-baa2-cd56408aac34 ceph-58ef1d0f-272b-4273-82b1-689946254645 -wi-a----- <16.00g
home cl -wi-ao---- <145.12g
root cl -wi-ao---- 50.00g
swap cl -wi-ao---- <3.88g
[root@ceph-207 ~]# vgremove ceph-58ef1d0f-272b-4273-82b1-689946254645
Do you really want to remove volume group "ceph-58ef1d0f-272b-4273-82b1-689946254645" containing 1 logical volumes? [y/n]: y
Do you really want to remove active logical volume ceph-58ef1d0f-272b-4273-82b1-689946254645/osd-block-e0efe172-778e-46e1-baa2-cd56408aac34? [y/n]: y
Logical volume "osd-block-e0efe172-778e-46e1-baa2-cd56408aac34" successfully remove
4、将主机上的磁盘重新加入新的ceph集群ceph-volume lvm create --data /dev/sdb
5、查询下 osd tree , 磁盘挂载情况[root@ceph-207 ~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.07997 root default
-3 0.03508 host ceph-206
0 hdd 0.01559 osd.0 up 1.00000 1.00000
1 hdd 0.01949 osd.1 up 1.00000 1.00000
-5 0.04489 host ceph-207
2 hdd 0.01559 osd.2 up 1.00000 1.00000
[root@ceph-207 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 200G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 199G 0 part
├─cl-root 253:0 0 50G 0 lvm /
├─cl-swap 253:1 0 3.9G 0 lvm [SWAP]
└─cl-home 253:2 0 145.1G 0 lvm /home
sdb 8:16 0 16G 0 disk
└─ceph--c221ed63--d87a--4bbd--a503--d8f2ed9e806b-osd--block--530376b8--c7bc--4d64--bc0c--4f8692559562 253:3 0 16G 0 lvm
sr0
3.ceph如何修改配置文件

默认生成的ceph.conf文件如果需要改动的话需要加一些参数,如果配置文件变化也是通过ceph-deploy进行推送。请不要直接修改某个节点的"/etc/ceph/ceph.conf"文件,而是在部署机下修改ceph.conf,采用推送的方式更加方便安全。

 vi /etc/ceph-cluster/ceph.conf
[global]
fsid = f69afe6f-e559-4df7-998a-c5dc3e300209
public_network = 172.26.0.0/16
cluster_network = 10.0.0.0/24
mon_initial_members = ceph-master01, ceph-master02, ceph-master03
mon_host = 172.26.156.217,172.26.156.218,172.26.156.219
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx[mon]
mon_clock_drift_allowed = 0.10
mon clock drift warn backoff = 10

亲测,参数名称要不要下划线都可以

改完后将配置推到集群所有的机器

cephadmin@ceph-master01:/etc/ceph-cluster# ceph-deploy --overwrite-conf config push  ceph-master01  ceph-master02 ceph-master03

重启所有机器

root@ceph-master01:~# systemctl   restart ceph-mon@ceph-master01.service
root@ceph-master02:~# systemctl   restart ceph-mon@ceph-master02.service
root@ceph-master03:~# systemctl   restart ceph-mon@ceph-master03.service

查看集群状态

cephadmin@ceph-master01:/etc/ceph-cluster# ceph -s cluster:id:     f69afe6f-e559-4df7-998a-c5dc3e300209health: HEALTH_OKservices:mon: 3 daemons, quorum ceph-master01,ceph-master02,ceph-master03 (age 3m)mgr: ceph-master03(active, since 26h), standbys: ceph-master01, ceph-master02osd: 9 osds: 9 up (since 26h), 9 in (since 27h)data:pools:   2 pools, 33 pgsobjects: 1 objects, 100 MiBusage:   370 MiB used, 450 GiB / 450 GiB availpgs:     33 active+clean

报错记录

1. bash: python2: command not found

在这里插入图片描述

原因:ceph-master02节点没有安装python2.7

解决方案:

cephadmin@ceph-master01:~$ sudo apt install python2.7 -y
cephadmin@ceph-master01:~$ sudo ln -sv /usr/bin/python2.7 /usr/bin/python2
2.[ceph_deploy][ERROR ] RuntimeError: AttributeError: module ‘platform’ has no attribute ‘linux_distribution’

# ceph-deploy new --cluster-network 10.0.0.0/24 --public-network 172.26.0.0/16 master01

在这里插入图片描述

原因:

部署cepe的操作系统为ubuntu20.04,该版本python3.7后不再支持platform.linux_distribution

解决办法:

修改/usr/lib/python3/dist-packages/ceph_deploy/hosts/remotes.py文件为如下所示

def platform_information(_linux_distribution=None):""" detect platform information from remote host """"""linux_distribution = _linux_distribution or platform.linux_distributiondistro, release, codename = linux_distribution()"""distro = release = codename = Nonetry:linux_distribution = _linux_distribution or platform.linux_distributiondistro, release, codename = linux_distribution()except AttributeError:pass

修改前:

在这里插入图片描述

修改后:

在这里插入图片描述

3.apt-cache madison ceph-deploy 为1.5.38的低版本
cephadmin@ceph-master01:~$ sudo apt-cache madison ceph-deploy 
ceph-deploy | 1.5.38-0ubuntu1 | http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic/universe amd64 Packages
ceph-deploy | 1.5.38-0ubuntu1 | http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic/universe i386 Packages
cephadmin@ceph-master01:~$ 
4. RuntimeError: Failed to execute command: /usr/sbin/ceph-volume lvm zap /dev/sdd
cephadmin@ceph-master01:/etc/ceph-cluster# ceph-deploy disk zap ceph-master01 /dev/sdd 
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadmin/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy disk zap ceph-master01 /dev/sdd
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  debug                         : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : zap
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f8d53187280>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  host                          : ceph-master01
[ceph_deploy.cli][INFO  ]  func                          : <function disk at 0x7f8d5315d350>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  disk                          : ['/dev/sdd']
[ceph_deploy.osd][DEBUG ] zapping /dev/sdd on ceph-master01
[ceph-master01][DEBUG ] connection detected need for sudo
[ceph-master01][DEBUG ] connected to host: ceph-master01 
[ceph-master01][DEBUG ] detect platform information from remote host
[ceph-master01][DEBUG ] detect machine type
[ceph-master01][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO  ] Distro info: Ubuntu 18.04 bionic
[ceph-master01][DEBUG ] zeroing last few blocks of device
[ceph-master01][DEBUG ] find the location of an executable
[ceph-master01][INFO  ] Running command: sudo /usr/sbin/ceph-volume lvm zap /dev/sdd
[ceph-master01][WARNIN] --> Zapping: /dev/sdd
[ceph-master01][WARNIN] --> Zapping lvm member /dev/sdd. lv_path is /dev/ceph-657ad072-e5d8-4812-a561-19cac0b02e0c/osd-block-b05d277c-b899-4f81-9d00-e7a8cf46b357
[ceph-master01][WARNIN] Running command: /bin/dd if=/dev/zero of=/dev/ceph-657ad072-e5d8-4812-a561-19cac0b02e0c/osd-block-b05d277c-b899-4f81-9d00-e7a8cf46b357 bs=1M count=10 conv=fsync
[ceph-master01][WARNIN]  stderr: 10+0 records in
[ceph-master01][WARNIN] 10+0 records out
[ceph-master01][WARNIN]  stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0201594 s, 520 MB/s
[ceph-master01][WARNIN] --> --destroy was not specified, but zapping a whole device will remove the partition table
[ceph-master01][WARNIN]  stderr: wipefs: error: /dev/sdd: probing initialization failed: Device or resource busy
[ceph-master01][WARNIN] --> failed to wipefs device, will try again to workaround probable race condition
[ceph-master01][WARNIN]  stderr: wipefs: error: /dev/sdd: probing initialization failed: Device or resource busy
[ceph-master01][WARNIN] --> failed to wipefs device, will try again to workaround probable race condition
[ceph-master01][WARNIN]  stderr: wipefs: error: /dev/sdd: probing initialization failed: Device or resource busy
[ceph-master01][WARNIN] --> failed to wipefs device, will try again to workaround probable race condition
[ceph-master01][WARNIN]  stderr: wipefs: error: /dev/sdd: probing initialization failed: Device or resource busy
[ceph-master01][WARNIN] --> failed to wipefs device, will try again to workaround probable race condition
[ceph-master01][WARNIN]  stderr: wipefs: error: /dev/sdd: probing initialization failed: Device or resource busy
[ceph-master01][WARNIN] --> failed to wipefs device, will try again to workaround probable race condition
[ceph-master01][WARNIN]  stderr: wipefs: error: /dev/sdd: probing initialization failed: Device or resource busy
[ceph-master01][WARNIN] --> failed to wipefs device, will try again to workaround probable race condition
[ceph-master01][WARNIN]  stderr: wipefs: error: /dev/sdd: probing initialization failed: Device or resource busy
[ceph-master01][WARNIN] --> failed to wipefs device, will try again to workaround probable race condition
[ceph-master01][WARNIN]  stderr: wipefs: error: /dev/sdd: probing initialization failed: Device or resource busy
[ceph-master01][WARNIN] --> failed to wipefs device, will try again to workaround probable race condition
[ceph-master01][WARNIN] -->  RuntimeError: could not complete wipefs on device: /dev/sdd
[ceph-master01][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: /usr/sbin/ceph-volume lvm zap /dev/sdd

原因:该osd没有完全剔除,不能zap擦除

解决方案:1.取消磁盘挂载(可能是挂载正在使用中)2.完全移除osd

5. 使用zap擦除格式化磁盘时,报错[ERROR ] RuntimeError: Failed to execute command: /usr/sbin/ceph-volume lvm zap /dev/sdd
cephadmin@ceph-master01:/etc/ceph-cluster# ceph-deploy disk zap ceph-master01 /dev/sdd
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadmin/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy disk zap ceph-master01 /dev/sdd
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  debug                         : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : zap
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f8880b7c280>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  host                          : ceph-master01
[ceph_deploy.cli][INFO  ]  func                          : <function disk at 0x7f8880b52350>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  disk                          : ['/dev/sdd']
[ceph_deploy.osd][DEBUG ] zapping /dev/sdd on ceph-master01
[ceph-master01][DEBUG ] connection detected need for sudo
[ceph-master01][DEBUG ] connected to host: ceph-master01 
[ceph-master01][DEBUG ] detect platform information from remote host
[ceph-master01][DEBUG ] detect machine type
[ceph-master01][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO  ] Distro info: Ubuntu 18.04 bionic
[ceph-master01][DEBUG ] zeroing last few blocks of device
[ceph-master01][DEBUG ] find the location of an executable
[ceph-master01][INFO  ] Running command: sudo /usr/sbin/ceph-volume lvm zap /dev/sdd
[ceph-master01][WARNIN] --> Zapping: /dev/sdd
[ceph-master01][WARNIN] --> Zapping lvm member /dev/sdd. lv_path is /dev/ceph-657ad072-e5d8-4812-a561-19cac0b02e0c/osd-block-b05d277c-b899-4f81-9d00-e7a8cf46b357
[ceph-master01][WARNIN] Running command: /bin/dd if=/dev/zero of=/dev/ceph-657ad072-e5d8-4812-a561-19cac0b02e0c/osd-block-b05d277c-b899-4f81-9d00-e7a8cf46b357 bs=1M count=10 conv=fsync
[ceph-master01][WARNIN]  stderr: 10+0 records in
[ceph-master01][WARNIN] 10+0 records out
[ceph-master01][WARNIN] 10485760 bytes (10 MB, 10 MiB) copied, 0.0244706 s, 429 MB/s
[ceph-master01][WARNIN]  stderr: 
[ceph-master01][WARNIN] --> --destroy was not specified, but zapping a whole device will remove the partition table
[ceph-master01][WARNIN]  stderr: wipefs: error: /dev/sdd: probing initialization failed: Device or resource busy
[ceph-master01][WARNIN] --> failed to wipefs device, will try again to workaround probable race condition
[ceph-master01][WARNIN]  stderr: wipefs: error: /dev/sdd: probing initialization failed: Device or resource busy
[ceph-master01][WARNIN] --> failed to wipefs device, will try again to workaround probable race condition
[ceph-master01][WARNIN]  stderr: wipefs: error: /dev/sdd: probing initialization failed: Device or resource busy
[ceph-master01][WARNIN] --> failed to wipefs device, will try again to workaround probable race condition
[ceph-master01][WARNIN]  stderr: wipefs: error: /dev/sdd: probing initialization failed: Device or resource busy
[ceph-master01][WARNIN] --> failed to wipefs device, will try again to workaround probable race condition
[ceph-master01][WARNIN]  stderr: wipefs: error: /dev/sdd: probing initialization failed: Device or resource busy
[ceph-master01][WARNIN] --> failed to wipefs device, will try again to workaround probable race condition
[ceph-master01][WARNIN]  stderr: wipefs: error: /dev/sdd: probing initialization failed: Device or resource busy
[ceph-master01][WARNIN] --> failed to wipefs device, will try again to workaround probable race condition
[ceph-master01][WARNIN]  stderr: wipefs: error: /dev/sdd: probing initialization failed: Device or resource busy
[ceph-master01][WARNIN] --> failed to wipefs device, will try again to workaround probable race condition
[ceph-master01][WARNIN]  stderr: wipefs: error: /dev/sdd: probing initialization failed: Device or resource busy
[ceph-master01][WARNIN] --> failed to wipefs device, will try again to workaround probable race condition
[ceph-master01][WARNIN] -->  RuntimeError: could not complete wipefs on device: /dev/sdd
[ceph-master01][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: /usr/sbin/ceph-volume lvm zap /dev/sdd

解决办法:

手动执行命令擦除磁盘,Device or resource busy,说明磁盘正在使用中

cephadmin@ceph-master01:/etc/ceph-cluster# sudo /usr/sbin/ceph-volume lvm zap /dev/sdd
--> Zapping: /dev/sdd
--> Zapping lvm member /dev/sdd. lv_path is /dev/ceph-657ad072-e5d8-4812-a561-19cac0b02e0c/osd-block-b05d277c-b899-4f81-9d00-e7a8cf46b357
Running command: /bin/dd if=/dev/zero of=/dev/ceph-657ad072-e5d8-4812-a561-19cac0b02e0c/osd-block-b05d277c-b899-4f81-9d00-e7a8cf46b357 bs=1M count=10 conv=fsyncstderr: 10+0 records in
10+0 records outstderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0241062 s, 435 MB/s
--> --destroy was not specified, but zapping a whole device will remove the partition tablestderr: wipefs: error: /dev/sdd: probing initialization failed: Device or resource busy
--> failed to wipefs device, will try again to workaround probable race conditionstderr: wipefs: error: /dev/sdd: probing initialization failed: Device or resource busy
--> failed to wipefs device, will try again to workaround probable race conditionstderr: wipefs: error: /dev/sdd: probing initialization failed: Device or resource busy
--> failed to wipefs device, will try again to workaround probable race conditionstderr: wipefs: error: /dev/sdd: probing initialization failed: Device or resource busy
--> failed to wipefs device, will try again to workaround probable race conditionstderr: wipefs: error: /dev/sdd: probing initialization failed: Device or resource busy
--> failed to wipefs device, will try again to workaround probable race conditionstderr: wipefs: error: /dev/sdd: probing initialization failed: Device or resource busy
--> failed to wipefs device, will try again to workaround probable race conditionstderr: wipefs: error: /dev/sdd: probing initialization failed: Device or resource busy
--> failed to wipefs device, will try again to workaround probable race conditionstderr: wipefs: error: /dev/sdd: probing initialization failed: Device or resource busy
--> failed to wipefs device, will try again to workaround probable race condition
-->  RuntimeError: could not complete wipefs on device: /dev/sdd

方式一:彻底清除磁盘

dd if=/dev/zero of=/dev/sdb bs=512K count=1
reboot

方式二:dmsetup移除

root@ceph-master01:~# lsblk 
sdi                                                                                               8:128  0 447.1G  0 disk 
└─ceph--3511f2c6--2be6--40fd--901d--3b75e433afa5-osd--block--ca994912--f215--4612--97fa--abe33b07985b253:7    0 447.1G  0 lvm
# dmsetup移除
root@ceph-master01:~# dmsetup remove ceph--3511f2c6--2be6--40fd--901d--3b75e433afa5-osd--block--ca994912--f215--4612--97fa--abe33b07985b
6. mons are allowing insecure global_id reclaim
如果AUTH_INSECURE_GLOBAL_ID_RECLAIM还没有引发健康警报并且auth_expose_insecure_global_id_reclaim尚未禁用该设置(默认情况下处于启用状态),则当前没有需要升级的客户端已连接,可以安全地禁止不安全的global_id回收:
ceph config set mon auth_allow_insecure_global_id_reclaim false
# 如果仍然有需要升级的客户端,则可以使用以下方法暂时使此警报停止:
ceph health mute AUTH_INSECURE_GLOBAL_ID_RECLAIM_ALLOWED 1w   # 1 week
# 不建议这样做,但是您也可以无限期地禁用此警告:
ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false
7. 1 pool(s) do not have an application enabled

执行ceph -s 出现pool(s) do not have an application enabled告警

在这里插入图片描述

原因:执行监控检查命令ceph health detail,发现提示application not enabled on pool ‘mypool’

在这里插入图片描述

意思是mypool存储池未设置任何应用(rdb,cephfs等),设置一个即可

cephadmin@ceph-master01:~# ceph osd pool application enable  mypool  rbd
enabled application 'rbd' on pool 'mypool'
cephadmin@ceph-master01:~# ceph -s cluster:id:     f69afe6f-e559-4df7-998a-c5dc3e300209health: HEALTH_WARNclock skew detected on mon.ceph-master02services:mon: 3 daemons, quorum ceph-master01,ceph-master02,ceph-master03 (age 26h)mgr: ceph-master03(active, since 26h), standbys: ceph-master01, ceph-master02osd: 9 osds: 9 up (since 25h), 9 in (since 26h)data:pools:   2 pools, 33 pgsobjects: 1 objects, 100 MiBusage:   370 MiB used, 450 GiB / 450 GiB availpgs:     33 active+clean
8.clock skew detected on mon.ceph-master02

多次执行ntpdate ntp.aliyun.com 同步时间,依然报警事件有误差

在这里插入图片描述

原因:

cephadmin@ceph-master01:~# ceph health detail
HEALTH_WARN clock skew detected on mon.ceph-master02
[WRN] MON_CLOCK_SKEW: clock skew detected on mon.ceph-master02mon.ceph-master02 clock skew 0.0662306s > max 0.05s (latency 0.00108482s)

服务器误差大于0.05秒,就会出现告警

解决办法:

修改默认参数

mon clock drift allowed #监视器间允许的时钟漂移量

mon clock drift warn backoff #时钟偏移警告的退避指数

cephadmin@ceph-master01:/etc/ceph-cluster# vi /etc/ceph/ceph.conf  
[mon]
mon_clock_drift_allowed = 0.10
mon clock drift warn backoff = 10#重启所有mon节点
root@ceph-master01:~# systemctl   restart ceph-mon@ceph-master01.service
root@ceph-master02:~# systemctl   restart ceph-mon@ceph-master02.service
root@ceph-master03:~# systemctl   restart ceph-mon@ceph-master03.service
9. 低内核无法挂载ceph rbd块存储

在这里插入图片描述

环境:

ceph 16.2.10

客户端 ceph-common 15.2.17

客户端内核 3.10.0-862

客户端操作系统 centos 7.5

所有的文章都告诉你 “低内核版本挂载rbd存储,rbd创建时只开放layering特性就能挂载”

rbd create myimg2 --size 3G --pool myrbd1 --image-format 2 --image-feature layering

但是低版本内核也分低版本,超低版本,在执行挂载命令 rbd -p myrbd1 map myimg2

centos 7.5 操作系统默认没升级内核的3.10.0-862版本不行。

centos 7.7 操作系统默认没升级内核的3.10.0-1062.4.3版本可以。

都是3.10版本,但小版本号必须要高于3.10.0-862

10. 1 filesystem is online with fewer MDS than max_mds

在这里插入图片描述

原因:

mds服务没有创建启动

解决办法:

ceph-deploy mds create  ceph-master01 ceph-master02 ceph-master03

本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如若转载,请注明出处:http://www.rhkb.cn/news/399329.html

如若内容造成侵权/违法违规/事实不符,请联系长河编程网进行投诉反馈email:809451989@qq.com,一经查实,立即删除!

相关文章

子串 前缀和 | Java | (hot100) 力扣560. 和为K的子数组

560. 和为K的子数组 暴力法&#xff08;连暴力法都没想出来……&#xff09; class Solution {public int subarraySum(int[] nums, int k) {int count0;int len nums.length;for(int i0; i<len; i) {int sum0;for(int ji; j<len; j) {sumnums[j];if(sum k) {count;}…

mysql注入-字符编码技巧

一、环境搭建 创建数据表 CREATE TABLE mysql_Bian_Man (id int(10) unsigned NOT NULL AUTO_INCREMENT,username varchar(255) COLLATE latin1_general_ci NOT NULL,password varchar(255) COLLATE latin1_general_ci NOT NULL,PRIMARY KEY (id) ) ENGINEMyISAM AUTO_INCREME…

Redis 缓存预热、雪崩、穿透、击穿

缓存预热 缓存预热是什么 缓存预热就是系统上线后&#xff0c;提前将相关的缓存数据直接加载到缓存系统。避免在用户请求的时候&#xff0c;先查询数据库&#xff0c;然后再将数据缓存的问题&#xff01;用户直接查询事先被预热的缓存数据&#xff01;解决方案 使用 PostConstr…

LeetCode旋转图像

题目描述&#xff1a; 给定一个 n n 的二维矩阵 matrix 表示一个图像。请你将图像顺时针旋转 90 度。 你必须在 原地 旋转图像&#xff0c;这意味着你需要直接修改输入的二维矩阵。请不要使用另一个矩阵来旋转图像。 示例 1&#xff1a; 输入&#xff1a;matrix [[1,2,3]…

opencv实战项目七:帧差法获取运动车辆蒙版

文章目录 一、简介二、实现方案三、算法实现步骤3.1 取出视频中间帧3.2 帧差法形成运动蒙版&#xff1a; 四、代码整体实现五&#xff1a;效果 一、简介 在智能交通系统中&#xff0c;车辆检测是一项重要的技术&#xff0c;而通常情况下一张图片中无用信息过多会带来额外的计算…

Linux--C语言之循环结构

文章目录 一、循环结构&#xff08;一&#xff09;循环的概念&#xff08;二&#xff09;循环的类型&#xff08;三&#xff09;循环的构成&#xff08;四&#xff09;当型循环的实现while死循环 &#xff08;五&#xff09;for...总结死循环 &#xff08;七&#xff09;循环实…

机器学习——逻辑回归(学习笔记)

目录 一、认识逻辑回归 二、二元逻辑回归&#xff08;LogisticRegression&#xff09; 1. 损失函数 2. 正则化 3. 梯度下降 4. 二元回归与多元回归 三、sklearn中的逻辑回归&#xff08;自查&#xff09; 1. 分类 2. 参数列表 3. 属性列表 4. 接口列表 四、逻辑回归…

大厂面试题分享第一期

大厂面试题分享第一期 Redis持久化方式AOF优缺点RDB优缺点 如何保证Redis和Myql的一致性索引下推输入url到浏览器发生了什么ReentranLock底层原理SpringBoot 的启动流程 Redis持久化方式 Redis提供了两种主要的持久化机制&#xff0c;分别是AOF&#xff08;Append-Only File&a…

Python 数据可视化,怎么选出合适数据的图表

数据可视化最佳实践 1. 引言&#xff1a;为什么数据可视化最佳实践很重要 数据可视化是数据分析和决策过程中不可或缺的一部分。通过有效的可视化&#xff0c;复杂的数据可以转化为易于理解的信息&#xff0c;从而帮助观众快速做出正确的判断。然而&#xff0c;糟糕的可视化可…

单片机IO灌入5V电压导致其他IO电压测量到大于供电电压问题

最近用GD32F103RCT6做项目&#xff0c;用了3个485收发器&#xff0c;都是直接接在单片机IO上的。 485收发器是5V供电的&#xff0c;这个时候就出现5V电平和3.3V电平兼容的问题了。 一开始只用了PA10、PC11这两个串口&#xff0c;他俩是兼容5V的&#xff0c;从手册可以看出IO最…

企业源代码也需要加密!十款好用的源代码加密软件排行榜

在当今竞争激烈的商业环境中&#xff0c;企业的源代码是其核心资产之一。为了保护这些宝贵的知识产权不被泄露&#xff0c;源代码加密成为了众多企业的重要举措。2024 年&#xff0c;市面上出现了众多功能强大的源代码加密软件。接下来&#xff0c;就让我们一同来探索十款备受好…

DockerCompose编排Nginx+Mysql并实现Nginx配置Mysql(TCP协议)负载均衡

场景 Nginx配置实例-负载均衡实例&#xff1a;平均访问多台服务器&#xff1a; Nginx配置实例-负载均衡实例&#xff1a;平均访问多台服务器_我想访问五个服务器的信息用nginx怎么做-CSDN博客 以上实现Nginx的http协议的负载均衡&#xff0c;如果使用Nginx实现TCP协议的负载…

Java的JVM中的概念之——新生代和老年代

JVM新生代和老年代是JVM中非常重要的概念&#xff0c;那么他们在JVM中扮演者什么样的角色和含义呢&#xff1f; 在Java虚拟机&#xff08;JVM&#xff09;的垃圾回收&#xff08;GC&#xff09;中&#xff0c;内存被分为不同的区域&#xff0c;其中两个主要区域是新生代&#…

PHP餐厅点餐系统小程序源码

&#x1f37d;️【餐厅点餐新纪元&#xff0c;点餐系统让用餐更便捷&#xff01;】&#x1f4f1; &#x1f50d; 一键浏览&#xff0c;菜单尽在掌握 &#x1f4f1; 走进餐厅&#xff0c;无需再担心找不到服务员或菜单被抢光&#xff01;餐厅点餐系统让你轻松扫描桌上的二维码…

HarmonyOS Developer之图片帧动画播放器

创建image-animator组件 在pages/index目录下的hml文件中创建一个image-animator组件&#xff0c;css文件中编写组件样式&#xff0c;js文件中引用图片。 设置image-animator组件属性 添加iteration&#xff08;播放次数&#xff09;、reverse&#xff08;播放顺序&#xf…

LVS原理及实例

目录 LVS原理 LVS概念 lvs集群的类型 lvs-nat 解释 传输过程 lvs-dr 解释 传输过程 特点 lvs-tun LVS&#xff08;Linux Virtual Server&#xff09;常见的调度算法 防火墙标记&#xff08;Firewall Marking&#xff09;结合轮询调度 实战案例 lvs的nat模式配置 …

使用Linux实现FTP云盘1

关于FTP服务器 FTP&#xff08;文件传输协议&#xff09;服务器是在互联网上提供文件存储和访问服务的计算机&#xff0c;它们依照FTP 协议提供服务。 FTP是File Transfer Protocol(文件传输协议)。 程序运行&#xff0c;服务端不断接收客户端指令&#xff0c;服务 端可同时处…

提取含有特定字符的行和列grep函数(含有替换)

目录 ①grep提取含有特定字符的列 ②grep提取含有特定字符的行 R语言进行字符的替换和删减gsub&#xff0c;substr函数R语言进行字符的替换和删减gsub&#xff0c;substr函数_r语言数据框字符替换-CSDN博客 ①grep提取含有特定字符的列 在一个dataframe中&#xff0c;需要提…

Element学习(axios异步加载数据、案例操作)(5)

1、这次学习的是上次还未完成好的恶element案例&#xff0c;对列表数据的异步加载&#xff0c;并渲染展示。 ——>axios来发送异步请求 &#xff08;1&#xff09; &#xff08;2&#xff09;在vue当中安装axios &#xff08;注意在当前的项目目录&#xff0c;并且安装完之后…

Xcode 在原生集成flutter项目

笔者公司有一个从2017年就开始开发的iOS和安卓原生项目&#xff0c;现在计划从外到内开始进行项目迁徙。 1》从gitee拉取flutter端的代码&#xff1b;&#xff08;Android报错Exception: Podfile missing&#xff09; 2》替换Xcode里的cocopods里Podfile的路径 然后报警 然后…