DevStack 部署 OpenStack 多节点

DevStack 支持OpenStack多节点部署，下面以一个控制节点和一个计算节点为例，介绍多节点多网卡部署流程。

官方文档：

https://docs.openstack.org/devstack/latest/guides/multinode-lab.html

https://docs.openstack.org/neutron/latest/contributor/testing/ml2_ovn_devstack.html

节点规划

基于Ubuntu 22.04 操作系统，使用 Devstack 部署工具搭建多节点的 OpenStack 开发环境。

节点规划，devstack21 为控制节点，devstack22为计算节点。

主机名	节点IP	CPU	内存	磁盘	网卡	OS	角色
devstack21	192.168.93.21	4C	8G	100G	2	Ubuntu22.04	控制节点
devstack22	192.168.93.22	4C	8G	100G	2	Ubuntu22.04	计算节点

网段规划，规划两个网络平面，对应管理网络和外部网络。租户网络（geneve隧道网络）与管理网络合并。

网段	网关	网卡模式	网络类型
192.168.93.0/24	192.168.93.2	NAT	管理网络
192.168.72.0/24	192.168.72.8	桥接	外部网络

网卡规划，每个节点两块网卡，分别对应管理网络和外部网络。

主机名	网卡	IP地址	网卡模式	网络类型
devstack01	ens33	192.168.93.21/24	NAT	管理网络
	ens37	NA	桥接	外部网络
devstack02	ens33	192.168.93.22/24	NAT	管理网络
	ens37	NA	桥接	外部网络

网络架构图如下：
在这里插入图片描述
说明：

管理网络需要能够访问外网，部署时需要通过该网络下载依赖，所以选择NAT模式
外部网络也可以选择hostonly模式，外部网络不一定是公网

测试环境： VMware Workstation 17
在这里插入图片描述
说明：

准备两块网卡，注意配置网卡模式
也可以勾选处理器–>虚拟化引擎选项，提高openstack创建的实例性能

前置准备

以下操作在所有节点执行。

配置http代理（可选，需要科技）

cat >/etc/profile.d/proxy.sh<<EOF
export http_proxy="http://192.168.93.1:7890"
export https_proxy="http://192.168.93.1:7890"
export no_proxy="192.168.93.0/24,10.0.0.0/8,192.168.72.0/24,localhost,127.0.0.0/8,.localdomain"
EOF
source /etc/profile

说明：否则你可能需要自行配置国内的apt源、pip源，例如阿里云国内源，切换git仓库源码地址为trystack源，另外devstack直接下载二进制文件时你可能只能使用export http_proxy=xxx这种方式，例如stack.sh脚本会执行以下操作。

wget --progress=dot:giga -t 2 -c https://github.com/etcd-io/etcd/releases/download/v3.4.27/etcd-v3.4.27-linux-amd64.tar.gz -O /opt/stack/devstack/files/etcd-v3.4.27-linux-amd64.tar.gz

系统更新

apt update -y && apt upgrade -y

配置主机名

hostnamectl set-hostname devstack21
hostnamectl set-hostname devstack22

配置时间同步

apt install -y chrony
timedatectl set-timezone Asia/Shanghai

devstack21节点网卡配置示例:

root@devstack21:~# cat /etc/netplan/00-installer-config.yaml
network:
  ethernets:
    ens33:
      addresses:
      - 192.168.93.21/24
      nameservers:
        addresses:
        - 223.5.5.5
        - 223.6.6.6
        search: []
      routes:
        - to: default
          via: 192.168.93.2
    ens37:
      dhcp4: false
  version: 2

devstack21网卡信息示例如下:

root@devstack21:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:b0:da:db brd ff:ff:ff:ff:ff:ff
    altname enp2s1
    inet 192.168.93.21/24 brd 192.168.93.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:feb0:dadb/64 scope link 
       valid_lft forever preferred_lft forever
3: ens37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:b0:da:e5 brd ff:ff:ff:ff:ff:ff
    altname enp2s5
    inet6 fe80::20c:29ff:feb0:dae5/64 scope link 
       valid_lft forever preferred_lft forever

说明：devstack22节点配置类似，仅IP不同。

添加 Stack 用户

以下操作在所有节点执行。

sudo useradd -s /bin/bash -d /opt/stack -m stack

确保 stack 用户的主目录对所有人都具有可执行权限，因为基于 RHEL 的发行版使用 700 创建它，而 Ubuntu 21.04+ 使用 750 创建它，这可能会在运行期间导致问题部署。

sudo chmod +x /opt/stack

由于该用户将对您的系统进行许多更改，因此它应该具有 sudo 权限：

echo "stack ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/stack
sudo -u stack -i

此时会切换到stack用户视图下。

下载 DevStack

以下操作在所有节点执行。

下载devstack源码

stack@devstack21:~$ pwd
/opt/stack
stack@devstack21:~$ git clone https://opendev.org/openstack/devstack

下载neutron源码

stack@devstack21:~$ pwd
/opt/stack
stack@devstack21:~$ git clone https://opendev.org/openstack/neutron.git

安装控制节点

以下操作在控制节点执行。

配置 DevStack 以使用 OVN 驱动程序。OVN 驱动程序附带了一个示例 DevStack 配置文件，位于neutron项目代码下。

复制neutron示例文件到devstack目录下：

stack@devstack21:~$ cd devstack
stack@devstack21:~/devstack$ pwd
/opt/stack/devstack
stack@devstack21:~$ egrep -v "^#|^$" ../neutron/devstack/ovn-local.conf.sample > ./local.conf

配置文件修改后如下

cat >local.conf<<'EOF'
[[local|localrc]]
# git_base
GIT_BASE="https://github.com"
#GIT_BASE=https://opendev.org
#GIT_BASE=http://git.trystack.cn

# password
ADMIN_PASSWORD=secret
DATABASE_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD
SERVICE_TOKEN=$ADMIN_PASSWORD

HOST_IP=192.168.93.21
SERVICE_HOST=192.168.93.21
MYSQL_HOST=$SERVICE_HOST
RABBIT_HOST=$SERVICE_HOST
GLANCE_HOSTPORT=$SERVICE_HOST:9292

Q_AGENT=ovn
Q_ML2_PLUGIN_MECHANISM_DRIVERS=ovn,logger
Q_ML2_PLUGIN_TYPE_DRIVERS=local,flat,vlan,geneve
Q_ML2_TENANT_NETWORK_TYPE="geneve"
LOGFILE=$DEST/logs/stack.sh.log
enable_service ovn-northd
enable_service ovn-controller
enable_service q-ovn-metadata-agent
enable_service q-svc
disable_service q-agt
disable_service q-l3
disable_service q-dhcp
disable_service q-meta
enable_plugin neutron https://github.com/openstack/neutron
enable_service q-trunk
enable_service q-dns
enable_service q-port-forwarding
enable_service q-qos
enable_service neutron-segments
enable_service q-log
enable_plugin neutron-tempest-plugin https://github.com/openstack/neutron-tempest-plugin
enable_service horizon
disable_service cinder c-sch c-api c-vol
OVN_BUILD_MODULES=True
ENABLE_CHASSIS_AS_GW=True

## Neutron options
FLOATING_RANGE="192.168.72.0/24"
Q_FLOATING_ALLOCATION_POOL=start=192.168.72.110,end=192.168.72.120
PUBLIC_NETWORK_GATEWAY="192.168.72.8"
PUBLIC_INTERFACE=ens37

# Open vSwitch provider networking configuration
Q_USE_PROVIDERNET_FOR_PUBLIC=True
OVS_PHYSICAL_BRIDGE=br-ex
PUBLIC_BRIDGE=br-ex
OVS_BRIDGE_MAPPINGS=public:br-ex

[[post-config|$NOVA_CONF]]
[scheduler]
discover_hosts_in_cells_interval = 2
EOF

主要配置参数说明：

GIT_BASE: 网络不好时可尝试使用GIT_BASE=http://git.trystack.cn
HOST_IP: 本机IP地址
FLOATING_RANGE：用于 GUEST VM 访问外部网络，以通过向GUEST VM分配 FLOATING IP 来从外部网络访问GUEST VM。
Q_FLOATING_ALLOCATION_POOL：为devstack分配指定的外部IP地址范围，因为本地其他机器也在使用该物理网段。
PUBLIC_INTERFACE: 外部网络网卡名称
PUBLIC_NETWORK_GATEWAY：外部网络网关地址
Q_USE_PROVIDERNET_FOR_PUBLIC：允许配置Neutron L3 Agent的外部接口使用provider网络，从而实现对OpenStack云实例的公共网络访问。
OVS_PHYSICAL_BRIDGE：指定了一个OVS网桥br-ex作为物理网桥，外部网络的流量将通过这个OVS网桥br-ex，底层物理网卡将连接到br-ex网桥上
PUBLIC_BRIDGE：直接将之前定义的br-ex网桥指定为公共网络的网桥，所有外部网络的流量都将通过br-ex网桥
OVS_BRIDGE_MAPPINGS：映射外部网络到之前定义的br-ex网桥

安装控制节点

$ ./stack.sh

确认安装成功

......
=========================
DevStack Component Timing
 (times are in seconds)  
=========================
wait_for_service      12
async_wait           102
osc                  180
apt-get               71
test_with_retry        5
dbsync                 7
pip_install          128
apt-get-update         1
run_process           27
git_timed             85
-------------------------
Unaccounted time     184
=========================
Total runtime        802

=================
 Async summary
=================
 Time spent in the background minus waits: 322 sec
 Elapsed time: 802 sec
 Time if we did everything serially: 1124 sec
 Speedup:  1.4015


Post-stack database query stats:
+------------+-----------+-------+
| db         | op        | count |
+------------+-----------+-------+
| keystone   | SELECT    | 31269 |
| keystone   | INSERT    |    82 |
| neutron    | SELECT    |  4482 |
| neutron    | CREATE    |     1 |
| neutron    | SHOW      |     4 |
| neutron    | INSERT    |  4113 |
| neutron    | DELETE    |    27 |
| neutron    | UPDATE    |   116 |
| placement  | SELECT    |    38 |
| placement  | INSERT    |    55 |
| placement  | SET       |     1 |
| nova_api   | SELECT    |   305 |
| nova_cell0 | SELECT    |    65 |
| nova_cell1 | SELECT    |   317 |
| nova_cell0 | INSERT    |     5 |
| nova_cell0 | UPDATE    |     3 |
| nova_cell1 | INSERT    |     4 |
| nova_cell1 | UPDATE    |    20 |
| placement  | UPDATE    |     3 |
| glance     | SELECT    |    57 |
| glance     | INSERT    |    14 |
| glance     | UPDATE    |     2 |
| nova_api   | INSERT    |    23 |
| nova_api   | SAVEPOINT |    11 |
| nova_api   | RELEASE   |    11 |
+------------+-----------+-------+



This is your host IP address: 192.168.93.21
This is your host IPv6 address: ::1
Horizon is now available at http://192.168.93.21/dashboard
Keystone is serving at http://192.168.93.21/identity/
The default users are: admin and demo
The password: secret

Services are running under systemd unit files.
For more information see: 
https://docs.openstack.org/devstack/latest/systemd.html

DevStack Version: 2024.1
Change: f4f09416ca8de09820b52599c7bcd2911b407466 Merge "Drop unused environments for TripleO and heat agents" 2024-03-19 18:55:14 +0000
OS Version: Ubuntu 22.04 jammy

修改网卡配置，增加bridges部分，使网桥开机自动启动

stack@devstack21:~/devstack$ sudo vim /etc/netplan/00-installer-config.yaml
# This is the network config written by 'subiquity'
network:
  ethernets:
    ens33:
      addresses:
      - 192.168.93.21/24
      nameservers:
        addresses:
        - 223.5.5.5
        - 223.6.6.6
        search: []
      routes:
        - to: default
          via: 192.168.93.2
    ens37:
      dhcp4: false
  bridges:
    br-ex:
      openvswitch: {}
      dhcp4: false
  version: 2

应用网卡配置

stack@devstack21:~/devstack$ sudo netplan apply 
stack@devstack21:~/devstack$

查看网卡信息

stack@devstack21:~/devstack$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:b0:da:db brd ff:ff:ff:ff:ff:ff
    altname enp2s1
    inet 192.168.93.21/24 brd 192.168.93.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:feb0:dadb/64 scope link 
       valid_lft forever preferred_lft forever
3: ens37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UP group default qlen 1000
    link/ether 00:0c:29:b0:da:e5 brd ff:ff:ff:ff:ff:ff
    altname enp2s5
    inet6 fe80::20c:29ff:feb0:dae5/64 scope link 
       valid_lft forever preferred_lft forever
4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 22:0d:02:9f:90:1d brd ff:ff:ff:ff:ff:ff
5: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether f2:ad:f1:af:54:9a brd ff:ff:ff:ff:ff:ff
6: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:f2:bf:a4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
7: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 00:0c:29:b0:da:e5 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::20c:29ff:feb0:dae5/64 scope link 
       valid_lft forever preferred_lft forever

查看网桥信息

stack@devstack21:~/devstack$ sudo ovs-vsctl show
8fc32620-e49f-4e8a-87a1-675d4185e1aa
    Manager "ptcp:6640:127.0.0.1"
        is_connected: true
    Bridge br-int
        fail_mode: secure
        datapath_type: system
        Port br-int
            Interface br-int
                type: internal
        Port patch-br-int-to-provnet-4ff27886-1c19-4ba4-bd52-05e6c6524575
            Interface patch-br-int-to-provnet-4ff27886-1c19-4ba4-bd52-05e6c6524575
                type: patch
                options: {peer=patch-provnet-4ff27886-1c19-4ba4-bd52-05e6c6524575-to-br-int}
    Bridge br-ex
        fail_mode: standalone
        Port br-ex
            Interface br-ex
                type: internal
        Port ens37
            Interface ens37
        Port patch-provnet-4ff27886-1c19-4ba4-bd52-05e6c6524575-to-br-int
            Interface patch-provnet-4ff27886-1c19-4ba4-bd52-05e6c6524575-to-br-int
                type: patch
                options: {peer=patch-br-int-to-provnet-4ff27886-1c19-4ba4-bd52-05e6c6524575}
    ovs_version: "2.17.9"

此时已经可以登录dashboard，并创建虚拟机，默认账号为admin/secret。

http://192.168.93.21/

安装计算节点

以下操作在计算节点执行。

从neutron文件夹下复制计算节点示例配置文件

stack@devstack22:~$ pwd
/opt/stack
stack@devstack22:~$ cd devstack
stack@devstack22:~$ egrep -v "^#|^$" ../neutron/devstack/ovn-compute-local.conf.sample > ./local.conf

配置文件修改后如下：

cat >local.conf<<'EOF'
[[local|localrc]]
# git_base
GIT_BASE="https://github.com"
#GIT_BASE=https://opendev.org
#GIT_BASE=http://git.trystack.cn

# password
ADMIN_PASSWORD=secret
DATABASE_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD
SERVICE_TOKEN=$ADMIN_PASSWORD

# OVN
Q_AGENT=ovn
NEUTRON_AGENT=$Q_AGENT
Q_ML2_PLUGIN_MECHANISM_DRIVERS=ovn,logger
Q_ML2_PLUGIN_TYPE_DRIVERS=local,flat,vlan,geneve
Q_ML2_TENANT_NETWORK_TYPE="geneve"
LOGFILE=$DEST/logs/stack.sh.log
enable_plugin neutron https://opendev.org/openstack/neutron
disable_all_services
enable_service n-cpu
enable_service placement-client
enable_service ovn-controller
enable_service q-ovn-metadata-agent
SERVICE_HOST=192.168.93.21
RABBIT_HOST=$SERVICE_HOST
Q_HOST=$SERVICE_HOST
OVN_SB_REMOTE=tcp:$SERVICE_HOST:6642
OVN_BUILD_MODULES=True
HOST_IP=192.168.93.22
NOVA_VNC_ENABLED=True
NOVNCPROXY_URL=http://$SERVICE_HOST:6080/vnc_lite.html
VNCSERVER_LISTEN=$HOST_IP
VNCSERVER_PROXYCLIENT_ADDRESS=$VNCSERVER_LISTEN

## Neutron options
Q_USE_SECGROUP=True
FLOATING_RANGE="192.168.72.0/24"
Q_FLOATING_ALLOCATION_POOL=start=192.168.72.110,end=192.168.72.120
PUBLIC_NETWORK_GATEWAY="192.168.72.8"
PUBLIC_INTERFACE=ens37

# Open vSwitch provider networking configuration
Q_USE_PROVIDERNET_FOR_PUBLIC=True
OVS_PHYSICAL_BRIDGE=br-ex
PUBLIC_BRIDGE=br-ex
OVS_BRIDGE_MAPPINGS=public:br-ex
EOF

主要配置参数说明：

HOST_IP: 本机IP地址
SERVICE_HOST：必须是控制节点IP地址

安装计算节点

$ ./stack.sh

确认安装完成

......
=========================
DevStack Component Timing
 (times are in seconds)  
=========================
wait_for_service       7
async_wait             0
osc                    0
apt-get              264
pip_install          103
apt-get-update         1
run_process            4
git_timed             45
-------------------------
Unaccounted time     100
=========================
Total runtime        524

=================
 Async summary
=================
 Time spent in the background minus waits: 1 sec
 Elapsed time: 524 sec
 Time if we did everything serially: 525 sec
 Speedup:  1.00191



This is your host IP address: 192.168.93.22
This is your host IPv6 address: ::1

Services are running under systemd unit files.
For more information see: 
https://docs.openstack.org/devstack/latest/systemd.html

DevStack Version: 2024.1
Change: f4f09416ca8de09820b52599c7bcd2911b407466 Merge "Drop unused environments for TripleO and heat agents" 2024-03-19 18:55:14 +0000
OS Version: Ubuntu 22.04 jammy

修改网卡配置，增加bridges部分，使网桥开机自动启动

stack@devstack22:~/devstack$ sudo vim /etc/netplan/00-installer-config.yaml
# This is the network config written by 'subiquity'
network:
  ethernets:
    ens33:
      addresses:
      - 192.168.93.22/24
      nameservers:
        addresses:
        - 223.5.5.5
        - 223.6.6.6
        search: []
      routes:
        - to: default
          via: 192.168.93.2
    ens37:
      dhcp4: false
  bridges:
    br-ex:
      openvswitch: {}
      dhcp4: false
  version: 2

应用网卡配置

stack@devstack22:~/devstack$ sudo netplan apply 
stack@devstack22:~/devstack$

查看网卡信息

stack@devstack22:~/devstack$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:31:a7:2c brd ff:ff:ff:ff:ff:ff
    altname enp2s1
    inet 192.168.93.22/24 brd 192.168.93.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe31:a72c/64 scope link 
       valid_lft forever preferred_lft forever
3: ens37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UP group default qlen 1000
    link/ether 00:0c:29:31:a7:36 brd ff:ff:ff:ff:ff:ff
    altname enp2s5
    inet6 fe80::20c:29ff:fe31:a736/64 scope link 
       valid_lft forever preferred_lft forever
4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 22:0d:02:9f:90:1d brd ff:ff:ff:ff:ff:ff
5: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 06:0d:1b:71:58:0e brd ff:ff:ff:ff:ff:ff
6: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:42:16:35 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
7: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 00:0c:29:31:a7:36 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::20c:29ff:fe31:a736/64 scope link 
       valid_lft forever preferred_lft forever
8: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000
    link/ether 1e:dd:eb:ff:7a:66 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::f4be:3fff:fe9b:6d89/64 scope link 
       valid_lft forever preferred_lft forever

查看网桥信息

stack@devstack22:~/devstack$ sudo ovs-vsctl show
15b21aff-829b-4c9f-8ef6-1fa372eddb3a
    Manager "ptcp:6640:127.0.0.1"
        is_connected: true
    Bridge br-ex
        fail_mode: standalone
        Port br-ex
            Interface br-ex
                type: internal
        Port ens37
            Interface ens37
        Port patch-provnet-4ff27886-1c19-4ba4-bd52-05e6c6524575-to-br-int
            Interface patch-provnet-4ff27886-1c19-4ba4-bd52-05e6c6524575-to-br-int
                type: patch
                options: {peer=patch-br-int-to-provnet-4ff27886-1c19-4ba4-bd52-05e6c6524575}
    Bridge br-int
        fail_mode: secure
        datapath_type: system
        Port patch-br-int-to-provnet-4ff27886-1c19-4ba4-bd52-05e6c6524575
            Interface patch-br-int-to-provnet-4ff27886-1c19-4ba4-bd52-05e6c6524575
                type: patch
                options: {peer=patch-provnet-4ff27886-1c19-4ba4-bd52-05e6c6524575-to-br-int}
        Port ovn-2018dc-0
            Interface ovn-2018dc-0
                type: geneve
                options: {csum="true", key=flow, remote_ip="192.168.93.21"}
                bfd_status: {diagnostic="No Diagnostic", flap_count="1", forwarding="true", remote_diagnostic="Neighbor Signaled Session Down", remote_state=up, state=up}
        Port br-int
            Interface br-int
                type: internal
    ovs_version: "2.17.9"

节点发现

以下操作在控制节点执行。

切换到管理员用户

stack@devstack21:~/devstack$ . openrc admin admin
WARNING: setting legacy OS_TENANT_NAME to support cli tools.
stack@devstack21:~/devstack$

执行节点发现脚本

stack@devstack21:~/devstack$ pwd
/opt/stack/devstack
stack@devstack21:~/devstack$ ./tools/discover_hosts.sh

执行结果如下

stack@devstack21:~/devstack$ ./tools/discover_hosts.sh 
INFO dbcounter [None req-1bb45b78-fd14-4fda-bd70-8cb20326bb8e None None] Registered counter for database nova_api
DEBUG dbcounter [-] [131423] Writer thread running {{(pid=131423) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:102}}
Found 2 cell mappings.
Skipping cell0 since it does not contain hosts.
Getting computes from cell 'cell1': ac941876-1395-4846-befe-ee823eb14d5e
INFO dbcounter [None req-1bb45b78-fd14-4fda-bd70-8cb20326bb8e None None] Registered counter for database nova_cell1
DEBUG dbcounter [-] [131423] Writer thread running {{(pid=131423) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:102}}
Found 0 unmapped computes in cell: ac941876-1395-4846-befe-ee823eb14d5e
stack@devstack21:~/devstack$

查看新发现的hypervisor

stack@devstack21:~/devstack$ openstack hypervisor list
+--------------------------------------+---------------------+-----------------+---------------+-------+
| ID                                   | Hypervisor Hostname | Hypervisor Type | Host IP       | State |
+--------------------------------------+---------------------+-----------------+---------------+-------+
| be439833-6735-4f3e-9ddd-3e37f768e18d | devstack21          | QEMU            | 192.168.93.21 | up    |
| a2b491c5-8124-4c49-b279-e327d98bb412 | devstack22          | QEMU            | 192.168.93.22 | up    |
+--------------------------------------+---------------------+-----------------+---------------+-------+
stack@devstack21:~/devstack$

还可以查看 OVN 和 OVS 以查看第二个主机已出现。例如，OVN_Southbound 数据库的 Chassis 表中将有第二个条目。可以使用 ovn-sbctl 实用程序列出chassis、其配置以及绑定到每个chassis的端口：

stack@devstack21:~/devstack$ sudo ovn-sbctl show
Chassis "2018dc69-862f-45d9-ad7d-0fc4dc5d08c4"
    hostname: devstack21
    Encap geneve
        ip: "192.168.93.21"
        options: {csum="true"}
    Port_Binding cr-lrp-7ff9bc36-6877-44c2-aee9-d9d8eb22c8e0
Chassis "1e889c0c-2509-4cc5-ad6e-f9d3bdfb5c40"
    hostname: devstack22
    Encap geneve
        ip: "192.168.93.22"
        options: {csum="true"}

还可以看到创建到另一个计算节点的隧道：

stack@devstack21:~/devstack$ sudo ovs-vsctl show
......
    Bridge br-int
        fail_mode: secure
        datapath_type: system
        Port br-int
            Interface br-int
                type: internal
        Port ovn-1e889c-0
            Interface ovn-1e889c-0
                type: geneve
                options: {csum="true", key=flow, remote_ip="192.168.93.22"}
......

两节点openstack已经部署完成，你可以通过openstack CLI或登录dashboard创建虚拟机实例进一步测试。

创建虚拟机

在本节中，我们将使用openstack CLI逐步完成创建两个虚拟机的步骤，这些虚拟机具有连接到 private Neutron 网络的虚拟 NIC。

DevStack 默认使用 libvirt 作为 Nova 后端。如果 KVM 可用，则会使用它。否则，它只会运行 qemu 模拟来宾。这对于我们的测试来说非常合适，因为我们只需要这些虚拟机能够发送和接收少量流量，因此性能并不是很重要。

虚拟机创建流程

1、切换到demo用户

stack@devstack21:~/devstack$ pwd
/opt/stack/devstack
stack@devstack21:~/devstack$ . openrc demo demo
WARNING: setting legacy OS_TENANT_NAME to support cli tools.
stack@devstack21:~/devstack$

2、获取网络 UUID。

查看可用的网络，devstack默认为我们创建了三个网络，我们将使用 private 租户网络创建虚拟机

stack@devstack21:~/devstack$ openstack network list
+--------------------------------------+---------+----------------------------------------------------------------------------+
| ID                                   | Name    | Subnets                                                                    |
+--------------------------------------+---------+----------------------------------------------------------------------------+
| 6211312f-ab8e-4346-bc58-aed8a63d4f62 | public  | 0cf53da9-2c68-4b7d-8043-8d682fe34b23, 8a1dbf88-3caa-4a88-afa4-6fb6823cb224 |
| 8b8cdb32-c009-49e1-88d4-3d58198e6d6c | private | 24bfad1c-46ab-45a8-b607-da0ed8ea0b3c, 912dae4c-691a-44be-afb1-252c19ebfe24 |
| 98a54015-c7ac-4825-80a8-b3364d41db43 | shared  | 0e754263-61ee-496a-8187-3103af900846                                       |
+--------------------------------------+---------+----------------------------------------------------------------------------+

首先从之前的 openstack network list 输出中获取 private 网络的 UUID 并将其保存：

PRIVATE_NET_ID=$(openstack network show private -c id -f value)

3、创建 SSH 密钥对。

接下来在 Nova 中创建 SSH 密钥对。稍后，当我们启动虚拟机时，我们会要求将公钥放入虚拟机中，以便我们可以通过 SSH 访问它。

openstack keypair create demo > id_rsa_demo
chmod 600 id_rsa_demo

查看我们创建的keypair

stack@devstack21:~/devstack$ openstack keypair list
+------+-------------------------------------------------+------+
| Name | Fingerprint                                     | Type |
+------+-------------------------------------------------+------+
| demo | dc:8b:e9:ce:37:c9:bd:16:57:df:c0:20:dc:8c:64:d0 | ssh  |
+------+-------------------------------------------------+------+
stack@devstack21:~/devstack$

3、选择一种flavor，devstack默认为我们创建了多种flavor。

我们需要为这些测试虚拟机提供最少的资源，因此 m1.nano 风格就足够了。

stack@devstack21:~/devstack$ openstack flavor list
+----+-----------+-------+------+-----------+-------+-----------+
| ID | Name      |   RAM | Disk | Ephemeral | VCPUs | Is Public |
+----+-----------+-------+------+-----------+-------+-----------+
| 1  | m1.tiny   |   512 |    1 |         0 |     1 | True      |
| 2  | m1.small  |  2048 |   20 |         0 |     1 | True      |
| 3  | m1.medium |  4096 |   40 |         0 |     2 | True      |
| 4  | m1.large  |  8192 |   80 |         0 |     4 | True      |
| 42 | m1.nano   |   128 |    1 |         0 |     1 | True      |
| 5  | m1.xlarge | 16384 |  160 |         0 |     8 | True      |
| 84 | m1.micro  |   192 |    1 |         0 |     1 | True      |
| c1 | cirros256 |   256 |    1 |         0 |     1 | True      |
| d1 | ds512M    |   512 |    5 |         0 |     1 | True      |
| d2 | ds1G      |  1024 |   10 |         0 |     1 | True      |
| d3 | ds2G      |  2048 |   10 |         0 |     2 | True      |
| d4 | ds4G      |  4096 |   20 |         0 |     4 | True      |
+----+-----------+-------+------+-----------+-------+-----------+

$ FLAVOR_ID=$(openstack flavor show m1.nano -c id -f value)

4、选择一个镜像，devstack 默认已经为我们创建了一个cirros镜像

DevStack 默认导入 CirrOS 镜像，这非常适合我们的测试。这是一个非常小的测试图像。

stack@devstack21:~/devstack$ openstack image list
+--------------------------------------+--------------------------+--------+
| ID                                   | Name                     | Status |
+--------------------------------------+--------------------------+--------+
| df855e1f-4752-4d3c-a81b-6dc5ebfa96ab | cirros-0.6.2-x86_64-disk | active |
+--------------------------------------+--------------------------+--------+

$ IMAGE_ID=$(openstack image list -c ID -f value)

5、设置安全规则，以便我们可以访问接下来要启动的虚拟机。

默认情况下，DevStack 不允许用户访问虚拟机，要启用它，我们需要添加一条规则。我们将允许 ICMP 和 SSH。

openstack security group rule create --ingress --ethertype IPv4 --dst-port 22 --protocol tcp default
openstack security group rule create --ingress --ethertype IPv4 --protocol ICMP default

查看创建的安全组规则

$ openstack security group rule list
+--------------------------------------+-------------+-----------+------------+--------------------------------------+--------------------------------------+
| ID                                   | IP Protocol | IP Range  | Port Range | Remote Security Group                | Security Group                       |
+--------------------------------------+-------------+-----------+------------+--------------------------------------+--------------------------------------+
...
| ade97198-db44-429e-9b30-24693d86d9b1 | tcp         | 0.0.0.0/0 | 22:22      | None                                 | a47b14da-5607-404a-8de4-3a0f1ad3649c |
| d0861a98-f90e-4d1a-abfb-827b416bc2f6 | icmp        | 0.0.0.0/0 |            | None                                 | a47b14da-5607-404a-8de4-3a0f1ad3649c |
...
+--------------------------------------+-------------+-----------+------------+--------------------------------------+--------------------------------------+

6、启动一些虚拟机。

现在我们将启动两个虚拟机。我们将它们命名为 test1 和 test2 。

stack@devstack21:~/devstack$ openstack server create --nic net-id=$PRIVATE_NET_ID --flavor $FLAVOR_ID --image $IMAGE_ID --key-name demo test1
+--------------------------------------+-----------------------------------------------------------------+
| Field                                | Value                                                           |
+--------------------------------------+-----------------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                                          |
| OS-EXT-AZ:availability_zone          |                                                                 |
| OS-EXT-STS:power_state               | NOSTATE                                                         |
| OS-EXT-STS:task_state                | scheduling                                                      |
| OS-EXT-STS:vm_state                  | building                                                        |
| OS-SRV-USG:launched_at               | None                                                            |
| OS-SRV-USG:terminated_at             | None                                                            |
| accessIPv4                           |                                                                 |
| accessIPv6                           |                                                                 |
| addresses                            |                                                                 |
| adminPass                            | Y5j83PELiYKs                                                    |
| config_drive                         |                                                                 |
| created                              | 2024-03-24T04:42:57Z                                            |
| flavor                               | m1.nano (42)                                                    |
| hostId                               |                                                                 |
| id                                   | 88ea2c47-7fe9-460b-b98a-0e4ebf773c98                            |
| image                                | cirros-0.6.2-x86_64-disk (df855e1f-4752-4d3c-a81b-6dc5ebfa96ab) |
| key_name                             | demo                                                            |
| name                                 | test1                                                           |
| os-extended-volumes:volumes_attached | []                                                              |
| progress                             | 0                                                               |
| project_id                           | 6525123a5d8d4f048d51edf1ed18f06b                                |
| properties                           |                                                                 |
| security_groups                      | name='default'                                                  |
| status                               | BUILD                                                           |
| updated                              | 2024-03-24T04:42:57Z                                            |
| user_id                              | 55e4cf4eec0f48efa3ffa182046af847                                |
+--------------------------------------+-----------------------------------------------------------------+
stack@devstack21:~/devstack$ 

stack@devstack21:~/devstack$ openstack server create --nic net-id=$PRIVATE_NET_ID --flavor $FLAVOR_ID --image $IMAGE_ID --key-name demo test2
+--------------------------------------+-----------------------------------------------------------------+
| Field                                | Value                                                           |
+--------------------------------------+-----------------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                                          |
| OS-EXT-AZ:availability_zone          |                                                                 |
| OS-EXT-STS:power_state               | NOSTATE                                                         |
| OS-EXT-STS:task_state                | scheduling                                                      |
| OS-EXT-STS:vm_state                  | building                                                        |
| OS-SRV-USG:launched_at               | None                                                            |
| OS-SRV-USG:terminated_at             | None                                                            |
| accessIPv4                           |                                                                 |
| accessIPv6                           |                                                                 |
| addresses                            |                                                                 |
| adminPass                            | m3dhpx8meFHh                                                    |
| config_drive                         |                                                                 |
| created                              | 2024-03-24T04:43:35Z                                            |
| flavor                               | m1.nano (42)                                                    |
| hostId                               |                                                                 |
| id                                   | 8a32e126-7f97-4e1a-9a06-059867643785                            |
| image                                | cirros-0.6.2-x86_64-disk (df855e1f-4752-4d3c-a81b-6dc5ebfa96ab) |
| key_name                             | demo                                                            |
| name                                 | test2                                                           |
| os-extended-volumes:volumes_attached | []                                                              |
| progress                             | 0                                                               |
| project_id                           | 6525123a5d8d4f048d51edf1ed18f06b                                |
| properties                           |                                                                 |
| security_groups                      | name='default'                                                  |
| status                               | BUILD                                                           |
| updated                              | 2024-03-24T04:43:34Z                                            |
| user_id                              | 55e4cf4eec0f48efa3ffa182046af847                                |
+--------------------------------------+-----------------------------------------------------------------+

两个虚拟机启动后，它们的状态将为 ACTIVE ：

stack@devstack22:~/devstack$ openstack server list
+--------------------------------------+-------+--------+--------------------------------------------------------+--------------------------+---------+
| ID                                   | Name  | Status | Networks                                               | Image                    | Flavor  |
+--------------------------------------+-------+--------+--------------------------------------------------------+--------------------------+---------+
| 8a32e126-7f97-4e1a-9a06-059867643785 | test2 | ACTIVE | private=10.0.0.54, fd66:a9a:3259:0:f816:3eff:fe5e:4f23 | cirros-0.6.2-x86_64-disk | m1.nano |
| 88ea2c47-7fe9-460b-b98a-0e4ebf773c98 | test1 | ACTIVE | private=10.0.0.36, fd66:a9a:3259:0:f816:3eff:fe6f:4b92 | cirros-0.6.2-x86_64-disk | m1.nano |
+--------------------------------------+-------+--------+--------------------------------------------------------+--------------------------+---------+
stack@devstack22:~/devstack$

虚拟机连接

我们可以通过关联公共网络中的浮动 IP 地址来连接到我们的虚拟机。

stack@devstack22:~/devstack$ TEST1_PORT_ID=$(openstack port list --server test1 -c id -f value)
stack@devstack22:~/devstack$ openstack floating ip create --port $TEST1_PORT_ID public
+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field               | Value                                                                                                                                                                              |
+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| created_at          | 2024-03-24T05:26:11Z                                                                                                                                                               |
| description         |                                                                                                                                                                                    |
| dns_domain          |                                                                                                                                                                                    |
| dns_name            |                                                                                                                                                                                    |
| fixed_ip_address    | 10.0.0.36                                                                                                                                                                          |
| floating_ip_address | 192.168.72.119                                                                                                                                                                     |
| floating_network_id | 6211312f-ab8e-4346-bc58-aed8a63d4f62                                                                                                                                               |
| id                  | 8b830932-5b79-4445-8730-2143e1779e54                                                                                                                                               |
| name                | 192.168.72.119                                                                                                                                                                     |
| port_details        | {'name': '', 'network_id': '8b8cdb32-c009-49e1-88d4-3d58198e6d6c', 'mac_address': 'fa:16:3e:6f:4b:92', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id':                    |
|                     | '88ea2c47-7fe9-460b-b98a-0e4ebf773c98', 'device_owner': 'compute:nova'}                                                                                                            |
| port_forwardings    | []                                                                                                                                                                                 |
| port_id             | 5fc6a820-8b4e-404b-899d-b57090708efd                                                                                                                                               |
| project_id          | 6525123a5d8d4f048d51edf1ed18f06b                                                                                                                                                   |
| qos_policy_id       | None                                                                                                                                                                               |
| revision_number     | 0                                                                                                                                                                                  |
| router_id           | c0e52c09-8791-4a4e-9c82-9cd40d27a65e                                                                                                                                               |
| status              | DOWN                                                                                                                                                                               |
| subnet_id           | None                                                                                                                                                                               |
| tags                | []                                                                                                                                                                                 |
| updated_at          | 2024-03-24T05:26:11Z                                                                                                                                                               |
+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
stack@devstack22:~/devstack$

现在应该能够通过其浮动 IP 地址连接到虚拟机。首先，ping 地址。

stack@devstack21:~/devstack$ ping 192.168.72.119 -c 3
PING 192.168.72.119 (192.168.72.119) 56(84) bytes of data.
64 bytes from 192.168.72.119: icmp_seq=1 ttl=128 time=2.43 ms
64 bytes from 192.168.72.119: icmp_seq=2 ttl=128 time=1.75 ms
64 bytes from 192.168.72.119: icmp_seq=3 ttl=128 time=1.03 ms

--- 192.168.72.119 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 1.031/1.735/2.425/0.569 ms
stack@devstack21:~/devstack$

现在通过 SSH 连接到虚拟机：

stack@devstack21:~/devstack$ ssh -i id_rsa_demo cirros@192.168.72.119 hostname
test1
stack@devstack21:~/devstack$

登录dashboard

切换到demo项目，查看虚拟机管理器
在这里插入图片描述
查看test1实例位于devstack22节点上

查看test2实例位于devstack21节点上

查看网络拓扑
在这里插入图片描述