From 7df935f477a2f8a645aff8e1441fcee4f7bbd90b Mon Sep 17 00:00:00 2001 From: "Martin, Chen" Date: Fri, 11 Sep 2020 11:24:52 +0800 Subject: [PATCH] Rook Deployment guide for r5 release Story: 2005527 Task: 41161 Small text edits to fix HTML formatting. Change-Id: I8438265ddddc5078e57ff2a3099c70c0f7ef78c9 Signed-off-by: Martin, Chen Signed-off-by: MCamp859 --- .../aio_duplex_install_kubernetes.rst | 159 +++- .../aio_simplex_install_kubernetes.rst | 115 ++- .../r5_release/bare_metal/rook_storage.rst | 22 + .../bare_metal/rook_storage_hardware.rst | 62 ++ .../rook_storage_install_kubernetes.rst | 728 ++++++++++++++++++ .../r5_release/desc_rook_storage.txt | 22 + .../r5_release/index.rst | 2 + .../virtual/aio_duplex_install_kubernetes.rst | 165 +++- .../aio_simplex_install_kubernetes.rst | 128 ++- .../r5_release/virtual/rook_storage.rst | 21 + .../virtual/rook_storage_environ.rst | 61 ++ .../rook_storage_install_kubernetes.rst | 539 +++++++++++++ 12 files changed, 1946 insertions(+), 78 deletions(-) create mode 100644 doc/source/deploy_install_guides/r5_release/bare_metal/rook_storage.rst create mode 100644 doc/source/deploy_install_guides/r5_release/bare_metal/rook_storage_hardware.rst create mode 100644 doc/source/deploy_install_guides/r5_release/bare_metal/rook_storage_install_kubernetes.rst create mode 100644 doc/source/deploy_install_guides/r5_release/desc_rook_storage.txt create mode 100644 doc/source/deploy_install_guides/r5_release/virtual/rook_storage.rst create mode 100644 doc/source/deploy_install_guides/r5_release/virtual/rook_storage_environ.rst create mode 100644 doc/source/deploy_install_guides/r5_release/virtual/rook_storage_install_kubernetes.rst diff --git a/doc/source/deploy_install_guides/r5_release/bare_metal/aio_duplex_install_kubernetes.rst b/doc/source/deploy_install_guides/r5_release/bare_metal/aio_duplex_install_kubernetes.rst index 4b6a4fbf0..ebcd9754d 100644 --- a/doc/source/deploy_install_guides/r5_release/bare_metal/aio_duplex_install_kubernetes.rst +++ b/doc/source/deploy_install_guides/r5_release/bare_metal/aio_duplex_install_kubernetes.rst @@ -175,19 +175,56 @@ Configure controller-0 system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org -#. Configure Ceph storage backend +************************************************************** +Optionally, initialize a Ceph-based Persistent Storage Backend +************************************************************** - .. important:: +.. important:: - This step is required only if your application requires - persistent storage. + A persistent storage backend is required if your application requires + Persistent Volume Claims (PVCs). The StarlingX OpenStack application + (stx-openstack) requires PVCs, therefore if you plan on using the + stx-openstack application, then you must configure a persistent storage + backend. - **If you want to install the StarlingX Openstack application - (stx-openstack) this step is mandatory.** + There are two options for persistent storage backend: + 1) the host-based Ceph solution and + 2) the Rook container-based Ceph solution. + + The Rook container-based Ceph backend is installed after both AIO-Controllers + are configured and unlocked. + +For host-based Ceph, + +#. Initialize with add ceph backend: :: - system storage-backend-add ceph --confirmed + system storage-backend-add ceph --confirmed + +#. Add an OSD on controller-0 for host-based Ceph: + + :: + + system host-disk-list controller-0 + system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {} + system host-stor-list controller-0 + +For Rook container-based Ceph: + +#. Initialize with add ceph-rook backend: + + :: + + system storage-backend-add ceph-rook --confirmed + +#. Assign Rook host labels to controller-0 in support of installing the + rook-ceph-apps manifest/helm-charts later: + + :: + + system host-label-assign controller-0 ceph-mon-placement=enabled + system host-label-assign controller-0 ceph-mgr-placement=enabled #. Configure data interfaces for controller-0. Use the DATA port names, for example eth0, applicable to your deployment environment. @@ -245,20 +282,6 @@ Configure controller-0 system interface-datanetwork-assign ${NODE} ${DATA0IFUUID} ${PHYSNET0} system interface-datanetwork-assign ${NODE} ${DATA1IFUUID} ${PHYSNET1} -#. Add an OSD on controller-0 for Ceph. The following example adds an OSD - to the `sdb` disk: - - .. important:: - - This step requires a configured Ceph storage backend - - :: - - echo ">>> Add OSDs to primary tier" - system host-disk-list controller-0 - system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {} - system host-stor-list controller-0 - #. If required, and not already done as part of bootstrap, configure Docker to use a proxy server. @@ -407,19 +430,30 @@ Configure controller-1 system interface-datanetwork-assign ${NODE} ${DATA0IFUUID} ${PHYSNET0} system interface-datanetwork-assign ${NODE} ${DATA1IFUUID} ${PHYSNET1} -#. Add an OSD on controller-1 for Ceph: +************************************************************************************* +Optionally, configure host-specific details for Ceph-based Persistent Storage Backend +************************************************************************************* - .. important:: +For host-based Ceph: - This step requires a configured Ceph storage backend +#. Add an OSD on controller-1 for host-based Ceph: :: - echo ">>> Add OSDs to primary tier" system host-disk-list controller-1 system host-disk-list controller-1 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-1 {} system host-stor-list controller-1 +For Rook container-based Ceph: + +#. Assign Rook host labels to controller-1 in support of installing the + rook-ceph-apps manifest/helm-charts later: + + :: + + system host-label-assign controller-1 ceph-mon-placement=enabled + system host-label-assign controller-1 ceph-mgr-placement=enabled + ************************************* OpenStack-specific host configuration ************************************* @@ -473,6 +507,81 @@ Controller-1 will reboot in order to apply configuration changes and come into service. This can take 5-10 minutes, depending on the performance of the host machine. +-------------------------------------------------------------------------- +Optionally, finish configuration of Ceph-based Persistent Storage Backend +-------------------------------------------------------------------------- + +For host-based Ceph: Nothing else is required. + +For Rook container-based Ceph: + +On **virtual** controller-0 and controller-1: + +#. Wait for the ``rook-ceph-apps`` application to be uploaded + + :: + + $ source /etc/platform/openrc + $ system application-list + +---------------------+---------+-------------------------------+---------------+----------+-----------+ + | application | version | manifest name | manifest file | status | progress | + +---------------------+---------+-------------------------------+---------------+----------+-----------+ + | oidc-auth-apps | 1.0-0 | oidc-auth-manifest | manifest.yaml | uploaded | completed | + | platform-integ-apps | 1.0-8 | platform-integration-manifest | manifest.yaml | uploaded | completed | + | rook-ceph-apps | 1.0-1 | rook-ceph-manifest | manifest.yaml | uploaded | completed | + +---------------------+---------+-------------------------------+---------------+----------+-----------+ + +#. Configure Rook to use /dev/sdb on controller-0 and controller-1 as a ceph osd + + :: + + $ system host-disk-wipe -s --confirm controller-0 /dev/sdb + $ system host-disk-wipe -s --confirm controller-1 /dev/sdb + + values.yaml for rook-ceph-apps. + :: + + cluster: + storage: + nodes: + - name: controller-0 + devices: + - name: /dev/disk/by-path/pci-0000:00:03.0-ata-2.0 + - name: controller-1 + devices: + - name: /dev/disk/by-path/pci-0000:00:03.0-ata-2.0 + + :: + + system helm-override-update rook-ceph-apps rook-ceph kube-system --values values.yaml + +#. Apply the rook-ceph-apps application. + + :: + + system application-apply rook-ceph-apps + +#. Wait for OSDs pod to be ready. + + :: + + kubectl get pods -n kube-system + rook-ceph-crashcollector-controller-0-f984688ff-jsr8t 1/1 Running 0 4m9s + rook-ceph-crashcollector-controller-1-7f9b6f55b6-699bb 1/1 Running 0 2m5s + rook-ceph-mgr-a-7f9d588c5b-49cbg 1/1 Running 0 3m5s + rook-ceph-mon-a-75bcbd8664-pvq99 1/1 Running 0 4m27s + rook-ceph-mon-b-86c67658b4-f4snf 1/1 Running 0 4m10s + rook-ceph-mon-c-7f48b58dfb-4nx2n 1/1 Running 0 3m30s + rook-ceph-operator-77b64588c5-bhfg7 1/1 Running 0 7m6s + rook-ceph-osd-0-6949657cf7-dkfp2 1/1 Running 0 2m6s + rook-ceph-osd-1-5d4b58cf69-kdg82 1/1 Running 0 2m4s + rook-ceph-osd-prepare-controller-0-wcvsn 0/1 Completed 0 2m27s + rook-ceph-osd-prepare-controller-1-98h76 0/1 Completed 0 2m26s + rook-ceph-tools-5778d7f6c-2h8s8 1/1 Running 0 5m55s + rook-discover-xc22t 1/1 Running 0 6m2s + rook-discover-xndld 1/1 Running 0 6m2s + storage-init-rook-ceph-provisioner-t868q 0/1 Completed 0 108s + ---------- Next steps ---------- diff --git a/doc/source/deploy_install_guides/r5_release/bare_metal/aio_simplex_install_kubernetes.rst b/doc/source/deploy_install_guides/r5_release/bare_metal/aio_simplex_install_kubernetes.rst index d700f88b5..6df511b3b 100644 --- a/doc/source/deploy_install_guides/r5_release/bare_metal/aio_simplex_install_kubernetes.rst +++ b/doc/source/deploy_install_guides/r5_release/bare_metal/aio_simplex_install_kubernetes.rst @@ -181,19 +181,56 @@ Configure controller-0 system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org -#. Configure Ceph storage backend +************************************************************** +Optionally, initialize a Ceph-based Persistent Storage Backend +************************************************************** - .. important:: +.. important:: - This step required only if your application requires - persistent storage. + A persistent storage backend is required if your application requires + Persistent Volume Claims (PVCs). The StarlingX OpenStack application + (stx-openstack) requires PVCs, therefore if you plan on using the + stx-openstack application, then you must configure a persistent storage + backend. - **If you want to install the StarlingX Openstack application - (stx-openstack) this step is mandatory.** + There are two options for persistent storage backend: + 1) the host-based Ceph solution and + 2) the Rook container-based Ceph solution. + + The Rook container-based Ceph backend is installed after both AIO-Controllers + are configured and unlocked. + +For host-based Ceph, + +#. Initialize with add ceph backend: :: - system storage-backend-add ceph --confirmed + system storage-backend-add ceph --confirmed + +#. Add an OSD on controller-0 for host-based Ceph: + + :: + + system host-disk-list controller-0 + system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {} + system host-stor-list controller-0 + +For Rook container-based Ceph: + +#. Initialize with add ceph-rook backend: + + :: + + system storage-backend-add ceph-rook --confirmed + +#. Assign Rook host labels to controller-0 in support of installing the + rook-ceph-apps manifest/helm-charts later: + + :: + + system host-label-assign controller-0 ceph-mon-placement=enabled + system host-label-assign controller-0 ceph-mgr-placement=enabled #. Configure data interfaces for controller-0. Use the DATA port names, for example eth0, applicable to your deployment environment. @@ -406,6 +443,70 @@ service. This can take 5-10 minutes, depending on the performance of the host ma .. incl-unlock-controller-0-aio-simplex-end: +-------------------------------------------------------------------------- +Optionally, finish configuration of Ceph-based Persistent Storage Backend +-------------------------------------------------------------------------- + +For host-based Ceph: Nothing else is required. + +For Rook container-based Ceph: + +On **virtual** controller-0: + +#. Wait for application rook-ceph-apps uploaded + + :: + + $ source /etc/platform/openrc + $ system application-list + +---------------------+---------+-------------------------------+---------------+----------+-----------+ + | application | version | manifest name | manifest file | status | progress | + +---------------------+---------+-------------------------------+---------------+----------+-----------+ + | oidc-auth-apps | 1.0-0 | oidc-auth-manifest | manifest.yaml | uploaded | completed | + | platform-integ-apps | 1.0-8 | platform-integration-manifest | manifest.yaml | uploaded | completed | + | rook-ceph-apps | 1.0-1 | rook-ceph-manifest | manifest.yaml | uploaded | completed | + +---------------------+---------+-------------------------------+---------------+----------+-----------+ + +#. Configure rook to use /dev/sdb disk on controller-0 as a ceph osd + + :: + + system host-disk-wipe -s --confirm controller-0 /dev/sdb + + values.yaml for rook-ceph-apps. + :: + + cluster: + storage: + nodes: + - name: controller-0 + devices: + - name: /dev/disk/by-path/pci-0000:00:03.0-ata-2.0 + + :: + + system helm-override-update rook-ceph-apps rook-ceph kube-system --values values.yaml + +#. Apply the rook-ceph-apps application. + + :: + + system application-apply rook-ceph-apps + +#. Wait for OSDs pod ready + + :: + + kubectl get pods -n kube-system + rook--ceph-crashcollector-controller-0-764c7f9c8-bh5c7 1/1 Running 0 62m + rook--ceph-mgr-a-69df96f57-9l28p 1/1 Running 0 63m + rook--ceph-mon-a-55fff49dcf-ljfnx 1/1 Running 0 63m + rook--ceph-operator-77b64588c5-nlsf2 1/1 Running 0 66m + rook--ceph-osd-0-7d5785889f-4rgmb 1/1 Running 0 62m + rook--ceph-osd-prepare-controller-0-cmwt5 0/1 Completed 0 2m14s + rook--ceph-tools-5778d7f6c-22tms 1/1 Running 0 64m + rook--discover-kmv6c 1/1 Running 0 65m + ---------- Next steps ---------- diff --git a/doc/source/deploy_install_guides/r5_release/bare_metal/rook_storage.rst b/doc/source/deploy_install_guides/r5_release/bare_metal/rook_storage.rst new file mode 100644 index 000000000..f3f377863 --- /dev/null +++ b/doc/source/deploy_install_guides/r5_release/bare_metal/rook_storage.rst @@ -0,0 +1,22 @@ +======================================================= +Bare metal Standard with Rook Storage Installation R5.0 +======================================================= + +-------- +Overview +-------- + +.. include:: ../desc_rook_storage.txt + +.. include:: ../ipv6_note.txt + + +------------ +Installation +------------ + +.. toctree:: + :maxdepth: 1 + + rook_storage_hardware + rook_storage_install_kubernetes diff --git a/doc/source/deploy_install_guides/r5_release/bare_metal/rook_storage_hardware.rst b/doc/source/deploy_install_guides/r5_release/bare_metal/rook_storage_hardware.rst new file mode 100644 index 000000000..0ee2dd69d --- /dev/null +++ b/doc/source/deploy_install_guides/r5_release/bare_metal/rook_storage_hardware.rst @@ -0,0 +1,62 @@ +===================== +Hardware Requirements +===================== + +This section describes the hardware requirements and server preparation for a +**StarlingX R5.0 bare metal Standard with Rook Storage** deployment +configuration. + +.. contents:: + :local: + :depth: 1 + +----------------------------- +Minimum hardware requirements +----------------------------- + +The recommended minimum hardware requirements for bare metal servers for various +host types are: + ++---------------------+---------------------------+-----------------------+-----------------------+ +| Minimum Requirement | Controller Node | Worker Node for rook | Worker Node for | +| | | storage | application | ++=====================+===========================+=======================+=======================+ +| Number of servers | 2 | 2-9 | 2-100 | ++---------------------+---------------------------+-----------------------+-----------------------+ +| Minimum processor | Dual-CPU Intel® Xeon® E5 26xx family (SandyBridge) 8 cores/socket | +| class | | ++---------------------+---------------------------+-----------------------+-----------------------+ +| Minimum memory | 64 GB | 64 GB | 32 GB | ++---------------------+---------------------------+-----------------------+-----------------------+ +| Primary disk | 500 GB SSD or NVMe (see | 120 GB (min. 10k RPM) | 120 GB (min. 10k RPM) | +| | :doc:`../../nvme_config`) | | | ++---------------------+---------------------------+-----------------------+-----------------------+ +| Additional disks | None | - 1 or more 500 GB | - For OpenStack, | +| | | (min. 10K RPM) for | recommend 1 or more | +| | | Ceph OSD | 500 GB (min. 10K | +| | | - Recommended, but | RPM) for VM | +| | | not required: 1 or | ephemeral storage | +| | | more SSDs or NVMe | | +| | | drives for Ceph | | +| | | journals (min. 1024 | | +| | | MiB per OSD | | +| | | journal) | | ++---------------------+---------------------------+-----------------------+-----------------------+ +| Minimum network | - Mgmt/Cluster: | - Mgmt/Cluster: | - Mgmt/Cluster: | +| ports | 1x10GE | 1x10GE | 1x10GE | +| | - OAM: 1x1GE | | - Data: 1 or more | +| | | | x 10GE | ++---------------------+---------------------------+-----------------------+-----------------------+ +| BIOS settings | - Hyper-Threading technology enabled | +| | - Virtualization technology enabled | +| | - VT for directed I/O enabled | +| | - CPU power and performance policy set to performance | +| | - CPU C state control disabled | +| | - Plug & play BMC detection disabled | ++---------------------+---------------------------+-----------------------+-----------------------+ + +-------------------------- +Prepare bare metal servers +-------------------------- + +.. include:: prep_servers.txt diff --git a/doc/source/deploy_install_guides/r5_release/bare_metal/rook_storage_install_kubernetes.rst b/doc/source/deploy_install_guides/r5_release/bare_metal/rook_storage_install_kubernetes.rst new file mode 100644 index 000000000..93719d47d --- /dev/null +++ b/doc/source/deploy_install_guides/r5_release/bare_metal/rook_storage_install_kubernetes.rst @@ -0,0 +1,728 @@ +===================================================================== +Install StarlingX Kubernetes on Bare Metal Standard with Rook Storage +===================================================================== + +This section describes the steps to install the StarlingX Kubernetes platform +on a **StarlingX R5.0 bare metal Standard with Rook Storage** deployment +configuration. + +.. contents:: + :local: + :depth: 1 + +------------------- +Create bootable USB +------------------- + +Refer to :doc:`/deploy_install_guides/bootable_usb` for instructions on how to +create a bootable USB with the StarlingX ISO on your system. + +-------------------------------- +Install software on controller-0 +-------------------------------- + +.. incl-install-software-controller-0-standard-start: + +#. Insert the bootable USB into a bootable USB port on the host you are + configuring as controller-0. + +#. Power on the host. + +#. Attach to a console, ensure the host boots from the USB, and wait for the + StarlingX Installer Menus. + +#. Make the following menu selections in the installer: + + #. First menu: Select 'Standard Controller Configuration' + #. Second menu: Select 'Graphical Console' or 'Textual Console' depending on + your terminal access to the console port + +#. Wait for non-interactive install of software to complete and server to reboot. + This can take 5-10 minutes, depending on the performance of the server. + +.. incl-install-software-controller-0-standard-end: + +-------------------------------- +Bootstrap system on controller-0 +-------------------------------- + +.. incl-bootstrap-sys-controller-0-standard-start: + +#. Login using the username / password of "sysadmin" / "sysadmin". + + When logging in for the first time, you will be forced to change the password. + + :: + + Login: sysadmin + Password: + Changing password for sysadmin. + (current) UNIX Password: sysadmin + New Password: + (repeat) New Password: + +#. Verify and/or configure IP connectivity. + + External connectivity is required to run the Ansible bootstrap playbook. The + StarlingX boot image will DHCP out all interfaces so the server may have + obtained an IP address and have external IP connectivity if a DHCP server is + present in your environment. Verify this using the :command:`ip addr` and + :command:`ping 8.8.8.8` commands. + + Otherwise, manually configure an IP address and default IP route. Use the + PORT, IP-ADDRESS/SUBNET-LENGTH and GATEWAY-IP-ADDRESS applicable to your + deployment environment. + + :: + + sudo ip address add / dev + sudo ip link set up dev + sudo ip route add default via dev + ping 8.8.8.8 + +#. Specify user configuration overrides for the Ansible bootstrap playbook. + + Ansible is used to bootstrap StarlingX on controller-0. Key files for Ansible + configuration are: + + ``/etc/ansible/hosts`` + The default Ansible inventory file. Contains a single host: localhost. + + ``/usr/share/ansible/stx-ansible/playbooks/bootstrap.yml`` + The Ansible bootstrap playbook. + + ``/usr/share/ansible/stx-ansible/playbooks/host_vars/bootstrap/default.yml`` + The default configuration values for the bootstrap playbook. + + ``sysadmin home directory ($HOME)`` + The default location where Ansible looks for and imports user + configuration override files for hosts. For example: ``$HOME/.yml``. + + .. include:: ../ansible_install_time_only.txt + + Specify the user configuration override file for the Ansible bootstrap + playbook using one of the following methods: + + #. Use a copy of the default.yml file listed above to provide your overrides. + + The default.yml file lists all available parameters for bootstrap + configuration with a brief description for each parameter in the file comments. + + To use this method, copy the default.yml file listed above to + ``$HOME/localhost.yml`` and edit the configurable values as desired. + + #. Create a minimal user configuration override file. + + To use this method, create your override file at ``$HOME/localhost.yml`` + and provide the minimum required parameters for the deployment configuration + as shown in the example below. Use the OAM IP SUBNET and IP ADDRESSing + applicable to your deployment environment. + + :: + + cd ~ + cat < localhost.yml + system_mode: duplex + + dns_servers: + - 8.8.8.8 + - 8.8.4.4 + + external_oam_subnet: / + external_oam_gateway_address: + external_oam_floating_address: + external_oam_node_0_address: + external_oam_node_1_address: + + admin_username: admin + admin_password: + ansible_become_pass: + + # Add these lines to configure Docker to use a proxy server + # docker_http_proxy: http://my.proxy.com:1080 + # docker_https_proxy: https://my.proxy.com:1443 + # docker_no_proxy: + # - 1.2.3.4 + + EOF + + Refer to :doc:`/deploy_install_guides/r5_release/ansible_bootstrap_configs` + for information on additional Ansible bootstrap configurations for advanced + Ansible bootstrap scenarios, such as Docker proxies when deploying behind a + firewall, etc. Refer to :doc:`/../../configuration/docker_proxy_config` for + details about Docker proxy settings. + +#. Run the Ansible bootstrap playbook: + + :: + + ansible-playbook /usr/share/ansible/stx-ansible/playbooks/bootstrap.yml + + Wait for Ansible bootstrap playbook to complete. + This can take 5-10 minutes, depending on the performance of the host machine. + +.. incl-bootstrap-sys-controller-0-standard-end: + + +---------------------- +Configure controller-0 +---------------------- + +.. incl-config-controller-0-storage-start: + +#. Acquire admin credentials: + + :: + + source /etc/platform/openrc + +#. Configure the OAM and MGMT interfaces of controller-0 and specify the + attached networks. Use the OAM and MGMT port names, for example eth0, that are + applicable to your deployment environment. + + :: + + OAM_IF= + MGMT_IF= + system host-if-modify controller-0 lo -c none + IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}') + for UUID in $IFNET_UUIDS; do + system interface-network-remove ${UUID} + done + system host-if-modify controller-0 $OAM_IF -c platform + system interface-network-assign controller-0 $OAM_IF oam + system host-if-modify controller-0 $MGMT_IF -c platform + system interface-network-assign controller-0 $MGMT_IF mgmt + system interface-network-assign controller-0 $MGMT_IF cluster-host + +#. Configure NTP servers for network time synchronization: + + :: + + system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org + +#. If required, and not already done as part of bootstrap, configure Docker to + use a proxy server. + + #. List Docker proxy parameters: + + :: + + system service-parameter-list platform docker + + #. Refer to :doc:`/../../configuration/docker_proxy_config` for + details about Docker proxy settings. + +************************************* +OpenStack-specific host configuration +************************************* + +.. important:: + + **This step is required only if the StarlingX OpenStack application + (stx-openstack) will be installed.** + +#. **For OpenStack only:** Assign OpenStack host labels to controller-0 in + support of installing the stx-openstack manifest and helm-charts later. + + :: + + system host-label-assign controller-0 openstack-control-plane=enabled + +#. **For OpenStack only:** Configure the system setting for the vSwitch. + + StarlingX has OVS (kernel-based) vSwitch configured as default: + + * Runs in a container; defined within the helm charts of stx-openstack + manifest. + * Shares the core(s) assigned to the platform. + + If you require better performance, OVS-DPDK (OVS with the Data Plane + Development Kit, which is supported only on bare metal hardware) should be + used: + + * Runs directly on the host (it is not containerized). + * Requires that at least 1 core be assigned/dedicated to the vSwitch function. + + To deploy the default containerized OVS: + + :: + + system modify --vswitch_type none + + Do not run any vSwitch directly on the host, instead, use the containerized + OVS defined in the helm charts of stx-openstack manifest. + + To deploy OVS-DPDK, run the following command: + + :: + + system modify --vswitch_type ovs-dpdk + system host-cpu-modify -f vswitch -p0 1 controller-0 + + Once vswitch_type is set to OVS-DPDK, any subsequent nodes created will + default to automatically assigning 1 vSwitch core for AIO controllers and 2 + vSwitch cores for compute-labeled worker nodes. + + When using OVS-DPDK, configure vSwitch memory per NUMA node with the following + command: + + :: + + system host-memory-modify -f -1G <1G hugepages number> + + For example: + + :: + + system host-memory-modify -f vswitch -1G 1 worker-0 0 + + VMs created in an OVS-DPDK environment must be configured to use huge pages + to enable networking and must use a flavor with property: hw:mem_page_size=large + + Configure the huge pages for VMs in an OVS-DPDK environment with the command: + + :: + + system host-memory-modify -1G <1G hugepages number> + + For example: + + :: + + system host-memory-modify worker-0 0 -1G 10 + + .. note:: + + After controller-0 is unlocked, changing vswitch_type requires + locking and unlocking all compute-labeled worker nodes (and/or AIO + controllers) to apply the change. + +.. incl-config-controller-0-storage-end: + +******************************** +Rook-specific host configuration +******************************** + +.. important:: + + **This step is required only if the StarlingX Rook application will be + installed.** + +**For Rook only:** Assign Rook host labels to controller-0 in support of +installing the rook-ceph-apps manifest/helm-charts later and add ceph-rook +as storage backend: + +:: + + system host-label-assign controller-0 ceph-mon-placement=enabled + system host-label-assign controller-0 ceph-mgr-placement=enabled + system storage-backend-add ceph-rook --confirmed + +------------------- +Unlock controller-0 +------------------- + +Unlock controller-0 in order to bring it into service: + +:: + + system host-unlock controller-0 + +Controller-0 will reboot in order to apply configuration changes and come into +service. This can take 5-10 minutes, depending on the performance of the host machine. + +------------------------------------------------- +Install software on controller-1 and worker nodes +------------------------------------------------- + +#. Power on the controller-1 server and force it to network boot with the + appropriate BIOS boot options for your particular server. + +#. As controller-1 boots, a message appears on its console instructing you to + configure the personality of the node. + +#. On the console of controller-0, list hosts to see newly discovered controller-1 + host (hostname=None): + + :: + + system host-list + +----+--------------+-------------+----------------+-------------+--------------+ + | id | hostname | personality | administrative | operational | availability | + +----+--------------+-------------+----------------+-------------+--------------+ + | 1 | controller-0 | controller | unlocked | enabled | available | + | 2 | None | None | locked | disabled | offline | + +----+--------------+-------------+----------------+-------------+--------------+ + +#. Using the host id, set the personality of this host to 'controller': + + :: + + system host-update 2 personality=controller + + This initiates the install of software on controller-1. + This can take 5-10 minutes, depending on the performance of the host machine. + +#. While waiting for the previous step to complete, power on the worker nodes. + Set the personality to 'worker' and assign a unique hostname for each. + + For example, power on worker-0 and wait for the new host (hostname=None) to + be discovered by checking 'system host-list': + + :: + + system host-update 3 personality=worker hostname=worker-0 + + Repeat for worker-1. Power on worker-1 and wait for the new host (hostname=None) to + be discovered by checking 'system host-list': + + :: + + system host-update 4 personality=worker hostname=worker-1 + + For rook storage, there is no storage personality. Some hosts with worker personality + providers storage service. Here we still named these worker host storage-x. + Repeat for storage-0 and storage-1. Power on storage-0, storage-1 and wait for the + new host (hostname=None) to be discovered by checking 'system host-list': + + :: + + system host-update 5 personality=worker hostname=storage-0 + system host-update 6 personality=worker hostname=storage-1 + +#. Wait for the software installation on controller-1, worker-0, and worker-1 to + complete, for all servers to reboot, and for all to show as locked/disabled/online in + 'system host-list'. + + :: + + system host-list + + +----+--------------+-------------+----------------+-------------+--------------+ + | id | hostname | personality | administrative | operational | availability | + +----+--------------+-------------+----------------+-------------+--------------+ + | 1 | controller-0 | controller | unlocked | enabled | available | + | 2 | controller-1 | controller | locked | disabled | online | + | 3 | worker-0 | worker | locked | disabled | online | + | 4 | worker-1 | worker | locked | disabled | online | + | 5 | storage-0 | worker | locked | disabled | online | + | 6 | storage-1 | worker | locked | disabled | online | + +----+--------------+-------------+----------------+-------------+--------------+ + +---------------------- +Configure controller-1 +---------------------- + +.. incl-config-controller-1-start: + +Configure the OAM and MGMT interfaces of controller-0 and specify the attached +networks. Use the OAM and MGMT port names, for example eth0, that are applicable +to your deployment environment. + +(Note that the MGMT interface is partially set up automatically by the network +install procedure.) + +:: + + OAM_IF= + MGMT_IF= + system host-if-modify controller-1 $OAM_IF -c platform + system interface-network-assign controller-1 $OAM_IF oam + system interface-network-assign controller-1 $MGMT_IF cluster-host + +************************************* +OpenStack-specific host configuration +************************************* + +.. important:: + + **This step is required only if the StarlingX OpenStack application + (stx-openstack) will be installed.** + +**For OpenStack only:** Assign OpenStack host labels to controller-1 in support +of installing the stx-openstack manifest and helm-charts later. + +:: + + system host-label-assign controller-1 openstack-control-plane=enabled + +.. incl-config-controller-1-end: + +******************************** +Rook-specific host configuration +******************************** + +.. important:: + + **This step is required only if the StarlingX Rook application will be + installed.** + +**For Rook only:** Assign Rook host labels to controller-1 in support of +installing the rook-ceph-apps manifest/helm-charts later: + +:: + + system host-label-assign controller-1 ceph-mon-placement=enabled + system host-label-assign controller-1 ceph-mgr-placement=enabled + +------------------- +Unlock controller-1 +------------------- + +.. incl-unlock-controller-1-start: + +Unlock controller-1 in order to bring it into service: + +:: + + system host-unlock controller-1 + +Controller-1 will reboot in order to apply configuration changes and come into +service. This can take 5-10 minutes, depending on the performance of the host +machine. + +.. incl-unlock-controller-1-end: + +---------------------- +Configure worker nodes +---------------------- + +#. Assign the cluster-host network to the MGMT interface for the worker nodes: + + (Note that the MGMT interfaces are partially set up automatically by the + network install procedure.) + + :: + + for NODE in worker-0 worker-1; do + system interface-network-assign $NODE mgmt0 cluster-host + done + +#. Configure data interfaces for worker nodes. Use the DATA port names, for + example eth0, that are applicable to your deployment environment. + + .. important:: + + This step is **required** for OpenStack. + + This step is optional for Kubernetes: Do this step if using SRIOV network + attachments in hosted application containers. + + For Kubernetes SRIOV network attachments: + + * Configure SRIOV device plug in: + + :: + + for NODE in worker-0 worker-1; do + system host-label-assign ${NODE} sriovdp=enabled + done + + * If planning on running DPDK in containers on this host, configure the number + of 1G Huge pages required on both NUMA nodes: + + :: + + for NODE in worker-0 worker-1; do + system host-memory-modify ${NODE} 0 -1G 100 + system host-memory-modify ${NODE} 1 -1G 100 + done + + For both Kubernetes and OpenStack: + + :: + + DATA0IF= + DATA1IF= + PHYSNET0='physnet0' + PHYSNET1='physnet1' + SPL=/tmp/tmp-system-port-list + SPIL=/tmp/tmp-system-host-if-list + + # configure the datanetworks in sysinv, prior to referencing it + # in the ``system host-if-modify`` command'. + system datanetwork-add ${PHYSNET0} vlan + system datanetwork-add ${PHYSNET1} vlan + + for NODE in worker-0 worker-1; do + echo "Configuring interface for: $NODE" + set -ex + system host-port-list ${NODE} --nowrap > ${SPL} + system host-if-list -a ${NODE} --nowrap > ${SPIL} + DATA0PCIADDR=$(cat $SPL | grep $DATA0IF |awk '{print $8}') + DATA1PCIADDR=$(cat $SPL | grep $DATA1IF |awk '{print $8}') + DATA0PORTUUID=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $2}') + DATA1PORTUUID=$(cat $SPL | grep ${DATA1PCIADDR} | awk '{print $2}') + DATA0PORTNAME=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $4}') + DATA1PORTNAME=$(cat $SPL | grep ${DATA1PCIADDR} | awk '{print $4}') + DATA0IFUUID=$(cat $SPIL | awk -v DATA0PORTNAME=$DATA0PORTNAME '($12 ~ DATA0PORTNAME) {print $2}') + DATA1IFUUID=$(cat $SPIL | awk -v DATA1PORTNAME=$DATA1PORTNAME '($12 ~ DATA1PORTNAME) {print $2}') + system host-if-modify -m 1500 -n data0 -c data ${NODE} ${DATA0IFUUID} + system host-if-modify -m 1500 -n data1 -c data ${NODE} ${DATA1IFUUID} + system interface-datanetwork-assign ${NODE} ${DATA0IFUUID} ${PHYSNET0} + system interface-datanetwork-assign ${NODE} ${DATA1IFUUID} ${PHYSNET1} + set +ex + done + +************************************* +OpenStack-specific host configuration +************************************* + +.. important:: + + **This step is required only if the StarlingX OpenStack application + (stx-openstack) will be installed.** + +#. **For OpenStack only:** Assign OpenStack host labels to the worker nodes in + support of installing the stx-openstack manifest and helm-charts later. + + :: + + for NODE in worker-0 worker-1; do + system host-label-assign $NODE openstack-compute-node=enabled + system host-label-assign $NODE openvswitch=enabled + system host-label-assign $NODE sriov=enabled + done + +#. **For OpenStack only:** Set up disk partition for nova-local volume group, + which is needed for stx-openstack nova ephemeral disks. + + :: + + for NODE in worker-0 worker-1; do + echo "Configuring Nova local for: $NODE" + ROOT_DISK=$(system host-show ${NODE} | grep rootfs | awk '{print $4}') + ROOT_DISK_UUID=$(system host-disk-list ${NODE} --nowrap | grep ${ROOT_DISK} | awk '{print $2}') + PARTITION_SIZE=10 + NOVA_PARTITION=$(system host-disk-partition-add -t lvm_phys_vol ${NODE} ${ROOT_DISK_UUID} ${PARTITION_SIZE}) + NOVA_PARTITION_UUID=$(echo ${NOVA_PARTITION} | grep -ow "| uuid | [a-z0-9\-]* |" | awk '{print $4}') + system host-lvg-add ${NODE} nova-local + system host-pv-add ${NODE} nova-local ${NOVA_PARTITION_UUID} + done + +-------------------- +Unlock worker nodes +-------------------- + +Unlock worker nodes in order to bring them into service: + +:: + + for NODE in worker-0 worker-1; do + system host-unlock $NODE + done + +The worker nodes will reboot in order to apply configuration changes and come into +service. This can take 5-10 minutes, depending on the performance of the host machine. + +----------------------- +Configure storage nodes +----------------------- + +#. Assign the cluster-host network to the MGMT interface for the storage nodes. + + Note that the MGMT interfaces are partially set up by the network install procedure. + + :: + + for NODE in storage-0 storage-1; do + system interface-network-assign $NODE mgmt0 cluster-host + done + +#. **For Rook only:** Assign Rook host labels to storage-0 in support + of installing the rook-ceph-apps manifest/helm-charts later: + + :: + + system host-label-assign storage-0 ceph-mon-placement=enabled + +-------------------- +Unlock storage nodes +-------------------- + +Unlock storage nodes in order to bring them into service: + +:: + + for STORAGE in storage-0 storage-1; do + system host-unlock $STORAGE + done + +The storage nodes will reboot in order to apply configuration changes and come +into service. This can take 5-10 minutes, depending on the performance of the host machine. + +------------------------------------------------- +Install Rook application manifest and helm-charts +------------------------------------------------- + +On host storage-0 and storage-1: + +#. Erase gpt header of disk sdb. + + :: + + $ system host-disk-wipe -s --confirm storage-0 /dev/sdb + $ system host-disk-wipe -s --confirm storage-1 /dev/sdb + +#. Wait for application "rook-ceph-apps" uploaded + + :: + + $ source /etc/platform/openrc + $ system application-list + +---------------------+---------+-------------------------------+---------------+----------+-----------+ + | application | version | manifest name | manifest file | status | progress | + +---------------------+---------+-------------------------------+---------------+----------+-----------+ + | oidc-auth-apps | 1.0-0 | oidc-auth-manifest | manifest.yaml | uploaded | completed | + | platform-integ-apps | 1.0-8 | platform-integration-manifest | manifest.yaml | uploaded | completed | + | rook-ceph-apps | 1.0-1 | rook-ceph-manifest | manifest.yaml | uploaded | completed | + +---------------------+---------+-------------------------------+---------------+----------+-----------+ + +#. Edit values.yaml for rook-ceph-apps. + + :: + + cluster: + storage: + nodes: + - name: storage-0 + devices: + - name: /dev/disk/by-path/pci-0000:00:03.0-ata-2.0 + - name: storage-1 + devices: + - name: /dev/disk/by-path/pci-0000:00:03.0-ata-2.0 + +#. Update rook-ceph-apps override value. + + :: + + system helm-override-update rook-ceph-apps rook-ceph kube-system --values values.yaml + +#. Apply the rook-ceph-apps application. + + :: + + system application-apply rook-ceph-apps + +#. Wait for OSDs pod ready. + + :: + + kubectl get pods -n kube-system + rook-ceph-mgr-a-ddffc8fbb-zkvln 1/1 Running 0 66s + rook-ceph-mon-a-c67fdb6c8-tlbvk 1/1 Running 0 2m11s + rook-ceph-mon-b-76969d8685-wcq62 1/1 Running 0 2m2s + rook-ceph-mon-c-5bc47c6cb9-vm4j8 1/1 Running 0 97s + rook-ceph-operator-6fc8cfb68b-bb57z 1/1 Running 1 7m9s + rook-ceph-osd-0-689b6f65b-2nvcx 1/1 Running 0 12s + rook-ceph-osd-1-7bfd69fdf9-vjqmp 1/1 Running 0 4s + rook-ceph-osd-prepare-rook-storage-0-hf28p 0/1 Completed 0 50s + rook-ceph-osd-prepare-rook-storage-1-r6lsd 0/1 Completed 0 50s + rook-ceph-tools-84c7fff88c-x5trx 1/1 Running 0 6m11s + +---------- +Next steps +---------- + +.. include:: ../kubernetes_install_next.txt diff --git a/doc/source/deploy_install_guides/r5_release/desc_rook_storage.txt b/doc/source/deploy_install_guides/r5_release/desc_rook_storage.txt new file mode 100644 index 000000000..eb5ebc287 --- /dev/null +++ b/doc/source/deploy_install_guides/r5_release/desc_rook_storage.txt @@ -0,0 +1,22 @@ +The Standard with Rook Storage deployment option is a standard installation +with independent controller and worker nodes. + +A Standard with Rook Storage configuration provides the following benefits: + +* A pool of up to 100 worker nodes +* A 2x node high availability (HA) controller cluster with HA services running + across the controller nodes in either active/active or active/standby mode +* A Rook storage back end solution using a two-to-9x node HA Rook storage + cluster that supports a replication factor of two or three + +.. note:: + + If you are behind a corporate firewall or proxy, you need to set proxy + settings. Refer to :doc:`/../../configuration/docker_proxy_config` for + details. + +.. figure:: ../figures/starlingx-deployment-options-controller-storage.png + :scale: 50% + :alt: Standard with Rook Storage deployment configuration + + *Figure 1: Standard with Rook Storage deployment configuration* diff --git a/doc/source/deploy_install_guides/r5_release/index.rst b/doc/source/deploy_install_guides/r5_release/index.rst index 1ebbddde2..fa09a8291 100644 --- a/doc/source/deploy_install_guides/r5_release/index.rst +++ b/doc/source/deploy_install_guides/r5_release/index.rst @@ -17,6 +17,7 @@ Install StarlingX Kubernetes in a virtual environment virtual/aio_duplex virtual/controller_storage virtual/dedicated_storage + virtual/rook_storage .. toctree:: :hidden: @@ -36,6 +37,7 @@ Install StarlingX Kubernetes on bare metal bare_metal/controller_storage bare_metal/dedicated_storage bare_metal/ironic + bare_metal/rook_storage .. toctree:: :hidden: diff --git a/doc/source/deploy_install_guides/r5_release/virtual/aio_duplex_install_kubernetes.rst b/doc/source/deploy_install_guides/r5_release/virtual/aio_duplex_install_kubernetes.rst index 82f415ab7..dfa5c301d 100644 --- a/doc/source/deploy_install_guides/r5_release/virtual/aio_duplex_install_kubernetes.rst +++ b/doc/source/deploy_install_guides/r5_release/virtual/aio_duplex_install_kubernetes.rst @@ -183,19 +183,56 @@ On virtual controller-0: system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org -#. Configure Ceph storage backend +************************************************************** +Optionally, initialize a Ceph-based Persistent Storage Backend +************************************************************** - .. important:: +.. important:: - This step required only if your application requires - persistent storage. + A persistent storage backend is required if your application requires + Persistent Volume Claims (PVCs). The StarlingX OpenStack application + (stx-openstack) requires PVCs, therefore if you plan on using the + stx-openstack application, then you must configure a persistent storage + backend. - **If you want to install the StarlingX Openstack application - (stx-openstack) this step is mandatory.** + There are two options for persistent storage backend: + 1) the host-based Ceph solution and + 2) the Rook container-based Ceph solution. + + The Rook container-based Ceph backend is installed after both + AIO-Controllers are configured and unlocked. + +For host-based Ceph, + +#. Initialize with add ceph backend: :: - system storage-backend-add ceph --confirmed + system storage-backend-add ceph --confirmed + +#. Add an OSD on controller-0 for host-based Ceph: + + :: + + system host-disk-list controller-0 + system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {} + system host-stor-list controller-0 + +For Rook container-based Ceph: + +#. Initialize with add ceph-rook backend: + + :: + + system storage-backend-add ceph-rook --confirmed + +#. Assign Rook host labels to controller-0 in support of installing the + rook-ceph-apps manifest/helm-charts later: + + :: + + system host-label-assign controller-0 ceph-mon-placement=enabled + system host-label-assign controller-0 ceph-mgr-placement=enabled #. Configure data interfaces for controller-0. @@ -238,18 +275,6 @@ On virtual controller-0: system interface-datanetwork-assign ${NODE} ${DATA0IFUUID} ${PHYSNET0} system interface-datanetwork-assign ${NODE} ${DATA1IFUUID} ${PHYSNET1} -#. Add an OSD on controller-0 for Ceph: - - .. important:: - - This step requires a configured Ceph storage backend. - - :: - - system host-disk-list controller-0 - system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {} - system host-stor-list controller-0 - #. If required, and not already done as part of bootstrap, configure Docker to use a proxy server. @@ -262,7 +287,6 @@ On virtual controller-0: #. Refer to :doc:`/../../configuration/docker_proxy_config` for details about Docker proxy settings. - ************************************* OpenStack-specific host configuration ************************************* @@ -396,18 +420,29 @@ On virtual controller-0: system interface-datanetwork-assign ${NODE} ${DATA0IFUUID} ${PHYSNET0} system interface-datanetwork-assign ${NODE} ${DATA1IFUUID} ${PHYSNET1} -#. Add an OSD on controller-1 for Ceph: +************************************************************************************* +Optionally, configure host-specific details for Ceph-based Persistent Storage Backend +************************************************************************************* - .. important:: +For host-based Ceph: - This step requires a configured Ceph storage backend +#. Add an OSD on controller-1 for host-based Ceph: :: - echo ">>> Add OSDs to primary tier" - system host-disk-list controller-1 - system host-disk-list controller-1 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-1 {} - system host-stor-list controller-1 + system host-disk-list controller-1 + system host-disk-list controller-1 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-1 {} + system host-stor-list controller-1 + +For Rook container-based Ceph: + +#. Assign Rook host labels to controller-1 in support of installing the + rook-ceph-apps manifest/helm-charts later: + + :: + + system host-label-assign controller-1 ceph-mon-placement=enabled + system host-label-assign controller-1 ceph-mgr-placement=enabled ************************************* OpenStack-specific host configuration @@ -460,6 +495,82 @@ Unlock virtual controller-1 in order to bring it into service: Controller-1 will reboot in order to apply configuration changes and come into service. This can take 5-10 minutes, depending on the performance of the host machine. + +-------------------------------------------------------------------------- +Optionally, finish configuration of Ceph-based Persistent Storage Backend +-------------------------------------------------------------------------- + +For host-based Ceph: Nothing else is required. + +For Rook container-based Ceph: + +On **virtual** controller-0 and controller-1: + +#. Wait for the ``rook-ceph-apps`` application to be uploaded + + :: + + $ source /etc/platform/openrc + $ system application-list + +---------------------+---------+-------------------------------+---------------+----------+-----------+ + | application | version | manifest name | manifest file | status | progress | + +---------------------+---------+-------------------------------+---------------+----------+-----------+ + | oidc-auth-apps | 1.0-0 | oidc-auth-manifest | manifest.yaml | uploaded | completed | + | platform-integ-apps | 1.0-8 | platform-integration-manifest | manifest.yaml | uploaded | completed | + | rook-ceph-apps | 1.0-1 | rook-ceph-manifest | manifest.yaml | uploaded | completed | + +---------------------+---------+-------------------------------+---------------+----------+-----------+ + +#. Configure Rook to use /dev/sdb on controller-0 and controller-1 as a ceph osd + + :: + + $ system host-disk-wipe -s --confirm controller-0 /dev/sdb + $ system host-disk-wipe -s --confirm controller-1 /dev/sdb + + values.yaml for rook-ceph-apps. + :: + + cluster: + storage: + nodes: + - name: controller-0 + devices: + - name: /dev/disk/by-path/pci-0000:00:03.0-ata-2.0 + - name: controller-1 + devices: + - name: /dev/disk/by-path/pci-0000:00:03.0-ata-2.0 + + :: + + system helm-override-update rook-ceph-apps rook-ceph kube-system --values values.yaml + +#. Apply the rook-ceph-apps application. + + :: + + system application-apply rook-ceph-apps + +#. Wait for OSDs pod to be ready. + + :: + + kubectl get pods -n kube-system + rook-ceph-crashcollector-controller-0-f984688ff-jsr8t 1/1 Running 0 4m9s + rook-ceph-crashcollector-controller-1-7f9b6f55b6-699bb 1/1 Running 0 2m5s + rook-ceph-mgr-a-7f9d588c5b-49cbg 1/1 Running 0 3m5s + rook-ceph-mon-a-75bcbd8664-pvq99 1/1 Running 0 4m27s + rook-ceph-mon-b-86c67658b4-f4snf 1/1 Running 0 4m10s + rook-ceph-mon-c-7f48b58dfb-4nx2n 1/1 Running 0 3m30s + rook-ceph-operator-77b64588c5-bhfg7 1/1 Running 0 7m6s + rook-ceph-osd-0-6949657cf7-dkfp2 1/1 Running 0 2m6s + rook-ceph-osd-1-5d4b58cf69-kdg82 1/1 Running 0 2m4s + rook-ceph-osd-prepare-controller-0-wcvsn 0/1 Completed 0 2m27s + rook-ceph-osd-prepare-controller-1-98h76 0/1 Completed 0 2m26s + rook-ceph-tools-5778d7f6c-2h8s8 1/1 Running 0 5m55s + rook-discover-xc22t 1/1 Running 0 6m2s + rook-discover-xndld 1/1 Running 0 6m2s + storage-init-rook-ceph-provisioner-t868q 0/1 Completed 0 108s + ---------- Next steps ---------- diff --git a/doc/source/deploy_install_guides/r5_release/virtual/aio_simplex_install_kubernetes.rst b/doc/source/deploy_install_guides/r5_release/virtual/aio_simplex_install_kubernetes.rst index ae8f4326b..e4d90b8ff 100644 --- a/doc/source/deploy_install_guides/r5_release/virtual/aio_simplex_install_kubernetes.rst +++ b/doc/source/deploy_install_guides/r5_release/virtual/aio_simplex_install_kubernetes.rst @@ -174,19 +174,56 @@ On virtual controller-0: system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org -#. Configure Ceph storage backend +************************************************************** +Optionally, initialize a Ceph-based Persistent Storage Backend +************************************************************** - .. important:: +.. important:: - This step required only if your application requires - persistent storage. + A persistent storage backend is required if your application requires + Persistent Volume Claims (PVCs). The StarlingX OpenStack application + (stx-openstack) requires PVCs, therefore if you plan on using the + stx-openstack application, then you must configure a persistent storage + backend. - **If you want to install the StarlingX Openstack application - (stx-openstack) this step is mandatory.** + There are two options for persistent storage backend: + 1) the host-based Ceph solution and + 2) the Rook container-based Ceph solution. + + The Rook container-based Ceph backend is installed after both + AIO-Controllers are configured and unlocked. + +For host-based Ceph, + +#. Initialize with add ceph backend: :: - system storage-backend-add ceph --confirmed + system storage-backend-add ceph --confirmed + +#. Add an OSD on controller-0 for host-based Ceph: + + :: + + system host-disk-list controller-0 + system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {} + system host-stor-list controller-0 + +For Rook container-based Ceph: + +#. Initialize with add ceph-rook backend: + + :: + + system storage-backend-add ceph-rook --confirmed + +#. Assign Rook host labels to controller-0 in support of installing the + rook-ceph-apps manifest/helm-charts later: + + :: + + system host-label-assign controller-0 ceph-mon-placement=enabled + system host-label-assign controller-0 ceph-mgr-placement=enabled #. Configure data interfaces for controller-0. @@ -229,18 +266,6 @@ On virtual controller-0: system interface-datanetwork-assign ${NODE} ${DATA0IFUUID} ${PHYSNET0} system interface-datanetwork-assign ${NODE} ${DATA1IFUUID} ${PHYSNET1} -#. Add an OSD on controller-0 for Ceph: - - .. important:: - - This step requires a configured Ceph storage backend - - :: - - system host-disk-list controller-0 - system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {} - system host-stor-list controller-0 - #. If required, and not already done as part of bootstrap, configure Docker to use a proxy server. @@ -317,6 +342,71 @@ Unlock virtual controller-0 to bring it into service: Controller-0 will reboot to apply configuration changes and come into service. This can take 5-10 minutes, depending on the performance of the host machine. +-------------------------------------------------------------------------- +Optionally, finish configuration of Ceph-based Persistent Storage Backend +-------------------------------------------------------------------------- + +For host-based Ceph: Nothing else is required. + +For Rook container-based Ceph: + +On **virtual** controller-0: + +#. Wait for application rook-ceph-apps uploaded + + :: + + $ source /etc/platform/openrc + $ system application-list + +---------------------+---------+-------------------------------+---------------+----------+-----------+ + | application | version | manifest name | manifest file | status | progress | + +---------------------+---------+-------------------------------+---------------+----------+-----------+ + | oidc-auth-apps | 1.0-0 | oidc-auth-manifest | manifest.yaml | uploaded | completed | + | platform-integ-apps | 1.0-8 | platform-integration-manifest | manifest.yaml | uploaded | completed | + | rook-ceph-apps | 1.0-1 | rook-ceph-manifest | manifest.yaml | uploaded | completed | + +---------------------+---------+-------------------------------+---------------+----------+-----------+ + +#. Configure rook to use /dev/sdb disk on controller-0 as a ceph osd + + :: + + $ system host-disk-wipe -s --confirm controller-0 /dev/sdb + + + values.yaml for rook-ceph-apps. + :: + + cluster: + storage: + nodes: + - name: controller-0 + devices: + - name: /dev/disk/by-path/pci-0000:00:03.0-ata-2.0 + + :: + + system helm-override-update rook-ceph-apps rook-ceph kube-system --values values.yaml + +#. Apply the rook-ceph-apps application. + + :: + + system application-apply rook-ceph-apps + +#. Wait for OSDs pod ready + + :: + + kubectl get pods -n kube-system + rook--ceph-crashcollector-controller-0-764c7f9c8-bh5c7 1/1 Running 0 62m + rook--ceph-mgr-a-69df96f57-9l28p 1/1 Running 0 63m + rook--ceph-mon-a-55fff49dcf-ljfnx 1/1 Running 0 63m + rook--ceph-operator-77b64588c5-nlsf2 1/1 Running 0 66m + rook--ceph-osd-0-7d5785889f-4rgmb 1/1 Running 0 62m + rook--ceph-osd-prepare-controller-0-cmwt5 0/1 Completed 0 2m14s + rook--ceph-tools-5778d7f6c-22tms 1/1 Running 0 64m + rook--discover-kmv6c 1/1 Running 0 65m + ---------- Next steps ---------- diff --git a/doc/source/deploy_install_guides/r5_release/virtual/rook_storage.rst b/doc/source/deploy_install_guides/r5_release/virtual/rook_storage.rst new file mode 100644 index 000000000..9982438c4 --- /dev/null +++ b/doc/source/deploy_install_guides/r5_release/virtual/rook_storage.rst @@ -0,0 +1,21 @@ +==================================================== +Virtual Standard with Rook Storage Installation R5.0 +==================================================== + +-------- +Overview +-------- + +.. include:: ../desc_rook_storage.txt + +.. include:: ../ipv6_note.txt + +------------ +Installation +------------ + +.. toctree:: + :maxdepth: 1 + + rook_storage_environ + rook_storage_install_kubernetes diff --git a/doc/source/deploy_install_guides/r5_release/virtual/rook_storage_environ.rst b/doc/source/deploy_install_guides/r5_release/virtual/rook_storage_environ.rst new file mode 100644 index 000000000..4c7fd175c --- /dev/null +++ b/doc/source/deploy_install_guides/r5_release/virtual/rook_storage_environ.rst @@ -0,0 +1,61 @@ +============================ +Prepare Host and Environment +============================ + +This section describes how to prepare the physical host and virtual environment +for a **StarlingX R5.0 virtual Standard with Rook Storage** deployment +configuration. + +.. contents:: + :local: + :depth: 1 + +------------------------------------ +Physical host requirements and setup +------------------------------------ + +.. include:: physical_host_req.txt + +--------------------------------------- +Prepare virtual environment and servers +--------------------------------------- + +.. note:: + + The following commands for host, virtual environment setup, and host + power-on use KVM / virsh for virtual machine and VM management + technology. For an alternative virtualization environment, see: + :doc:`Install StarlingX in VirtualBox `. + +#. Prepare virtual environment. + + Set up virtual platform networks for virtual deployment: + + :: + + bash setup_network.sh + +#. Prepare virtual servers. + + Create the XML definitions for the virtual servers required by this + configuration option. This will create the XML virtual server definition for: + + * rookstorage-controller-0 + * rookstorage-controller-1 + * rookstorage-worker-0 + * rookstorage-worker-1 + * rookstorage-worker-2 + * rookstorage-worker-3 + + The following command will start/virtually power on: + + * The 'rookstorage-controller-0' virtual server + * The X-based graphical virt-manager application + + :: + + export WORKER_NODES_NUMBER=4 ; bash setup_configuration.sh -c controllerstorage -i ./bootimage.iso + + If there is no X-server present errors will occur and the X-based GUI for the + virt-manager application will not start. The virt-manager GUI is not absolutely + required and you can safely ignore errors and continue. diff --git a/doc/source/deploy_install_guides/r5_release/virtual/rook_storage_install_kubernetes.rst b/doc/source/deploy_install_guides/r5_release/virtual/rook_storage_install_kubernetes.rst new file mode 100644 index 000000000..9f46c180c --- /dev/null +++ b/doc/source/deploy_install_guides/r5_release/virtual/rook_storage_install_kubernetes.rst @@ -0,0 +1,539 @@ +================================================================== +Install StarlingX Kubernetes on Virtual Standard with Rook Storage +================================================================== + +This section describes the steps to install the StarlingX Kubernetes platform +on a **StarlingX R5.0 virtual Standard with Rook Storage** deployment configuration, +deploy rook ceph cluster replacing default native ceph cluster. + +.. contents:: + :local: + :depth: 1 + +-------------------------------- +Install software on controller-0 +-------------------------------- + +In the last step of :doc:`rook_storage_environ`, the controller-0 virtual +server 'rookstorage-controller-0' was started by the +:command:`setup_configuration.sh` command. + +On the host, attach to the console of virtual controller-0 and select the appropriate +installer menu options to start the non-interactive install of +StarlingX software on controller-0. + +.. note:: + + When entering the console, it is very easy to miss the first installer menu + selection. Use ESC to navigate to previous menus, to ensure you are at the + first installer menu. + +:: + + virsh console rookstorage-controller-0 + +Make the following menu selections in the installer: + +#. First menu: Select 'Standard Controller Configuration' +#. Second menu: Select 'Serial Console' + +Wait for the non-interactive install of software to complete and for the server +to reboot. This can take 5-10 minutes depending on the performance of the host +machine. + +-------------------------------- +Bootstrap system on controller-0 +-------------------------------- + +.. include:: controller_storage_install_kubernetes.rst + :start-after: incl-bootstrap-controller-0-virt-controller-storage-start: + :end-before: incl-bootstrap-controller-0-virt-controller-storage-end: + +---------------------- +Configure controller-0 +---------------------- + +On virtual controller-0: + +#. Acquire admin credentials: + + :: + + source /etc/platform/openrc + +#. Configure the OAM and MGMT interfaces of controller-0 and specify the + attached networks: + + :: + + OAM_IF=enp7s1 + MGMT_IF=enp7s2 + system host-if-modify controller-0 lo -c none + IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}') + for UUID in $IFNET_UUIDS; do + system interface-network-remove ${UUID} + done + system host-if-modify controller-0 $OAM_IF -c platform + system interface-network-assign controller-0 $OAM_IF oam + system host-if-modify controller-0 $MGMT_IF -c platform + system interface-network-assign controller-0 $MGMT_IF mgmt + system interface-network-assign controller-0 $MGMT_IF cluster-host + +#. Configure NTP servers for network time synchronization: + + .. note:: + + In a virtual environment, this can sometimes cause Ceph clock skew alarms. + Also, the virtual instance clock is synchronized with the host clock, + so it is not absolutely required to configure NTP here. + + :: + + system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org + + +************************************* +OpenStack-specific host configuration +************************************* + +.. important:: + + **This step is required only if the StarlingX OpenStack application + (stx-openstack) will be installed.** + +#. **For OpenStack only:** Assign OpenStack host labels to controller-0 in + support of installing the stx-openstack manifest/helm-charts later: + + :: + + system host-label-assign controller-0 openstack-control-plane=enabled + +#. **For OpenStack only:** A vSwitch is required. + + The default vSwitch is containerized OVS that is packaged with the + stx-openstack manifest/helm-charts. StarlingX provides the option to use + OVS-DPDK on the host, however, in the virtual environment OVS-DPDK is NOT + supported, only OVS is supported. Therefore, simply use the default OVS + vSwitch here. + +******************************** +Rook-specific host configuration +******************************** + +.. important:: + + **This step is required only if the StarlingX Rook application will be + installed.** + +**For Rook only:** Assign Rook host labels to controller-0 in support of +installing the rook-ceph-apps manifest/helm-charts later and add ceph-rook +as storage backend: + +:: + + system host-label-assign controller-0 ceph-mon-placement=enabled + system host-label-assign controller-0 ceph-mgr-placement=enabled + system storage-backend-add ceph-rook --confirmed + +------------------- +Unlock controller-0 +------------------- + +Unlock virtual controller-0 in order to bring it into service: + +:: + + system host-unlock controller-0 + +Controller-0 will reboot in order to apply configuration changes and come into +service. This can take 5-10 minutes, depending on the performance of the host machine. + +----------------------------------------------------------------- +Install software on controller-1 and worker nodes +----------------------------------------------------------------- + +#. On the host, power on the controller-1 virtual server, + 'rookstorage-controller-1'. It will automatically attempt to network + boot over the management network: + + :: + + virsh start rookstorage-controller-1 + +#. Attach to the console of virtual controller-1: + + :: + + virsh console rookstorage-controller-1 + +#. As controller-1 VM boots, a message appears on its console instructing you to + configure the personality of the node. + +#. On the console of controller-0, list hosts to see newly discovered + controller-1 host (hostname=None): + + :: + + system host-list + +----+--------------+-------------+----------------+-------------+--------------+ + | id | hostname | personality | administrative | operational | availability | + +----+--------------+-------------+----------------+-------------+--------------+ + | 1 | controller-0 | controller | unlocked | enabled | available | + | 2 | None | None | locked | disabled | offline | + +----+--------------+-------------+----------------+-------------+--------------+ + +#. Using the host id, set the personality of this host to 'controller': + + :: + + system host-update 2 personality=controller + + This initiates software installation on controller-1. + This can take 5-10 minutes, depending on the performance of the host machine. + +#. While waiting on the previous step to complete, start up and set the personality + for 'rookstorage-worker-0' and 'rookstorage-worker-1'. Set the + personality to 'worker' and assign a unique hostname for each. + + For example, start 'rookstorage-worker-0' from the host: + + :: + + virsh start rookstorage-worker-0 + + Wait for new host (hostname=None) to be discovered by checking + ‘system host-list’ on virtual controller-0: + + :: + + system host-update 3 personality=worker hostname=rook-storage-0 + + Repeat for 'rookstorage-worker-1'. On the host: + + :: + + virsh start rookstorage-worker-1 + + And wait for new host (hostname=None) to be discovered by checking + ‘system host-list’ on virtual controller-0: + + :: + + system host-update 4 personality=worker hostname=rook-storage-1 + + This initiates software installation on storage-0 and storage-1. + This can take 5-10 minutes, depending on the performance of the host machine. + +#. While waiting on the previous step to complete, start up and set the personality + for 'rookstorage-worker-2' and 'rookstorage-worker-3'. Set the + personality to 'worker' and assign a unique hostname for each. + + For example, start 'rookstorage-worker-2' from the host: + + :: + + virsh start rookstorage-worker-2 + + Wait for new host (hostname=None) to be discovered by checking + ‘system host-list’ on virtual controller-0: + + :: + + system host-update 5 personality=worker hostname=worker-0 + + Repeat for 'rookstorage-worker-3'. On the host: + + :: + + virsh start controllerstorage-worker-3 + + And wait for new host (hostname=None) to be discovered by checking + ‘system host-list’ on virtual controller-0: + + :: + + system host-update 6 personality=worker hostname=worker-1 + + This initiates software installation on worker-0 and worker-1. + +#. Wait for the software installation on controller-1, storage-0, storage-1, + worker-0, and worker-1 to complete, for all virtual servers to reboot, and for all + to show as locked/disabled/online in 'system host-list'. + + :: + + system host-list + +----+----------------+-------------+----------------+-------------+--------------+ + | id | hostname | personality | administrative | operational | availability | + +----+----------------+-------------+----------------+-------------+--------------+ + | 1 | controller-0 | controller | unlocked | enabled | available | + | 2 | controller-1 | controller | unlocked | enabled | available | + | 3 | rook-storage-0 | worker | locked | disabled | offline | + | 4 | rook-storage-1 | worker | locked | disabled | offline | + | 5 | worker-0 | worker | locked | disabled | offline | + | 6 | worker-1 | worker | locked | disabled | offline | + +----+----------------+-------------+----------------+-------------+--------------+ + +---------------------- +Configure controller-1 +---------------------- + +.. include:: controller_storage_install_kubernetes.rst + :start-after: incl-config-controller-1-virt-controller-storage-start: + :end-before: incl-config-controller-1-virt-controller-storage-end: + +******************************** +Rook-specific host configuration +******************************** + +.. important:: + + **This step is required only if the StarlingX Rook application will be + installed.** + +**For Rook only:** Assign Rook host labels to controller-1 in +support of installing the rook-ceph-apps manifest/helm-charts later: + +:: + + system host-label-assign controller-1 ceph-mon-placement=enabled + system host-label-assign controller-1 ceph-mgr-placement=enabled + +------------------- +Unlock controller-1 +------------------- + +.. include:: controller_storage_install_kubernetes.rst + :start-after: incl-unlock-controller-1-virt-controller-storage-start: + :end-before: incl-unlock-controller-1-virt-controller-storage-end: + +----------------------- +Configure storage nodes +----------------------- + +On virtual controller-0: + +#. Assign the cluster-host network to the MGMT interface for the storage nodes. + + Note that the MGMT interfaces are partially set up by the network install procedure. + + :: + + for NODE in rook-storage-0 rook-storage-1; do + system interface-network-assign $NODE mgmt0 cluster-host + done + +#. **For Rook only:** Assign Rook host labels to storage-0 in + support of installing the rook-ceph-apps manifest/helm-charts later: + + :: + + system host-label-assign rook-storage-0 ceph-mon-placement=enabled + +-------------------- +Unlock storage nodes +-------------------- + +Unlock virtual storage nodes in order to bring them into service: + +:: + + for STORAGE in rook-storage-0 rook-storage-1; do + system host-unlock $STORAGE + done + +The storage nodes will reboot in order to apply configuration changes and come +into service. This can take 5-10 minutes, depending on the performance of the host machine. + +---------------------- +Configure worker nodes +---------------------- + +On virtual controller-0: + +#. Assign the cluster-host network to the MGMT interface for the worker nodes. + + Note that the MGMT interfaces are partially set up automatically by the + network install procedure. + + :: + + for NODE in worker-0 worker-1; do + system interface-network-assign $NODE mgmt0 cluster-host + done + +#. Configure data interfaces for worker nodes. + + .. important:: + + **This step is required only if the StarlingX OpenStack application + (stx-openstack) will be installed.** + + 1G Huge Pages are not supported in the virtual environment and there is no + virtual NIC supporting SRIOV. For that reason, data interfaces are not + applicable in the virtual environment for the Kubernetes-only scenario. + + For OpenStack only: + + :: + + DATA0IF=eth1000 + DATA1IF=eth1001 + PHYSNET0='physnet0' + PHYSNET1='physnet1' + SPL=/tmp/tmp-system-port-list + SPIL=/tmp/tmp-system-host-if-list + + Configure the datanetworks in sysinv, prior to referencing it in the + :command:`system host-if-modify` command. + + :: + + system datanetwork-add ${PHYSNET0} vlan + system datanetwork-add ${PHYSNET1} vlan + + for NODE in worker-0 worker-1; do + echo "Configuring interface for: $NODE" + set -ex + system host-port-list ${NODE} --nowrap > ${SPL} + system host-if-list -a ${NODE} --nowrap > ${SPIL} + DATA0PCIADDR=$(cat $SPL | grep $DATA0IF |awk '{print $8}') + DATA1PCIADDR=$(cat $SPL | grep $DATA1IF |awk '{print $8}') + DATA0PORTUUID=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $2}') + DATA1PORTUUID=$(cat $SPL | grep ${DATA1PCIADDR} | awk '{print $2}') + DATA0PORTNAME=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $4}') + DATA1PORTNAME=$(cat $SPL | grep ${DATA1PCIADDR} | awk '{print $4}') + DATA0IFUUID=$(cat $SPIL | awk -v DATA0PORTNAME=$DATA0PORTNAME '($12 ~ DATA0PORTNAME) {print $2}') + DATA1IFUUID=$(cat $SPIL | awk -v DATA1PORTNAME=$DATA1PORTNAME '($12 ~ DATA1PORTNAME) {print $2}') + system host-if-modify -m 1500 -n data0 -c data ${NODE} ${DATA0IFUUID} + system host-if-modify -m 1500 -n data1 -c data ${NODE} ${DATA1IFUUID} + system interface-datanetwork-assign ${NODE} ${DATA0IFUUID} ${PHYSNET0} + system interface-datanetwork-assign ${NODE} ${DATA1IFUUID} ${PHYSNET1} + set +ex + done + +************************************* +OpenStack-specific host configuration +************************************* + +.. important:: + + **This step is required only if the StarlingX OpenStack application + (stx-openstack) will be installed.** + +#. **For OpenStack only:** Assign OpenStack host labels to the worker nodes in + support of installing the stx-openstack manifest/helm-charts later: + + :: + + for NODE in worker-0 worker-1; do + system host-label-assign $NODE openstack-compute-node=enabled + system host-label-assign $NODE openvswitch=enabled + system host-label-assign $NODE sriov=enabled + done + +#. **For OpenStack only:** Set up disk partition for nova-local volume group, + which is needed for stx-openstack nova ephemeral disks: + + :: + + for NODE in worker-0 worker-1; do + echo "Configuring Nova local for: $NODE" + ROOT_DISK=$(system host-show ${NODE} | grep rootfs | awk '{print $4}') + ROOT_DISK_UUID=$(system host-disk-list ${NODE} --nowrap | grep ${ROOT_DISK} | awk '{print $2}') + PARTITION_SIZE=10 + NOVA_PARTITION=$(system host-disk-partition-add -t lvm_phys_vol ${NODE} ${ROOT_DISK_UUID} ${PARTITION_SIZE}) + NOVA_PARTITION_UUID=$(echo ${NOVA_PARTITION} | grep -ow "| uuid | [a-z0-9\-]* |" | awk '{print $4}') + system host-lvg-add ${NODE} nova-local + system host-pv-add ${NODE} nova-local ${NOVA_PARTITION_UUID} + done + +------------------- +Unlock worker nodes +------------------- + +Unlock virtual worker nodes to bring them into service: + +:: + + for NODE in worker-0 worker-1; do + system host-unlock $NODE + done + +The worker nodes will reboot in order to apply configuration changes and come into +service. This can take 5-10 minutes, depending on the performance of the host machine. + +------------------------------------------------- +Install Rook application manifest and helm-charts +------------------------------------------------- + +On virtual storage-0 and storage-1: + +#. Erase gpt header of disk sdb. + + :: + + $ system host-disk-wipe -s --confirm rook-storage-0 /dev/sdb + $ system host-disk-wipe -s --confirm rook-storage-1 /dev/sdb + +#. Wait for application "rook-ceph-apps" uploaded + + :: + + $ source /etc/platform/openrc + $ system application-list + +---------------------+---------+-------------------------------+---------------+----------+-----------+ + | application | version | manifest name | manifest file | status | progress | + +---------------------+---------+-------------------------------+---------------+----------+-----------+ + | oidc-auth-apps | 1.0-0 | oidc-auth-manifest | manifest.yaml | uploaded | completed | + | platform-integ-apps | 1.0-8 | platform-integration-manifest | manifest.yaml | uploaded | completed | + | rook-ceph-apps | 1.0-1 | rook-ceph-manifest | manifest.yaml | uploaded | completed | + +---------------------+---------+-------------------------------+---------------+----------+-----------+ + +#. Edit values.yaml for rook-ceph-apps. + + :: + + cluster: + storage: + nodes: + - name: rook-storage-0 + devices: + - name: /dev/disk/by-path/pci-0000:00:03.0-ata-2.0 + - name: rook-storage-1 + devices: + - name: /dev/disk/by-path/pci-0000:00:03.0-ata-2.0 + +#. Update rook-ceph-apps override value. + + :: + + system helm-override-update rook-ceph-apps rook-ceph kube-system --values values.yaml + +#. Apply the rook-ceph-apps application. + + :: + + system application-apply rook-ceph-apps + +#. Wait for OSDs pod ready. + + :: + + kubectl get pods -n kube-system + rook-ceph-mgr-a-ddffc8fbb-zkvln 1/1 Running 0 66s + rook-ceph-mon-a-c67fdb6c8-tlbvk 1/1 Running 0 2m11s + rook-ceph-mon-b-76969d8685-wcq62 1/1 Running 0 2m2s + rook-ceph-mon-c-5bc47c6cb9-vm4j8 1/1 Running 0 97s + rook-ceph-operator-6fc8cfb68b-bb57z 1/1 Running 1 7m9s + rook-ceph-osd-0-689b6f65b-2nvcx 1/1 Running 0 12s + rook-ceph-osd-1-7bfd69fdf9-vjqmp 1/1 Running 0 4s + rook-ceph-osd-prepare-rook-storage-0-hf28p 0/1 Completed 0 50s + rook-ceph-osd-prepare-rook-storage-1-r6lsd 0/1 Completed 0 50s + rook-ceph-tools-84c7fff88c-x5trx 1/1 Running 0 6m11s + +---------- +Next steps +---------- + +.. include:: ../kubernetes_install_next.txt