From fe009c0754b1b1810f15d59424273b6cb4492c5f Mon Sep 17 00:00:00 2001 From: Juanita-Balaraj Date: Mon, 31 Jan 2022 17:04:40 -0500 Subject: [PATCH] Updated K8 Upgrade Sections of the Distributed Cloud Guide / Updates Guide Removed additional Change-ID Updated Patchset 6 comments Fixed build errors Updated Patchset 3 comments Updated Patchset 2 comments Added an additional command "system health-query-kube-upgrade" Updated CLI output to include [--to-version TO_VERSION] Updated Patchset 1 comments Signed-off-by: Juanita-Balaraj Change-Id: I1fddb8eef40de8bcdc976b43a3219a653abbf604 --- ...ate-orchestration-on-distributed-cloud.rst | 15 +++- .../index-dist-cloud-kub-95bef233eef0.rst | 6 +- ...ted-cloud-update-orchestration-process.rst | 22 +++--- ...ing-the-systemcontroller-using-the-cli.rst | 31 ++++---- ...guring-kubernetes-update-orchestration.rst | 74 +++++++++++++++---- .../manual-kubernetes-components-upgrade.rst | 33 ++++++++- ...pgrading-all-in-one-duplex-or-standard.rst | 2 + .../upgrading-all-in-one-simplex.rst | 9 ++- 8 files changed, 141 insertions(+), 51 deletions(-) diff --git a/doc/source/dist_cloud/kubernetes/configuring-kubernetes-update-orchestration-on-distributed-cloud.rst b/doc/source/dist_cloud/kubernetes/configuring-kubernetes-update-orchestration-on-distributed-cloud.rst index 503d123e2..d2aa3be83 100644 --- a/doc/source/dist_cloud/kubernetes/configuring-kubernetes-update-orchestration-on-distributed-cloud.rst +++ b/doc/source/dist_cloud/kubernetes/configuring-kubernetes-update-orchestration-on-distributed-cloud.rst @@ -3,7 +3,7 @@ .. _configuring-kubernetes-update-orchestration-on-distributed-cloud: ==================================================================== -Kubernetes Version Upgrade Distributed Cloud Orchestration Procedure +Configure Kubernetes Version Upgrade Distributed Cloud Orchestration ==================================================================== You can configure a *Kubernetes Version Upgrade Distributed Cloud @@ -130,6 +130,7 @@ controller for access by subclouds. For example: [--max-parallel-subclouds MAX_PARALLEL_SUBCLOUDS] [--stop-on-failure] [--force] [--group GROUP] + [--to-version TO_VERSION] [cloud_name] @@ -158,6 +159,16 @@ controller for access by subclouds. For example: strategy that is only applied to all subclouds in the specified group. If not specified, all subcloud groups are upgraded. + **--force** + Ignore the audit status of subclouds when selecting them for + orchestration. This allows subclouds that are in-sync to be orchestrated. + + **to-version** + Specify a target version for Kubernetes orchestration. The subcloud + will orchestrate to its 'available' version if the 'to-version' is + greater or equal to the available version. The 'to-version' can be a + partial version. For example, if the available version is 1.20.5, + selecting 1.20 would upgrade to that version. #. Optional: Display the strategy in summary, if required. The Kubernetes upgrade strategy :command:`show` command displays the strategy in a summary. @@ -172,7 +183,7 @@ controller for access by subclouds. For example: | subcloud apply type | None | | max parallel subclouds | None | | stop on failure | False | - | state | complete | + | state | initial | | created_at | 2020-11-09 23:00:07.210958 | | updated_at | 2020-11-09 23:01:15.697763 | +------------------------+----------------------------+ diff --git a/doc/source/dist_cloud/kubernetes/index-dist-cloud-kub-95bef233eef0.rst b/doc/source/dist_cloud/kubernetes/index-dist-cloud-kub-95bef233eef0.rst index a98ee99ea..d8e4d9897 100644 --- a/doc/source/dist_cloud/kubernetes/index-dist-cloud-kub-95bef233eef0.rst +++ b/doc/source/dist_cloud/kubernetes/index-dist-cloud-kub-95bef233eef0.rst @@ -108,9 +108,9 @@ FPGA device image update management device-image-update-orchestration ----------------------------------------------------------- -Kubernetes Version Upgrade Distributed Cloud Orchestration ----------------------------------------------------------- +-------------------------------------------------------------------- +Configure Kubernetes Version Upgrade Distributed Cloud Orchestration +-------------------------------------------------------------------- .. toctree:: :maxdepth: 1 diff --git a/doc/source/dist_cloud/kubernetes/the-kubernetes-distributed-cloud-update-orchestration-process.rst b/doc/source/dist_cloud/kubernetes/the-kubernetes-distributed-cloud-update-orchestration-process.rst index bf5e817b4..24dd94261 100644 --- a/doc/source/dist_cloud/kubernetes/the-kubernetes-distributed-cloud-update-orchestration-process.rst +++ b/doc/source/dist_cloud/kubernetes/the-kubernetes-distributed-cloud-update-orchestration-process.rst @@ -2,9 +2,13 @@ .. fez1617811988954 .. _the-kubernetes-distributed-cloud-update-orchestration-process: -==================================================================== -Kubernetes Version Upgrade Distributed Cloud Orchestration Overviews -==================================================================== +=================================================================== +Kubernetes Version Upgrade Distributed Cloud Orchestration Overview +=================================================================== + +Before you create a *Kubernetes Version Upgrade Distributed Cloud Orchestration +Strategy*, ensure that the Kubernetes version on the System Controller has been upgraded +using the procedure documented in :ref:`Manual Kubernetes Version Upgrade `. For an orchestrated Kubernetes version upgrade across a |prod-dc|, you need to first create a *Kubernetes Version Upgrade Distributed Cloud Orchestration @@ -13,18 +17,18 @@ orchestrated across all subclouds of the Distributed Cloud system. You must use the :command:`dcmanager` CLI tool to **create**, and then **apply** the upgrade strategy. A created strategy can be monitored with the -**show** command. For more information, see :ref:`Kubernetes Version Upgrade -Distributed Cloud Orchestration Procedure -`. +**show** command. For more information, see :ref:`Create Kubernetes Version +Upgrade Cloud Orchestration Strategy `. Kubernetes upgrade orchestration automatically iterates through all managed -online subclouds. +online subclouds if you do not specify a group of subclouds to be upgraded. The specific steps involved in a Kubernetes Version Upgrade Distributed Cloud Orchestration for a single or group of hosts in each subcloud includes: .. _fez1617811988954-ol-a1b-v5s-tlb: -#. Create the subcloud kube upgrade strategy based on the new version installed as part of the platform. +#. Creating the subcloud kube upgrade strategy based on the new version installed + as part of the platform. -#. Apply the subcloud kube upgrade strategy. +#. Applying the subcloud kube upgrade strategy. diff --git a/doc/source/dist_cloud/kubernetes/upgrading-the-systemcontroller-using-the-cli.rst b/doc/source/dist_cloud/kubernetes/upgrading-the-systemcontroller-using-the-cli.rst index 2f323c9e0..bcc074b90 100644 --- a/doc/source/dist_cloud/kubernetes/upgrading-the-systemcontroller-using-the-cli.rst +++ b/doc/source/dist_cloud/kubernetes/upgrading-the-systemcontroller-using-the-cli.rst @@ -66,24 +66,27 @@ Follow the steps below to manually upgrade the System Controller: #. Confirm that the system is healthy. Check the current system health status, resolve any alarms and other issues - reported by the :command:`health-query-upgrade` command, then recheck the - system health status to confirm that all **System Health** fields are set - to **OK**. + reported by the :command:`system health-query-upgrade` command then recheck + the system health status to confirm that all **System Health** fields are + set to **OK**. .. code-block:: none ~(keystone_admin)]$ system health-query-upgrade - System Health: - All hosts are provisioned: [OK] - All hosts are unlocked/enabled: [OK] - All hosts have current configurations: [OK] - All hosts are patch current: [OK] - Ceph Storage Healthy: [OK] - No alarms: [OK] - All kubernetes nodes are ready: [OK] - All kubernetes control plane pods are ready: [OK] - Required patches are applied: [OK] - License valid for upgrade: [OK] + System Health: + All hosts are provisioned: [OK] + All hosts are unlocked/enabled: [OK] + All hosts have current configurations: [OK] + All hosts are patch current: [OK] + Ceph Storage Healthy: [OK] + No alarms: [OK] + All kubernetes nodes are ready: [OK] + All kubernetes control plane pods are ready: [OK] + Required patches are applied: [OK] + License valid for upgrade: [OK] + No instances running on controller-1: [OK] + All kubernetes applications are in a valid state: [OK] + Active controller is controller-0: [OK] By default, the upgrade process cannot run and is not recommended to run with active alarms present. It is strongly recommended that you clear your diff --git a/doc/source/updates/kubernetes/configuring-kubernetes-update-orchestration.rst b/doc/source/updates/kubernetes/configuring-kubernetes-update-orchestration.rst index 19b621d18..03dcda22c 100644 --- a/doc/source/updates/kubernetes/configuring-kubernetes-update-orchestration.rst +++ b/doc/source/updates/kubernetes/configuring-kubernetes-update-orchestration.rst @@ -2,9 +2,9 @@ .. noc1590162360081 .. _configuring-kubernetes-update-orchestration: -============================================== -Kubernetes Version Upgrade Cloud Orchestration -============================================== +============================================================== +Create Kubernetes Version Upgrade Cloud Orchestration Strategy +============================================================== You can configure *Kubernetes Version Upgrade Orchestration Strategy* using the :command:`sw-manager` CLI. @@ -76,7 +76,9 @@ For example: - If you are using NetApp Trident, ensure that your NetApp version is compatible with Trident 21.04 before upgrading Kubernetes to version 1.19 - and after updating |prod| to version |prod-ver|. + and after updating |prod| to version |prod-ver|. For more information, + see :ref:`Upgrade the NetApp Trident Software `. + .. only:: partner @@ -90,13 +92,50 @@ For example: .. code-block:: none ~(keystone_admin)$ system kube-version-list - +-----------------+--------+-----------+ - | version | target | state | - +-----------------+--------+-----------+ - | v1.18.1 | True | active | - | v1.19.13 | False | available | - +-----------------+--------+-----------+ + +-----------------+--------+-------------+ + | version | target | state | + +-----------------+--------+-------------+ + | v1.18.1 | True | active | + | v1.19.13 | False | available | + | v1.20.9 | False | unavailable | + | v1.21.8 | False | unavailable | + +-----------------+--------+-------------+ +#. Confirm that the system is healthy. + + Check the current system health status, resolve any alarms and other issues + reported by the :command:`system health-query-kube-upgrade` command then + recheck the system health status to confirm that all **System Health** + fields are set to **OK**. + + By default, the upgrade process cannot be run and is not recommended to be + run with active alarms present. Use the :command:`system kube-upgrade-start --force` + command to force the upgrade process to start and ignore non-management-affecting + alarms. + + .. note:: + It is strongly recommended that you clear your system of any and all + alarms before doing an upgrade. While the :command:`--force` option is + available to run the upgrade, it is a best practice to clear any + alarms. + + .. code-block:: none + + ~(keystone_admin)]$ system health-query-kube-upgrade + System Health: + All hosts are provisioned: [OK] + All hosts are unlocked/enabled: [OK] + All hosts have current configurations: [OK] + All hosts are patch current: [OK] + Ceph Storage Healthy: [OK] + No alarms: [OK] + All kubernetes nodes are ready: [OK] + All kubernetes control plane pods are ready: [OK] + Required patches are applied: [OK] + License valid for upgrade: [OK] + No instances running on controller-1: [OK] + All kubernetes applications are in a valid state: [OK] + Active controller is controller-0: [OK] #. Create the strategy. @@ -335,12 +374,14 @@ For example: +--------------+--------------------------------------+ ~(keystone_admin)$ system kube-version-list - +-----------------+--------+-----------+ - | version | target | state | - +-----------------+--------+-----------+ - | v1.18.1 | False | available | - | v1.19.13 | True | active | - +-----------------+--------+-----------+ + +-----------------+--------+-------------+ + | version | target | state | + +-----------------+--------+-------------+ + | v1.18.1 | True | unavailable | + | v1.19.13 | False | active | + | v1.20.9 | False | available | + | v1.21.8 | False | unavailable | + +-----------------+--------+-------------+ #. Delete the strategy. @@ -356,3 +397,4 @@ For example: ~(keystone_admin)$ sw-manager kube-upgrade-strategy delete Strategy deleted. + diff --git a/doc/source/updates/kubernetes/manual-kubernetes-components-upgrade.rst b/doc/source/updates/kubernetes/manual-kubernetes-components-upgrade.rst index 1addba3de..bd8c499b6 100644 --- a/doc/source/updates/kubernetes/manual-kubernetes-components-upgrade.rst +++ b/doc/source/updates/kubernetes/manual-kubernetes-components-upgrade.rst @@ -51,9 +51,9 @@ and upgrade various systems. version 1.19. .. note:: - The default version on a fresh install will be Kubernetes 1.21.3, while + The default version on a fresh install will be Kubernetes 1.21.8, while on an upgrade from |prod| |prod-ver| it will be 1.18.1. You will need to - upgrade Kubernetes to each version up to 1.21.3 in order to be ready to + upgrade Kubernetes to each version up to 1.21.8 in order to be ready to upgrade to the next version of |prod|. .. rubric:: |proc| @@ -76,7 +76,7 @@ and upgrade various systems. | v1.18.1 | False | unavailable | | v1.19.13| False | unavailable | | v1.20.9 | False | unavailable | - | v1.21.3 | True | active | + | v1.21.8 | True | active | +---------+--------+-------------+ If |prod| was upgraded to |prod-ver|, the following appears: @@ -90,7 +90,7 @@ and upgrade various systems. | v1.18.1 | True | active | | v1.19.13| False | available | | v1.20.9 | False | unavailable | - | v1.21.3 | False | unavailable | + | v1.21.8 | False | unavailable | +---------+--------+-------------+ The following meanings apply to the output shown: @@ -116,6 +116,31 @@ and upgrade various systems. downgrade or it requires an intermediate upgrade first. Kubernetes can be only upgraded one version at a time. +#. Confirm that the system is healthy. + + Check the current system health status, resolve any alarms and other issues + reported by the :command:`system health-query-kube-upgrade` command then + recheck the system health status to confirm that all **System Health** + fields are set to **OK**. + + .. code-block:: none + + ~(keystone_admin)]$ system health-query-kube-upgrade + System Health: + All hosts are provisioned: [OK] + All hosts are unlocked/enabled: [OK] + All hosts have current configurations: [OK] + All hosts are patch current: [OK] + Ceph Storage Healthy: [OK] + No alarms: [OK] + All kubernetes nodes are ready: [OK] + All kubernetes control plane pods are ready: [OK] + Required patches are applied: [OK] + License valid for upgrade: [OK] + No instances running on controller-1: [OK] + All kubernetes applications are in a valid state: [OK] + Active controller is controller-0: [OK] + #. Start the Kubernetes upgrade. For example: diff --git a/doc/source/updates/kubernetes/upgrading-all-in-one-duplex-or-standard.rst b/doc/source/updates/kubernetes/upgrading-all-in-one-duplex-or-standard.rst index 5dda8bdcc..8983028b2 100644 --- a/doc/source/updates/kubernetes/upgrading-all-in-one-duplex-or-standard.rst +++ b/doc/source/updates/kubernetes/upgrading-all-in-one-duplex-or-standard.rst @@ -139,6 +139,8 @@ of |prod| software. Required patches are applied: [OK] License valid for upgrade: [OK] No instances running on controller-1: [OK] + All kubernetes applications are in a valid state: [OK] + Active controller is controller-0: [OK] By default, the upgrade process cannot be run and is not recommended to be run with active alarms present. Use the command :command:`system upgrade-start --force` diff --git a/doc/source/updates/kubernetes/upgrading-all-in-one-simplex.rst b/doc/source/updates/kubernetes/upgrading-all-in-one-simplex.rst index 652a030ab..980cf8a40 100644 --- a/doc/source/updates/kubernetes/upgrading-all-in-one-simplex.rst +++ b/doc/source/updates/kubernetes/upgrading-all-in-one-simplex.rst @@ -132,9 +132,9 @@ the system contains more than 5 GBytes of these images, the upgrade start will f #. Confirm that the system is healthy. Check the current system health status, resolve any alarms and other issues - reported by the :command:`health-query-upgrade` command, then recheck the - system health status to confirm that all **System Health** fields are set - to **OK**. + reported by the :command:`system health-query-upgrade` command, then + recheck the system health status to confirm that all **System Health** + fields are set to **OK**. .. code-block:: none @@ -150,6 +150,9 @@ the system contains more than 5 GBytes of these images, the upgrade start will f All kubernetes control plane pods are ready: [OK] Required patches are applied: [OK] License valid for upgrade: [OK] + No instances running on controller-1: [OK] + All kubernetes applications are in a valid state: [OK] + Active controller is controller-0: [OK] By default, the upgrade process cannot be run and is not recommended to be run with Active Alarms present. However, management affecting alarms can be