starlingx/docs
Code Issues Proposed changes
docs/doc/source/updates/kubernetes/configuring-kubernetes-update-orchestration.rst
Ron Stone c782df8892 Platform k8s upgrades 
Initial draft based on downstream content.
Implemented patchset 1 review comments.
Implemented patchset 2 rewiew comments.
Implemented patchset 3 rewiew comments.
Implemented patchset 4 rewiew comments.
Implemented patchset 5 rewiew comments.
Implemented patchset 6 rewiew comments.

Story: 2008055
Task: 42401

Signed-off-by: Ron Stone <ronald.stone@windriver.com>
Change-Id: I6262018778fae44726985853ec4e01f1abf5b890
Signed-off-by: Ron Stone <ronald.stone@windriver.com>
2021-05-11 08:27:56 -04:00

355 lines
14 KiB
ReStructuredText
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

.. noc1590162360081
.. _configuring-kubernetes-update-orchestration:
==============================================
Kubernetes Version Upgrade Cloud Orchestration
==============================================
You can configure *Kubernetes Version Upgrade Orchestration Strategy* using the
:command:`sw-manager` CLI.
.. note::
You require administrator privileges to use :command:`sw-manager`. You must
log in to the active controller as **user sysadmin** and source the script
by using the command, source /etc/platform/openrc to obtain administrator
privileges. Do not use :command:`sudo`.
.. note::
Management-affecting alarms cannot be ignored using relaxed alarm rules
during an orchestrated Kubernetes version upgrade operation. For a list of
management-affecting alarms, see |fault-doc|: :ref:`Alarm Messages
<100-series-alarm-messages>`. To display management-affecting active
alarms, use the following command:
.. code-block:: none
~(keystone_admin)$ fm alarm-list --mgmt_affecting
During an orchestrated Kubernetes version upgrade operation, the following
alarms are ignored even when the default strict restrictions are selected:
.. _noc1590162360081-ul-vhg-jxs-tlb:
- 100.103: Memory threshold exceeded.
- 200.001: Locked host.
- 280.001: Subcloud resource off-line.
- 280.002: Subcloud resource out-of-sync.
- 700.004: |VM| stopped.
- 750.006: Configuration change requires reapply of cert-manager.
- 900.001: Patch in progress.
- 900.007: Kube upgrade in progress.
- 900.401: kube-upgrade-auto-apply-inprogress.
You can use ``help`` for the overall commands and also for each sub-command.
For example:
.. code-block:: none
~(keystone_admin)$ sw-manager kube-upgrade-strategy –help
usage: sw-manager kube-upgrade-strategy [-h] ...
optional arguments:
-h, --help show this help message and exit
Kubernetes Update Commands:
create Create a strategy
delete Delete a strategy
apply Apply a strategy
abort Abort a strategy
show Show a strategy
.. rubric:: |prereq|
.. _noc1590162360081-ul-ls2-pxs-tlb:
- Hosts that need to be upgraded must be in the ``unlocked-enabled`` state.
.. only:: partner
.. include:: ../../_includes/configuring-kubernetes-update-orchestration.rest
.. rubric:: |proc|
#. List available upgrades.
.. code-block: none
~(keystone_admin)$ system kube-version-list
+-----------------+--------+-----------+
| version | target | state |
+-----------------+--------+-----------+
| v1.18.1 | True | active |
| v1.18.1-upgrade | False | available |
+-----------------+--------+-----------+
#. Create the strategy.
The *Kubernetes Version Upgrade Orchestration Strategy* :command:`create`
command creates a series of stages with steps that apply the Kubernetes
version upgrade.
Kubernetes Version upgrade requires a reboot. Therefore, the created strategy
includes steps that automatically lock and unlock the host to bring the new
image function into service.
.. code-block:: none
~(keystone_admin)$ sw-manager kube-upgrade-strategy create --to-version v1.18.1-upgrade
Strategy Kubernetes Upgrade Strategy:
strategy-uuid: f7585178-cea6-4d2f-bda0-e0972145ebcf
controller-apply-type: serial
storage-apply-type: ignore
worker-apply-type: serial
default-instance-action: migrate
alarm-restrictions: strict
current-phase: build
current-phase-completion: 0%
state: building
inprogress: true
where:
``--to-version``
The version of Kubernetes to upgrade to. For example,
``v1.18.1-upgrade``. This argument is required.
``--controller-apply-type`` and ``--storage-apply-type``
These options cannot be changed from ``serial`` because Kubernetes
upgrade concurrency is only supported for worker hosts.
.. note::
Kubernetes version upgrade is currently only supported for hosts with
worker function. Any attempt to modify the controller or storage
apply type is rejected.
``--worker-apply-type``
This option specifies the host concurrency of the Kubernetes version
upgrade strategy:
- serial \(default\): worker hosts will be patched one at a time
- parallel: worker hosts will be upgraded in parallel
- At most, ``parallel`` will be upgraded at the same time
- At most, half of the hosts in a host aggregate will be upgraded
at the same time
- ignore: worker hosts will not be upgraded; strategy create will fail
Worker hosts with no instances are upgraded before worker hosts with
instances.
``--max-parallel-worker-hosts``
This option applies to the parallel worker apply type selection to
specify the maximum worker hosts to upgrade in parallel \(minimum: 2,
maximum: 10\).
``–instance-action``
This option only has significance when the |prefix|-openstack
application is loaded and there are instances running on worker hosts.
It specifies how the strategy deals with worker host instances over the
strategy execution.
``stop-start`` \(default\)
Instances will be stopped before the host lock operation following the
upgrade and then started again following the host unlock.
.. warning::
Using the ``stop-start`` option will result in an outage for each
instance, as it is stopped while the worker host is locked/unlocked.
In order to ensure this does not impact service, instances MUST be
grouped into anti-affinity \(or anti-affinity best effort\) server
groups, which will ensure that only a single instance in each server
group is stopped at a time.
``migrate``
Instances will be migrated off a host before it is patched \(this
applies to reboot patching only\).
``--alarm-restrictions``
This option sets how the how the Kubernetes version upgrade
orchestration behaves when alarms are present.
To display management-affecting active alarms, use the following
command:
.. code-block:: none
~(keystone_admin)$ fm alarm-list --mgmt_affecting
``strict`` \(default\)
The default strict option will result in patch orchestration failing if
there are any alarms present in the system \(except for a small list of
alarms\).
``relaxed``
This option allows orchestration to proceed if alarms are present, as
long as none of these alarms are management affecting.
.. code-block:: none
~(keystone_admin)]$ sw-manager kube-upgrade-strategy create --help
usage:sw-manager kube-upgrade-strategy [-h]
--to-version <kubernetesVersion>
[--controller-apply-type {ignore}]
[--storage-apply-type {ignore}]
[--worker-apply-type
{serial,parallel,ignore}]
[--max-parallel-worker-hosts
{2,3,4,5,6,7,8,9,10}]
[--instance-action {migrate,stop-start}]
[--alarm-restrictions {strict,relaxed}]
optional arguments:
-h, --help show this help message and exit
--controller-apply-type {serial,ignore}
defaults to serial
--storage-apply-type {serial,ignore}
defaults to serial
--worker-apply-type {serial,parallel,ignore}
defaults to serial
--max-parallel-worker-hosts {2,3,4,5,6,7,8,9,10}
maximum worker hosts to update in parallel
--instance-action {migrate,stop-start}
defaults to stop-start
--alarm-restrictions {strict,relaxed}
defaults to strict
#. Optional: Display the strategy in summary, if required. The Kubernetes
upgrade strategy :command:`show` command displays the strategy in a summary.
.. code-block:: none
~(keystone_admin)$ sw-manager kube-upgrade-strategy show
Strategy Kubernetes Upgrade Strategy:
strategy-uuid: f7585178-cea6-4d2f-bda0-e0972145ebcf
controller-apply-type: serial
storage-apply-type: ignore
worker-apply-type: serial
default-instance-action: migrate
alarm-restrictions: strict
current-phase: build
current-phase-completion: 100%
state: ready-to-apply
build-result: success
build-reason:
The :command:`show` strategy subcommand displays a summary of the current
state of the strategy. A complete view of the strategy can be shown using
the ``--details`` option.
The strategy steps and stages are displayed using the ``--details`` option.
#. Apply the strategy.
*Kubernetes Version Upgrade Orchestration Strategy* :command:`apply` command
executes the strategy stages and steps consecutively until the Kubernetes
upgrade on all the hosts in the strategy is complete.
- Use the ``-stage-id`` option to specify a specific stage to apply; one
at a time.
.. note::
When applying a single stage, only the next stage will be applied;
you cannot skip stages.
.. code-block:: none
~(keystone_admin)$ sw-manager kube-upgrade-strategy apply
Strategy Kubernetes upgrade Strategy:
strategy-uuid: 3e43c018-9c75-4ba8-a276-472c3bcbb268
controller-apply-type: ignore
storage-apply-type: ignore
worker-apply-type: serial
default-instance-action: stop-start
alarm-restrictions: strict
current-phase: apply
current-phase-completion: 0%
state: applying
inprogress: true
- Use the :command:`kube-upgrade-show` command to monitor Kubernetes
upgrade state and percentage completion.
.. code-block:: none
~(keystone_admin)$ system kube-upgrade-show
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | 3d2da123-bff4-4b3a-a64a-b320c3b498cc |
| from_version | v1.18.1 |
| to_version | v1.18.1-upgrade |
| state | downloading-images |
| created_at | 2021-02-23T00:08:24.579257+00:00 |
| updated_at | 2021-02-23T00:09:35.413307+00:00 |
+--------------+--------------------------------------+
You will see the ``state`` property transition through values such as
``downloading-images``, ``downloaded-images``, ``upgrading-first-master``,
``upgraded-first-master``, etc.
#. Optional: Abort the strategy, if required. This is only used to stop, and
abort the entire strategy.
The Kubernetes version upgrade strategy :command:`abort` command can be
used to abort the Kubernetes version upgrade strategy after the current
step of the currently applying stage is completed.
#. Confirm that the upgrade has completed successfully.
.. code-block:: none
~(keystone_admin)$ system kube-upgrade-show
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | 426d7e11-2de2-40ba-b482-ed3691625383 |
| from_version | v1.18.1 |
| to_version | v1.18.1-upgrade |
| state | upgrade-complete |
| created_at | 2021-04-12T17:58:36.492523+00:00 |
| updated_at | 2021-04-12T18:49:11.673259+00:00 |
+--------------+--------------------------------------+
~(keystone_admin)$ system kube-version-list
+-----------------+--------+-----------+
| version | target | state |
+-----------------+--------+-----------+
| v1.18.1 | False | available |
| v1.18.1-upgrade | True | active |
+-----------------+--------+-----------+
#. Delete the strategy.
.. note::
After the *Kubernetes Version Upgrade Orchestration Strategy* has been
applied \(or aborted\) it must be deleted before another Kubernetes
version upgrade strategy can be created. If a Kubernetes version
upgrade strategy application fails, you must address the issue that
caused the failure, then delete and re-create the strategy before
attempting to apply it again.
.. code-block:: none
~(keystone_admin)$ sw-manager kube-upgrade-strategy delete
Strategy deleted.
View Git Blame Copy Permalink