docs/doc/source/dist_cloud/kubernetes/subclouds-previous-release-management-5e986615cb4b.rst
Elisamara Aoki Gonçalves 371b5585d0 Distributed Cloud review updates/upgrades - draft (r10,dsR10)
Fix updates and upgrades in distributed cloud guide
Review of the Distributed Cloud Guide to adjust to USM
Fix merge conflict https://review.opendev.org/c/starlingx/docs/+/937086
Fix merge conflict with https://review.opendev.org/c/starlingx/docs/+/937352
Created Horizon Section for Orchestrate Subcloud Prestage

Change-Id: I8b288e35ec76fcbc11ccbe7715aefff70722094d
Signed-off-by: Elisamara Aoki Gonçalves <elisamaraaoki.goncalves@windriver.com>
(cherry picked from commit 46630063f40ebeeb5a03fd10441ed825660bae7e)
2025-01-30 23:54:15 +00:00

639 lines
28 KiB
ReStructuredText

.. _subclouds-previous-release-management-5e986615cb4b:
===========================================
Subclouds Previous Major Release Management
===========================================
.. only:: starlingx
Starting with |prod| |v_r8| and subsequent major releases, you can now
install, restore, and manage subclouds of the previous major software
release from a SystemController running the current major software release.
This capability can be used to revert to the previous release (i.e. major
release, or patched major release) when an unforeseen issue is encountered
during a subcloud upgrade or deployment, or to continue to manage (e.g.
update/patch, rehome, etc.) subclouds of the previous major release that
have not yet been upgraded.
.. only:: partner
.. include:: /_includes/deploy-restore-and-manage-subclouds-of-previous-release-5e986615cb4b.rest
:start-after: short-desc-begin
:end-before: short-desc-end
.. note::
This capability is not available for |prod| |v_r7|. If the system
controller is on |prod| |v_r8|, subclouds can be deployed or restored to
either |prod| |v_r8| or |prod| |v_r6|.
.. contents:: |minitoc|
:local:
:depth: 2
.. rubric:: |context|
This section describes the steps for deploying, restoring, and managing
subclouds of the previous major release from a SystemController running the
current major software release.
**Operations Supported for Subclouds of Previous Release**
- New subcloud deployment
- Subcloud re-install
- Subcloud backup removal
- Subcloud restore from backup
- Subcloud prestage (single subcloud)
- Orchestrated subcloud prestage (multiple subclouds)
- Orchestrated subcloud sw-deploy of major release or patched major release
(upgrade) (multiple subclouds)
- Orchestrated subcloud patch of major release (update/patch) (multiple
subclouds)
- Subcloud rehome
- Deploy files upload/show
.. note::
After deployment, subclouds running the previous major release can be managed,
unmanaged, audited, updated, upgraded, or deleted in the same way as
subclouds running the current release.
For more details, see :ref:`orchestrated-subcloud-patch-of-major-release`.
**Operations with Limited Support for Subclouds of Previous Major Release**
- Subcloud Backup
**Limitation:** The option to exclude the ``/opt/patching`` directory from
the backup, which helps to reduce the backup size, does not apply to
subclouds operating on |prod| |v_r6|. For additional details on the list of
configurable system backup parameters and to identify the specific
parameter(s) that do not apply to |prod| |v_r6| subclouds, see
:ref:`running-ansible-backup-playbook-locally-on-the-controller`.
.. warning::
Ensure that the RECONCILED state of each host is **true** before creating a
subcloud backup.
.. only:: starlingx
You cannot create a backup for a subcloud in a failed state when
running an upgrade from |prod| |v_r6| to |prod| |v_r8|.
.. only:: partner
.. include:: /_includes/deploy-restore-and-manage-subclouds-of-previous-release-5e986615cb4b.rest
:start-after: warning-begin
:end-before: warning-end
Use the following command to verify the RECONCILED state.
.. code-block::
kubectl get hosts -n deployment
- Subcloud Error Reporting
**Limitation:** In the event of a failure during subcloud install, bootstrap,
deployment, backup, restore, or upgrade, you can use the
:command:`dcmanager subcloud errors` command to obtain detailed information
about the failure.
The error reporting for backup and restore failures may vary depending on
the release version. For example, in the case of a backup failure during a
subcloud upgrade due to insufficient disk space, a |prod| |v_r8| subcloud
will generate a generic error report stating, ``Failed to generate upgrade
data. Please check sysinv.log on the subcloud for details``. The same
failure on |prod| |v_r8| subcloud would give a more detailed error report
such as, ``Not enough free space in /opt/platform-backup. It has
2010876KiB. It needs at least 3558912KiB``.
- Orchestrated subcloud patch of major release (update/patch) (multiple
subclouds)
**Operations Not Supported for Subclouds of Previous Release**
- Orchestrated subcloud kubernetes upgrade
- Orchestrated subcloud firmware update
- Admin network configuration
.. rubric:: |prereq|
To support subcloud install or subcloud restore to either the current (N) or
previous (N-1) release, ensure that the following prerequisites are met:
- The N load (pre-patched ISO) containing this capability must be imported to
the system controller using the :command:`software --os-region-name SystemController upload`
command. The ISO imported should be at the same patch level as the system
controller. This requirement ensures that the subcloud boot image is
aligned with the patch level of the load to be installed on the subcloud.
.. note::
Each time a new patch is applied on the system controller, it is
necessary to re-upload the new pre-patched ISO containing the same
patch(es).
- The only latest uploaded ISO (i.e. GA version of ISO or Patched ISO) for
each major release is kept at the DC SystemController.
To display latest uploaded ISO for a particular major release, at the DC
SystemController run:
.. code-block:: none
~(keystone_admin)]$ software list
.. _n-1-load-uploaded-system-controller:
- The N-1 load (pre-patched ISO) must be uploaded to the system controller
using the the following command:
.. code-block:: none
~(keystone_admin)]$ software --os-region-name SystemController upload <bootimage>.iso <bootimage>.sig
- The central registry (or source registry configured for the subclouds) must
contain platform and applications container images of both N and N-1
releases.
- Subcloud install values, bootstrap values, and deploy config files must be
compatible with the release the subcloud is being deployed to.
.. warning::
Since orchestrated Kubernetes upgrades are not supported for subclouds
running the previous release, these subclouds must be deployed with the
highest Kubernetes version for that release. Do not set the
``kubernetes_version`` value in subcloud ``bootstrap-values.yaml`` when
deploying a new N-1 subcloud. If this value is not the highest
Kubernetes version, the subcloud add request will be rejected.
.. warning::
If a subcloud was initially deployed in one release and re-deployed to
a different release, ensure that ``persistent_size`` value in the
subcloud install values is equal to or greater than ``persistent_size``
specified in the previous deployment. Failing to do so may result in a
subcloud installation failure, as the persistent
``/opt/platform-backup`` partition cannot be reduced in size.
- If the system controller was installed with default settings, it will use
the latest Kubernetes version for the release it's running. To support
upgrading the subcloud, ensure that the system controller has locally
stored copies (caches) of container images for all the previous versions of
Kubernetes relevant to the platform release. For example, if the system
controller is installed with Kubernetes 1.24, we need to ensure that all
the system images related to Kubernetes 1.21, 1.22, and 1.23 are stored
locally (cached) on the system controller for use when the subcloud tries
to upgrade to those versions.
To obtain more information about the list of images, see the corresponding
pre-build ISO and docker images section for |prod| |v_r6| and |v_r8|.
.. only:: starlingx
- `The pre-built ISO (CentOS and Debian) and Docker images for StarlingX r6 <https://mirror.starlingx.windriver.com/mirror/starlingx/release/latest_release/debian/monolithic/outputs/iso/>`_.
- `The pre-built ISO (Debian) for StarlingX r8 <https://mirror.starlingx.windriver.com/mirror/starlingx/release/8.0.0/debian/monolithic/outputs/iso/>`_.
.. only:: partner
.. include:: /_includes/deploy-restore-and-manage-subclouds-of-previous-release-5e986615cb4b.rest
:start-after: images-id-begin
:end-before: images-id-end
Upload Deploy/Prestage files
----------------------------
Use the :command:`dcmanager subcloud deploy upload` command with the
``--release`` option to upload deploy or prestage artifacts for a specific
major release version.
.. rubric:: |proc|
#. To upload deploy files use :command:`dcmanager subcloud deploy upload`
command with ``--deploy-playbook``, ``--deploy-chart`` and
``--deploy-overrides`` options. The following command uploads deploy files
for |prod| |v_r8|:
.. code-block:: none
~(keystone_admin)]$ dcmanager subcloud deploy upload
--deploy-playbook <deployment-manager-yaml-file>
--deploy-chart <deployment-manager-chart-tarball>
--deploy-overrides <deployment-manager-overrides-subcloud-yaml-file>
--release <xx.xx>
Where, <xx.xx> is the |prod| |v_r8| major version.
.. note::
Run the :command:`software --os-region-name SystemController upload
<bootimage>.iso <bootimage>.sig` command to automatically upload deploy
files for N-1 release.
It is not mandatory to upload deploy files for the N release. If the
deploy files are not uploaded, the software will use the default deploy
files included in the active load of the SystemController (running the
N release).
#. To upload prestage images list use :command:`dcmanager subcloud deploy
upload` command with ``--prestage-images`` option. The following command
uploads the images list for |prod| |v_r6|:
.. code-block:: none
~(keystone_admin)]$ dcmanager subcloud deploy upload --prestage-images <images-list-file> --release <xx.xx>
Where, <xx.xx> is the |prod| |v_r6| version.
#. View uploaded files for a particular release.
.. code-block:: none
~(keystone_admin)]$ dcmanager subcloud deploy show
--release <previous_or_current_major_release_number>
.. note::
If the ``--release`` option is not specified, the deploy/images list files
are uploaded or shown for the current major release.
Deploy a New Subcloud
---------------------
Use the :command:`dcmanager subcloud add` command with the ``--release`` option
to deploy a subcloud with the specified major release version.
To initiate the deployment of a subcloud running |prod| release |v_r6|, ensure
that the required |prod| |v_r6| deploy files are uploaded, and that the
specific subcloud configurations for |prod| |v_r6| are properly prepared and
run the following command:
.. note::
If controller-1 of the system controller is the active controller, make
sure that `/var/www/pages/feed/rel-<version>` exists before deploying a
|prod| |v_r6| subcloud. Use the :command:`scp` command to copy the entire
directory from controller-0 to controller-1.
.. code-block:: none
~(keystone_admin)]$ dcmanager subcloud add
--bootstrap-address <oam_ip_address_of_subclouds_controller-0>
--bootstrap-values /home/sysadmin/subcloud1-bootstrap-values.yaml
--install-values /home/sysadmin/subcloud1-install-values.yaml
--bmc-password <bmc_password>
--deploy-config /home/sysadmin/subcloud1-deployment-config.yaml
--sysadmin-password <sysadmin_password>
--release <xx.xx>
Where, <xx.xx> is the |prod| |v_r6| major version.
.. note::
If the ``--release`` option is not specified, the subcloud will be deployed
to the current major release of the system controller.
.. warning::
If the value of `software_version` specified in the subcloud
`install-values.yaml` does not align with the ``--release`` value in the
:command:`subcloud add` command, the operation will fail. It is recommended
to exclude `software_version` from the subcloud's ``install-values.yaml``
configuration to prevent potential conflicts.
.. note::
To deploy a patched ISO, you must have previsouly updated the
SystemController with the current ISO for the N-1 major release, see
:ref:`prerequisite <n-1-load-uploaded-system-controller>`.
Restore a Subcloud
------------------
.. note::
If a subcloud needs to be reverted to the previous major release after an
upgrade failure, it is recommended to take a backup of the subcloud before
the upgrade using the :command:`dcmanager subcloud-backup create` command.
For additional information, see
:ref:`backup-a-subcloud-group-of-subclouds-using-dcmanager-cli-f12020a8fc42`.
.. note::
A subcloud cannot be restored from a backup that was taken before it was
deleted and redeployed.
To restore a subcloud from a backup of the previous major release, ensure that
the pre-patched ISO for the previous major release, that aligns with the backup
being restored, has been uploaded at the SystemController (i.e. see line
:ref:`prerequisite <n-1-load-uploaded-system-controller>`), and use the
:command:`dcmanager subcloud-backup restore` command with the ``--release``
option.
.. code-block:: none
~(keystone_admin)]$ dcmanager subcloud-backup restore --subcloud <subcloud> --with-install --sysadmin-password <sysadmin_password> --release <xx.xx>
Where, <xx.xx> is the |prod| |v_r6| major version.
For additional information about the restore, refer to "Restore a single subcloud"
section in
:ref:`restore-a-subcloud-group-of-subclouds-from-backup-data-using-dcmanager-cli-f10c1b63a95e`.
.. note::
If the ``--release`` option is not specified, the subcloud will be
installed to the current major release and restored from a backup of the
current release.
.. warning::
The restore operation will fail if the backup is either non-existent or
incompatible with the release version to which it is being restored.
Delete a Subcloud Backup
------------------------
To delete a subcloud backup of a particular major release, use the
:command:`dcmanager subcloud-backup delete` command with the major release
number specified. For more information, see
:ref:`delete-subcloud-backup-data-using-dcmanager-cli-9cabe48bc4fd`.
- To delete |prod| |v_r8| backup data from central storage, use the following
command.
.. code-block:: none
~(keystone_admin)]$ dcmanager subcloud-backup delete <xx.xx> --subcloud
Where, <xx.xx> is the |prod| |v_r8| major version.
- To delete |prod| |v_r6| backup data from local storage, use the following
command.
.. code-block:: none
~(keystone_admin)]$ dcmanager subcloud-backup delete <xx.xx> --local-only --subcloud --sysadmin-password <sysadmin_password>
Where, <xx.xx> is the |prod| |v_r6| major version.
Prestage a Subcloud
-------------------
Use the :command:`dcmanager subcloud prestage` command with the ``--release``
option to prestage a subcloud for the specified software release. A subcloud
can be prestaged with data from either the current or the previous release,
irrespective of its current software release version.
The format of prestaged container images depends on the software release. For
|prod| |v_r6| release (CentOS) in the production environment, images are
created in the docker image archive format. However, starting from |prod|
|v_r8| and beyond (Debian), images are uploaded to the local registry,
and a snapshot of the local registry is captured. This format significantly
enhances the speed and efficiency of restoration during subcloud upgrades,
re-deployment, or restore processes.
Prestage a Subcloud with Current Release Data
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Before prestaging the subcloud(s) with N release data, be sure to complete one
or both of the following steps:
#. Import the pre-patched ISO for the N release as an active load using the
following command:
.. code-block:: none
~(keystone_admin)]$ software --os-region-name SystemController upload <bootimage>.iso <bootimage>.sig
#. |optional| Upload the N release images list by using the following command:
.. code-block:: none
~(keystone_admin)]$ dcmanager subcloud deploy upload --prestage-images <nn.nn_images.lst>
.. table:: Table 1. Container Images Prestage Behavior for |prod| |v_r8| and higher
:widths: auto
+----------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------+
| Subcloud Software Version | N Release Images List Available | N Release Images List Not Available |
+==============================================+=================================================================================================================================================================================================================================================================================================+========================================================================================+
| |prod| |v_r7| | The container images specified in the list are downloaded from the configured source(s) and pushed to the subcloud local registry. A snapshot of the local registry is then taken and saved in the prestage directory. Finally, downloaded images are removed from the subcloud local registry. | Images prestage is skipped. |
+----------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------+
| |prod| |v_r8| | The container images specified in the list are downloaded from the configured source(s) and pushed to the subcloud local registry. A snapshot of the local registry is then taken and saved in the prestage directory. | A snapshot of the subcloud local registry is taken and saved in the prestage directory.|
+----------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------+
.. note::
It is recommended to repeat subcloud prestaging for the current release
after a new patch of the current release has been applied on the system
controller. This requirement ensures that the subcloud boot image is
aligned with the patch level of the load to be installed on the subcloud
using prestaged data.
Prestage a Subcloud with N-1 Release Data
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Before prestaging the subcloud(s) with N-1 release data, be sure to complete
one or both of the following steps:
#. Import the pre-patched ISO for the N-1 release as an inactive load using
the following command:
.. code-block:: none
~(keystone_admin)]$ software --os-region-name SystemController upload <bootimage>.iso <bootimage>.sig
#. |optional| Upload the N-1 release images list by using the following
command:
.. code-block:: none
~(keystone_admin)]$ dcmanager subcloud deploy upload --prestage-images --release <N-1-major-software-version> <nn.nn_images.lst>
.. table:: Table 2. Container Images Prestage Behavior for |prod| |v_r6|
:widths: auto
+----------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------+
| Subcloud Software Version | N-1 Release Images List Available | N-1 Release Images List Not Available |
+==============================================+==================================================================================================================================================================================+=============================================================================================+
| |prod| |v_r6| | The container images specified in the list are downloaded from the configured source(s). Subsequently, Docker image bundles are generated and saved in the prestage directory. | Image bundles are generated from local registry content and saved in the Prestage directory.|
+----------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------+
| |prod| |v_r8| | The container images specified in the list are downloaded from the configured source(s). Subsequently, Docker image bundles are generated and saved in the prestage directory. | Images prestage is skipped. |
+----------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------+
- To prestage a single subcloud, use :command:`dcmanager subcloud prestage`
command. The following commands can be used to prestage a single subcloud
with |prod| |v_r6| data:
.. code-block:: none
~(keystone_admin)]$ dcmanager subcloud prestage --release <xx.xx> --sysadmin-password <sysadmin_password> <subcloud-name-or-id>
or
.. code-block:: none
~(keystone_admin)]$ dcmanager subcloud prestage --for-install --sysadmin-password <sysadmin-password> --release <xx.xx> <subcloud-name-or-id>
Where, <xx.xx> is the |prod| |v_r6| major version.
- To prestage a group of subclouds use :command:`dcmanager prestage-strategy`
related commands. The following commands can be used to prestage a group of
subclouds with |prod| |v_r8| data:
.. code-block:: none
~(keystone_admin)]$ dcmanager prestage-strategy create --release <xx.xx> --sysadmin-password <sysadmin_password> --group <group_name>
~(keystone_admin)]$ dcmanager prestage-strategy apply
or
.. code-block:: none
~(keystone_admin)]$ dcmanager prestage-strategy create --for-install --release <xx.xx> --sysadmin-password <sysadmin_password> --group <group_name>
~(keystone_admin)]$ dcmanager prestage-strategy apply
Where, <xx.xx> is the |prod| |v_r8| major version.
.. _orchestrated-subcloud-patch-of-major-release:
Orchestrated subcloud patch of major release
--------------------------------------------
To update/patch a subcloud of a previous major release, use the legacy patch
orchestration :command:`dcmanager patch-strategy create` command with the patch
ID specified.
#. Prestage the patch first using the ``--upload-only`` parameter.
.. code-block:: none
~(keystone_admin)]$ dcmanager patch-strategy create --upload-only <patch-file>
#. Create a patch strategy for specific patch, using the following command.
.. code-block:: none
~(keystone_admin)]$ dcmanager patch-strategy create <patch_id>
#. Apply the patch strategy.
.. code-block:: none
~(keystone_admin)]$ dcmanager patch-strategy apply
#. Delete a patch strategy for specific patch, using the following command.
.. code-block:: none
~(keystone_admin)]$ dcmanager patch-strategy delete
.. note::
Upgrading a subcloud from a previous release sometimes requires a patch to
enable Software Deploy orchestration.
Rehome a Subcloud
-----------------
N-1 subclouds can be migrated to another system controller running N software
release. Use the :command:`dcmanager subcloud add --migrate` command with the
``--release`` option to rehome a subcloud. It is not possible to migrate a
subcloud running N software release to a system controller running N-1 software
release.
.. note::
A backup of the subcloud should be taken prior to the rehoming procedure
in the event that a subcloud restore is required to recover from a failed
migration.
**See**: :ref:`Backup a Subcloud/Group of Subclouds using DCManager CLI <backup-a-subcloud-group-of-subclouds-using-dcmanager-cli-f12020a8fc42>`
Rehome |prod| |v_r6| Subcloud to |prod| |v_r8| System Controller
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To migrate |prod| |v_r6| subcloud from system controller A to system controller
B running |prod| |v_r8|, ensure that the |prod| |v_r6| pre-patched ISO has been
imported to System Controller B as an inactive load, and follow the |prod|
|v_r6| subcloud rehome procedure as follows:
#. Unmanage the subcloud from the previous system controller.
#. Update the admin password on the subcloud to match the new system
controller, if required.
#. Run the :command:`dcmanager subcloud add` command on the new System
Controller with the ``--migrate`` and ``--release`` options to initiate the
migration:
.. code-block:: none
~(keystone_admin)]$ dcmanager subcloud add --migrate
--bootstrap-address <oam_ip_address_of_subclouds_controller-0>
--bootstrap-values /home/sysadmin/subcloud1-bootstrap-values.yaml
--release <xx.xx>
Where, <xx.xx> is the |prod| |v_r6| version.
#. On the subcloud, lock/unlock the subcloud controller(s) to enable the new
configuration.
#. Use the :command:`dcmanager subcloud list` command to check the subcloud's
status. Ensure that the subcloud is online and fully operational before
proceeding to manage the subcloud.
#. On the new system controller, set the subcloud to ``managed`` and wait for
it to sync.
#. Delete the subcloud from the previous system controller.
For more detailed information, refer to the `Rehome-a-Subcloud
<https://docs.windriver.com/bundle/wind_river_cloud_platform_dist_cloud_21.12/page/rehoming-a-subcloud.html>`_
topic in the |prod| |v_r6| documentation.
Rehome |prod| |v_r8| Subcloud to |prod| |v_r8| System Controller
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
For more detailed information, see :ref:`rehoming-a-subcloud`.
.. note::
You can add the ``--release <xx.xx>`` option to the :command:`dcmanager
subcloud add --migrate` command but it is not mandatory.
Where, <xx.xx> is the |prod| |v_r8| version.