starlingx/docs
Code Issues Proposed changes
docs/doc/source/updates/kubernetes/upgrading-all-in-one-simplex.rst
Juanita-Balaraj fe009c0754 Updated K8 Upgrade Sections of the Distributed Cloud Guide / Updates Guide 
Removed additional Change-ID
Updated Patchset 6 comments
Fixed build errors
Updated Patchset 3 comments
Updated Patchset 2 comments
Added an additional command "system health-query-kube-upgrade"
Updated CLI output to include  [--to-version TO_VERSION]
Updated Patchset 1 comments
Signed-off-by: Juanita-Balaraj <juanita.balaraj@windriver.com>

Change-Id: I1fddb8eef40de8bcdc976b43a3219a653abbf604
2022-02-16 21:28:22 +00:00

419 lines
15 KiB
ReStructuredText
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

.. nfq1592854955302
.. _upgrading-all-in-one-simplex:
==========================
Upgrade All-in-One Simplex
==========================
You can upgrade a |prod| Simplex configuration with a new release of |prod|
software.
.. rubric:: |prereq|
.. _upgrading-all-in-one-simplex-ul-ezb-b11-cx:
- Perform a full backup to allow recovery.
.. note::
Back up files in the /home/sysadmin and /root directories prior to doing
an upgrade. Home directories are not preserved during backup or restore
operations, blade replacement, or upgrades.
- Ensure that the following conditions are met:
- The system is patch current.
- There should be sufficient free space in /opt/platform-backup. Remove
any unused files if necessary.
- The new software load has been imported.
- A valid license file has been installed.
- Transfer the new release software load to controller-0 \(or onto a USB
stick\); controller-0 must be active.
- Transfer the new release software license file to controller-0 \(or onto a
USB stick\).
- Transfer the new release software signature to controller-0 \(or onto a USB
stick\).
.. note::
The upgrade procedure includes steps to resolve system health issues.
End user container images in `registry.local` will be backed up during the
upgrade process. This only includes images other than |prod| system and
application images. These images are limited to 5 GBytes in total size. If
the system contains more than 5 GBytes of these images, the upgrade start will fail.
.. rubric:: |proc|
#. Source the platform environment.
.. code-block:: none
$ source /etc/platform/openrc
~(keystone_admin)]$
#. Install the license file for the release you are upgrading to, for example,
nn.nn.
.. code-block:: none
~(keystone_admin)]$ system license-install <license_file>
For example,
.. code-block:: none
~(keystone_admin)]$ system license-install license.lic
#. Import the new release.
#. Run the :command:`load-import` command on **controller-0** to import
the new release.
First, source /etc/platform/openrc.
You must specify an exact path to the \*.iso bootimage file and to the
\*.sig bootimage signature file.
.. code-block:: none
$ source /etc/platform/openrc
~(keystone_admin)]$ system load-import /home/sysadmin/<bootimage>.iso \
<bootimage>.sig
+--------------------+-----------+
| Property | Value |
+--------------------+-----------+
| id | 2 |
| state | importing |
| software_version | nn.nn |
| compatible_version | nn.nn |
| required_patches | |
+--------------------+-----------+
The :command:`load-import` must be done on **controller-0** and accepts
relative paths.
#. Check to ensure the load was successfully imported.
.. code-block:: none
~(keystone_admin)]$ system load-list
+----+----------+------------------+
| id | state | software_version |
+----+----------+------------------+
| 1 | active | nn.nn |
| 2 | imported | nn.nn |
+----+----------+------------------+
.. note::
This will take a few minutes.
#. Apply any required software updates.
The system must be 'patch current'. All software updates related to your
current |prod| software release must be, uploaded, applied, and installed.
All software updates to the new |prod| release, only need to be uploaded
and applied. The install of these software updates will occur automatically
during the software upgrade procedure as the hosts are reset to load the
new release of software.
To find and download applicable updates, visit the |dnload-loc|.
For more information, see :ref:`Manage Software Updates
<managing-software-updates>`.
#. Confirm that the system is healthy.
Check the current system health status, resolve any alarms and other issues
reported by the :command:`system health-query-upgrade` command, then
recheck the system health status to confirm that all **System Health**
fields are set to **OK**.
.. code-block:: none
~(keystone_admin)]$ system health-query-upgrade
System Health:
All hosts are provisioned: [OK]
All hosts are unlocked/enabled: [OK]
All hosts have current configurations: [OK]
All hosts are patch current: [OK]
Ceph Storage Healthy: [OK]
No alarms: [OK]
All kubernetes nodes are ready: [OK]
All kubernetes control plane pods are ready: [OK]
Required patches are applied: [OK]
License valid for upgrade: [OK]
No instances running on controller-1: [OK]
All kubernetes applications are in a valid state: [OK]
Active controller is controller-0: [OK]
By default, the upgrade process cannot be run and is not recommended to be
run with Active Alarms present. However, management affecting alarms can be
ignored with the :command:`--force` option with the :command:`system
upgrade-start` command to force the upgrade process to start.
.. note::
It is strongly recommended that you clear your system of any and all
alarms before doing an upgrade. While the :command:`--force` option is
available to run the upgrade, it is a best practice to clear any
alarms.
#. Start the upgrade.
.. code-block:: none
~(keystone_admin)]$ system upgrade-start
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | 61e5fcd7-a38d-40b0-ab83-8be55b87fee2 |
| state | starting |
| from_release | nn.nn |
| to_release | nn.nn |
+--------------+--------------------------------------+
This will back up the system data and images to /opt/platform-backup.
/opt/platform-backup is preserved when the host is reinstalled. With the
platform backup, the size of /home/sysadmin must be less than 2GB.
This process may take several minutes.
When the upgrade state is upgraded to **started** the process is complete.
Any changes made to the system after this point will be lost when the data
is restored.
The following upgrade state applies once this command is executed:
- started:
- State entered after :command:`system upgrade-start` completes.
- Release nn.nn system data \(for example, postgres databases\) has
been exported to be used in the upgrade.
- Configuration changes must not be made after this point, until the
upgrade is completed.
As part of the upgrade, the upgrade process checks the health of the system
and validates that the system is ready for an upgrade.
The upgrade process checks that no alarms are active before starting an
upgrade.
.. note::
Use the command :command:`system upgrade-start --force` to force the
upgrades process to start and to ignore management affecting alarms.
This should ONLY be done if you feel these alarms will not be an issue
over the upgrades process.
#. Check the upgrade state.
.. code-block:: none
~(keystone_admin)]$ system upgrade-show
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | 61e5fcd7-a38d-40b0-ab83-8be55b87fee2 |
| state | started |
| from_release | nn.nn |
| to_release | nn.nn |
+--------------+--------------------------------------+
Ensure the upgrade state is **started**. It will take several minutes to
transition to the started state.
#. \(Optional\) Copy the upgrade data from the system to an alternate safe
location \(such as a USB drive or remote server\).
The upgrade data is located under /opt/platform-backup. Example file names
are:
**lost+found upgrade\_data\_2020-06-23T033950\_61e5fcd7-a38d-40b0-ab83-8be55b87fee2.tgz**
.. code-block:: none
~(keystone_admin)]$ ls /opt/platform-backup/
#. Lock controller-0.
.. code-block:: none
~(keystone_admin)]$ system host-lock controller-0
#. Upgrade controller-0.
This is the point of no return. All data except /opt/platform-backup/ will
be erased from the system. This will wipe the **rootfs** and reboot the
host. The new release must then be manually installed \(via network or
USB\).
.. code-block:: none
~(keystone_admin)]$ system host-upgrade controller-0
WARNING: THIS OPERATION WILL COMPLETELY ERASE ALL DATA FROM THE SYSTEM.
Only proceed once the system data has been copied to another system.
Are you absolutely sure you want to continue? [yes/N]: yes
#. Install the new release of |prod-long| Simplex software via network or USB.
#. Verify and configure IP connectivity. External connectivity is required to
run the Ansible upgrade playbook. The |prod-long| boot image will DHCP out all
interfaces so the server may have obtained an IP address and have external IP
connectivity if a DHCP server is present in your environment. Verify this using
the :command:`ip addr` command. Otherwise, manually configure an IP address and default IP
route.
#. Restore the upgrade data.
.. code-block:: none
~(keystone_admin)]$ ansible-playbook /usr/share/ansible/stx-ansible/playbooks/upgrade_platform.yml
Once the host has installed the new load, this will restore the upgrade
data and migrate it to the new load.
The playbook can be run locally or remotely and must be provided with the
following parameter:
``ansible_become_pass``
The ansible playbook will check /home/sysadmin/<hostname\>.yml for these
user configuration override files for hosts. For example, if running
ansible locally, /home/sysadmin/localhost.yml.
By default the playbook will search for the upgrade data file under
/opt/platform-backup. If required, use the **upgrade\_data\_file**
parameter to specify the path to the **upgrade\_data**.
.. note::
This playbook does not support replay.
.. note::
This can take more than one hour to complete.
Once the data restoration is complete the upgrade state will be set to
**upgrading-hosts**.
#. Check the status of the upgrade.
.. code-block:: none
~(keystone_admin)]$ system upgrade-show
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | 61e5fcd7-a38d-40b0-ab83-8be55b87fee2 |
| state | upgrading-hosts |
| from_release | nn.nn |
| to_release | nn.nn |
+--------------+--------------------------------------+
#. Unlock controller-0.
.. code-block:: none
~(keystone_admin)]$ system host-unlock controller-0
This step is required only for Simplex systems that are not a subcloud.
#. Activate the upgrade.
During the running of the :command:`upgrade-activate` command, new
configurations are applied to the controller. 250.001 \(**hostname
Configuration is out-of-date**\) alarms are raised and are cleared as the
configuration is applied. The upgrade state goes from **activating** to
**activation-complete** once this is done.
.. code-block:: none
~(keystone_admin)]$ system upgrade-activate
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | 61e5fcd7-a38d-40b0-ab83-8be55b87fee2 |
| state | activating |
| from_release | nn.nn |
| to_release | nn.nn |
+--------------+--------------------------------------+
The following states apply when this command is executed.
**activation-requested**
State entered when :command:`system upgrade-activate` is executed.
**activating**
State entered when we have started activating the upgrade by applying
new configurations to the controller and compute hosts.
**activating-hosts**
State entered when applying host-specific configurations. This state is
entered only if needed.
**activation-complete**
State entered when new configurations have been applied to all
controller and compute hosts.
Check the status of the upgrade again to see it has reached
**activation-complete**
.. code-block:: none
.. note::
This can take more than half an hour to complete.
~(keystone_admin)]$ system upgrade-show
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | 61e5fcd7-a38d-40b0-ab83-8be55b87fee2 |
| state | activation-complete |
| from_release | nn.nn |
| to_release | nn.nn |
+--------------+--------------------------------------+
#. Complete the upgrade.
.. code-block:: none
~(keystone_admin)]$ system upgrade-complete
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | 61e5fcd7-a38d-40b0-ab83-8be55b87fee2 |
| state | completing |
| from_release | nn.nn |
| to_release | nn.nn |
+--------------+--------------------------------------+
#. Delete the imported load.
.. code-block:: none
~(keystone_admin)]$ system load-list
+----+----------+------------------+
| id | state | software_version |
+----+----------+------------------+
| 1 | imported | nn.nn |
| 2 | active | nn.nn |
+----+----------+------------------+
~(keystone_admin)]$ system load-delete 1
Deleted load: load 1
.. only:: partner
.. include:: /_includes/upgrading-all-in-one-simplex.rest
:start-after: upgradeAIO-begin
:end-before: upgradeAIO-end
View Git Blame Copy Permalink