.. _redundant-system-clock-synchronization-89ee23f54fbb:
======================================
Redundant System Clock Synchronization
======================================
.. rubric:: |context|
The ``phc2sys`` application can be configured to accept multiple source clock
inputs. The quality of these sources are compared to user-defined priority
values and the best available source is selected to set the system time.
The quality of the configured sources is continuously monitored by ``phc2sys``
application and will select a new best source if the current source degrades
or if another source becomes higher quality.
The following section describes how to configure ``phc2sys`` with multiple clock
sources, and manage and monitor the ``phc2sys`` application.
|HA| Phc2sys configuration
--------------------------
.. rubric:: |proc|
#. Create a ``phc2sys`` instance with **multiple source selection** enabled.
.. note::
See the :ref:`ha-phc2sys-cong-param` section for additional information
on parameters required for configuration.
.. code-block:: none
# Set bash variables to simplify commands
INSTANCE=phc2sys1
TYPE=phc2sys
IFNAME1=phc1-iface1
IFNAME2=phc1-iface2
HOST=controller-0
.. code-block:: none
# Create instance
system ptp-instance-add $INSTANCE $TYPE
# Create interface(s)
system ptp-interface-add $IFNAME1 $INSTANCE
system ptp-interface-add $IFNAME2 $INSTANCE
# Assign host interfaces
system host-if-ptp-assign $HOST data0 $IFNAME1
system host-if-ptp-assign $HOST data1 $IFNAME2
#. Assign global instance parameters. The parameters described in
:ref:`ha-phc2sys-cong-param` are required for |HA| ``Phc2sys``
functionality. The ``domainNumber`` value can be included if all interfaces
use the same ``ptp4l`` domain. If ``ptp4l`` instances use different domains,
see the interface configuration options for each interface domain numbers.
.. code-block:: none
# Assign required global instance parameters
system ptp-instance-parameter-add $INSTANCE \
ha_enabled=1 \
ha_phc2sys_com_socket=/var/run/$INSTANCE \
domainNumber=24
#. Assign optional global instance parameters. These parameters determine what
clock quality values are required by ``phc2sys`` to consider a source valid
for selection. See :ref:`ha-phc2sys-cong-param` for more details on the
behaviour of these parameters.
.. code-block:: none
system ptp-instance-parameter-add $INSTANCE \
ha_max_gm_clockClass=6
#. Assign CLI options. This parameter is required for |HA| ``Phc2sys``
operation.
.. code-block:: none
system ptp-instance-parameter-add $INSTANCE \
cmdline_opts='-w'
#. Assign required interface parameters. The ``ha_domainNumber`` parameter can
be ommitted if a shared domain number is used and is configured under the
global instance parameters.
.. code-block:: none
system ptp-interface-parameter-add $IFNAME1 \
ha_priority=100 \
ha_uds_address=/var/run/ptp4l-ptp1 \
ha_domainNumber=24
system ptp-interface-parameter-add $IFNAME2 \
ha_priority=90 \
ha_uds_address=/var/run/ptp4l-ptp2 \
ha_domainNumber=25
#. Assign the instance to a host and apply the |PTP| configuration.
.. code-block:: none
system host-ptp-instance-assign $HOST $INSTANCE
system ptp-instance-apply
.. _ha-phc2sys-cong-param:
|HA| Phc2sys Configuration Parameters
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. note::
The parameters listed here can be used in combination with the default
``phc2sys`` parameters. Use the ``phc2sys`` help page for additional
parameters.
Global Instance Parameters
**************************
``ha_enabled``
The ``ha_enabled`` is a global setting to enable or disable the |HA| multiple
``ptp4l`` inputs support. The default is 0 (disabled). Set 1 to enable |HA|
multiple input support.
``ha_stability_timer``
The ``ha_stability_timer`` option is a global setting used to configure the
stability timer. The value is expressed in seconds, and the value 0 disables
the timer. The default value is 0 (disabled).
When the timer is enabled, ``phc2sys`` waits a few seconds after a source
becomes a candidate for selection before allowing it to be selected as the
synchronization source. This can be useful to prevent frequent source switching
if a given source is frequently changing quality. A value of 300 would cause
``phc2sys`` to ensure that a source has had stable quality for 300 seconds
before it is eligible to be used as a synchronization source.
The stability timer only applies to sources that were in a degraded state
and have recovered to meet the selection threshold. If the active source
degrades and ``phc2sys`` determines that a new source must be selected, this
happens without delay.
The ``ha_stability_timer`` is important when the clock has different priorities
(ha_priority) to prevent multiple and consecutive clock switches. When the
service starts with both clocks in a valid state, the higher priority source
is selected, and when it degrades the lower priority is selected quickly. Then
when the higher priority recovers it will only become active after the
stability timer expires.
``ha_phc2sys_com_socket``
Provides a path for the communication socket used for monitoring and
maintenance activities. This value is required for system alarming to check
the ``phc2sys`` state. It is recommended to create the socket in the ``/var/run``
directory, for example, ``ha_phc2sys_com_socket=/var/run/<phc2sys instance name>``
See :ref: ``manage-phc2sys`` for managing interaction with ``phc2sys`` via this
socket.
Source Selection Parameters
***************************
The parameters in this section are applied at the instance level. For example,
``system ptp-instance-parameter-add <phc2sys instance name> ha_max_gm_clockClass=7``.
These parameters set the threshold for a source to be considered eligible for
use in setting the system time. A source that does not meet one or more of the
requirements will not be selected. ``Phc2sys`` then selects the best available
source that meets the threshold. The interface parameter ``ha_priority`` is
used as a tiebreaker for sources of equal quality, with the higher priority
being selected.
The default value for each parameter is in effect even if the parameter is not
explicitly defined in the configuration file.
.. note::
In most cases, users may use the "|GM| clock values" set of parameters to
configure the selection threshold. The local clock values are available for
additional fine tuning if required, but are generally not a good indicator
of the quality of a clock source compared to the |GM| values.
.. Cole we need values for GM
+---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Parameter | Default Value | Range | Description |
+=================================+===============+===============+=====================================================================================================================================================================================================================+
| **GM clock values** | | | |
+---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ha_max_gm_clockClass | 6 | 6-255 | The ha_max_gm_clockClass option is a global setting used to configure the maximal GM clock class requirement. It ranges from 6 to 255 and its default is 6. A source with a higher clockClass will not be selected. |
+---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ha_max_gm_clockAccuracy | 0xfe | 0x00-0xff | A source must have a clockAccuracy value equal or lower than this parameter to be selected. |
+---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ha_max_gm_offsetScaledLogVar | 0xffff | 0x0000-0xffff | A source must have an offsetScaledLogVariance value equal or lower than this parameter to be selected. |
+---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ha_gm_timeTraceable | 0 | 0/1 | If this parameter is set to 1, then a source must have the timeTraceable flag set in order to be selected. |
+---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ha_gm_frequencyTraceable | 0 | 0/1 | If this parameter is set to 1, then a source must have the frequencyTraceable flag set in order to be selected. |
+---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| **Local clock values** | | | |
+---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ha_max_local_clockClass | 255 | 6-255 | Same as ha_max_gm_clockClass, but only considers the local clock class, not the class received from the parent time source. |
+---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ha_max_local_clockAccuracy | 0xfe | 0x00-0xff | Only considers the local clockAccuracy. |
+---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ha_max_local_offsetScaledLogVar | 0xffff | 0x0000-0xffff | Only considers the local offsetScaledLogVariance |
+---------------------------------+---------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Interface Parameters
********************
``ha_uds_address``
The ``ha_uds_address`` is a **required** parameter for each source configured
in ``phc2sys``. This provides a path to the associated ``ptp4l`` instance on
that source which allows ``phc2sys`` to determine the quality of the source.
This value should match the ``uds_address`` parameter of the associated ``ptp4l``
instance. For example, ``ha_uds_address=/var/run/ptp4l-ptp1``.
``ha_priority``
The ``ha_priority`` option is an interface setting used to configure the clock
priority. The lowest priority is 0, highest is 254, and the default value is 0.
If all clock quality parameters are equal between multiple sources, then the
highest priority source is selected.
If multiple sources have the same clock quality and the same priority, then
``phc2sys`` will select the first interface in the configuration file.
An interface configured with ``ha_priority`` 254 will force ``phc2sys`` to
select that source, regardless of its status.
If more then one interface is configured with ``ha_priority`` 254 the first
interface in the configuration file with priority 254 is selected as clock
source.
``ha_domainNumber``
The ``ha_domainNumber`` option is an interface setting to configure the domain
number for a |UDS| socket. It ranges from 0 to 127. The ``ha_domainNumber``
has no default value, if the ``ha_domainNumber`` is not configured for a given
interface, the global ``domainNumber`` setting is used.
This allows for configurations where multiple ``ptp4l`` instances providing
clock source quality information may be in different domains.
If all ``ptp4l`` instances on the system are in the same domain, it is
recommended to use the domainNumber parameter in the global section.
Example Configuration File
~~~~~~~~~~~~~~~~~~~~~~~~~~
The following is an example configuration file for |HA| ``Phc2sys`` which
shows how services can be configured with multiple interfaces tracking separate
``ptp4l`` instances for quality information.
.. code-block:: none
[global]
##
## Default Data Set
##
domainNumber 24
ha_enabled 1
ha_gm_frequencyTraceable 1
ha_gm_timeTraceable 1
ha_max_gm_clockAccuracy 0x20
ha_max_gm_clockClass 6
ha_max_gm_offsetScaledLogVar 0x4e5d
ha_max_local_clockAccuracy 0x20
ha_max_local_clockClass 255
ha_max_local_offsetScaledLogVar 0xffff
ha_phc2sys_com_socket /var/run/phc1
ha_stability_timer 60
logging_level 7
message_tag phc1
[enp81s0f1]
##
## Associated interface: enp81s0f1
##
ha_priority 100
ha_uds_address /var/run/ptp4l-ptp11
[enp138s0f0]
##
## Associated interface: enp138s0f0
##
ha_priority 101
ha_uds_address /var/run/ptp4l-ptp12
.. _manage-phc2sys:
Manage Phc2sys
--------------
The ``ha_phc2sys_com_socket`` displays information about the state of the
source selection algorithm, as well as performs test and troubleshooting
operations.
To perform these operations, locate the socket file for the ``phc2sys`` instance.
This should be configured in the global instance parameters for the
``phc2sys`` instance. For example, ``/var/run/<instance name>``.
These commands must be run on the specific host being managed, as access to the
socket file is required. For example:
.. code-block:: none
[sysadmin@controller-0 ~(keystone_admin)]$ echo -n "<command>" | socat -t 1 - UNIX-CONNECT:<ha_phc2sys_com_socket>
Informational Commands
~~~~~~~~~~~~~~~~~~~~~~
``status``
The ``status`` command prints a list of configured sources along with the
quality characteristics that have been read from the associated ``ptp4l``
instances.
The ``status`` command also indicates which source has been selected.
.. code-block:: none
[sysadmin@controller-0 ~(keystone_admin)]$ echo -n "status" | socat -t 1 - UNIX-CONNECT:/var/run/phc0
act interface priority clockClass clockAcc offset time freq gm.clockClass gm.clockAcc gm.offset
* enp138s0f0 100 248 0xfe 0xffff yes yes 6 0x20 0x4e5d
enp81s0f1 100 248 0xfe 0xffff yes yes 6 0x20 0x4e5d
``forced lock``
The ``forced lock`` command prints "False" if ``phc2sys`` has not been forced to
select a specific interface via a management operation. It prints "True" if
a force lock has been set via a management operation.
.. code-block:: none
[sysadmin@controller-0 ~(keystone_admin)]$ echo -n "forced lock" | socat -t 1 - UNIX-CONNECT:<ha_phc2sys_com_socket>
``valid sources``
The ``valid sources`` command prints the interfaces that meet the quality
threshold for selection. If none of the configured interfaces meet the
quality threshold then the string "None" is printed.
.. code-block:: none
[sysadmin@controller-0 ~(keystone_admin)]$ echo -n "valid sources" | socat -t 1 - UNIX-CONNECT:<ha_phc2sys_com_socket>
``clock source``
The ``clock source`` command returns the current source that is selected by
``phc2sys``.
.. code-block:: none
[sysadmin@controller-0 ~(keystone_admin)]$ echo -n "clock source" | socat -t 1 - UNIX-CONNECT:<ha_phc2sys_com_socket>
Management Operations
~~~~~~~~~~~~~~~~~~~~~
``enable lock``
The ``enable lock <interface>`` command forces ``phc2sys`` to use the given
interface as the time source. This operation can be helpful for debugging
operations that require the clock source to stay fixed.
.. note::
``phc2sys`` can also be forced to select an interface by configuring it
with ``ha_priority=254`` which is preferred if the lock needs to
persist between reboots and restarts of the ``phc2sys`` service.
``Phc2sys`` can only be locked to a single interface at a time.
.. code-block:: none
[sysadmin@controller-0 ~(keystone_admin)]$ echo -n "enable lock enp81s0f0" | socat -t 1 - UNIX-CONNECT:<ha_phc2sys_com_socket>
``disable lock``
The ``disable lock`` command returns ``phc2sys`` to normal operations where
it selects the best quality clock source based on the configured thresholds.
.. code-block:: none
[sysadmin@controller-0 ~(keystone_admin)]$ echo -n "disable lock" | socat -t 1 - UNIX-CONNECT:<ha_phc2sys_com_socket>
``disable source <interface>``
The ``disable source <interface>`` command removes a given interface from the
list of sources considered by ``phc2sys`` for selection. This can be used for
troubleshooting and testing by allowing sources to be removed and re-enabled
without restarting the ``phc2sys`` service.
``Phc2sys`` will not allow a source to be disabled if it is the only source
currently configured and enabled.
.. code-block:: none
[sysadmin@controller-0 ~(keystone_admin)]$ echo -n "disable source enp81s0f0" | socat -t 1 - UNIX-CONNECT:<ha_phc2sys_com_socket>
``enable source <interface>``
The ``enable source <interface>`` command returns a previously disabled
interface to the selection pool, allowing ``phc2sys`` to consider it for
selection based on its quality.
.. code-block:: none
[sysadmin@controller-0 ~(keystone_admin)]$ echo -n "enable source enp81s0f0" | socat -t 1 - UNIX-CONNECT:<ha_phc2sys_com_socket>
|HA| Phc2sys Alarms
-------------------
New alarms have been added to report the state of |HA| ``Phc2sys``. To view
system alarms on |prod|, use the :command:`fm alarm-list` command.
**Phc2sys HA has no source clock**
Example EID:
``host=controller-0.phc2sys=phc0.phc2sys=no-source-clock``
The "no source clock" alarm is raised when ``phc2sys`` does not have any sources
that meet the specified threshold in the user configuration. This is due to the
quality of the available sources not meeting specified thresholds. The alarm
clears when an available source meets the specified quality requirements.
The ``ha_phc2sys_com_socket`` "status" and "valid sources" commands can be used
to assist with troubleshooting this alarm.
**phc2sys HA source clock is not locked to a PRC**
Example EID:
``host=controller-0.phc2sys=ptp01.interface=enp81s0f1.phc2sys=source-clock-no-prc-lock``
The "source clock is not locked to a PRC" alarm is raised when one of the |HA|
``phc2sys`` interfaces is indicating a clockClass that is of lesser quality than
the specified maximum clockClass in the user config. The alarm indicates which
interface is degraded. The presence of this alarm should coincide with ``phc2sys``
selecting a different source if a better one is available. The alarm will clear
when the clockClass for the source meets the selection threshold.
The ``ha_phc2sys_com_socket`` "status" and "valid sources" commands can be used to
assist with troubleshooting this alarm.
**phc2sys HA has selected a lower priority clock source**
Example EID:
``host=controller-0.phc2sys=phc1. phc2sys=source-clock-low-priority``
The "selected lower priority clock source" alarm is raised when ``phc2sys`` is
configured with multiple sources and the selected one that is not the highest
priority. This alarm indicates that there is degraded quality for the highest
priority source and that ``phc2sys`` has taken action to switch. The alarm will be
cleared when the highest priority source becomes available for selection.
**phc2sys HA automatic source selection has been disabled**
Example EID:
``host=controller-0.phc2sys=phc0.phc2sys=forced-clock-selection``
The "automatic source selection disabled" alarm is raised when an operator has
used the ``ha_phc2sys_com_socket`` :command:`enable lock <interface>`command
to force ``phc2sys`` to use a specific source. This alarm is informational and
can be cleared by running the :command:`disable lock` command via the
``ha_phc2sys_com_socket``.
**phc2sys HA source selection algorithm selected new active source**
Example EID:
``host=controller-0.phc2sys=ptp02.interface=enp138s0f0.phc2sys=source-failover``
The "selected new active source" event is raised when ``phc2sys`` switches between
sources for any reason. This is not raised as a persistent alarm, but rather
as an event, viewable via :command:`fm event-list`. This event enables users to
see when source selection events have occurred. There is no action required to
clear this event.
Phc2sys Source Selection Behaviour
----------------------------------
When ``phc2sys`` is configured with multiple sources and "ha_enabled", a selection
algorithm runs repeatedly to determine which source to sync the system time to.
The source selection algorithm operates by determining a list of valid sources
(i.e. sources that meet the configured selection requirements), and then
choosing the source from that list with the highest priority.
When ``phc2sys`` detects that a source's quality has changed, the list is
re-calculated and a new source is selected if a better one has become available.
The source selection operates in the following way:
.. code-block:: none
For each configured clock source:
if local.clockClass > ha_max_local_clockClass -> disqualify source
if local.clockAccuracy > ha_max_local_clockAccuracy -> disqualify source
if local.offsetScaledLogVariance > ha_max_gm_offsetScaledLogVar -> disqualify source
if source.timeTraceable = 0 AND ha_gm_timeTraceable = 1 -> disqualify source
if source.frequencyTraceable = 0 and ha_gm_frequencyTraceable = 1 -> disqualify source
if gm.clockClass > ha_max_gm_clockClass -> disqualify source
if gm.clockAccuracy > ha_max_gm_clockAccuracy -> disqualify source
if gm.offsetScaledLogVariance > ha_max_gm_offsetScaledLogVar -> disqualify source
From list of qualified sources, select the source with highest "ha_priority". If
qualified sources have the same "ha_priority", select the first one as it
appears in the ``phc2sys`` config file.
If there is no valid source, select the source with lowest "local.clockClass".
If "local.clockClass" is tied, select the first source in ``phc2sys`` config file
from among the tied sources.