starlingx/docs
Code Issues Proposed changes
docs/doc/source/node_management/kubernetes/install-power-metrics-application-a12de3db7478.rst
Elisamara Aoki Gonçalves f3dbc936c2 Remove Available Metrics sub-section (r9,r8) 
Remove sub-section as it is not applicable to stx.
Remove Tech preview from index title.
Add back sub-section Available Metrics and remove examples.

Story: 2011065
Task: 50262

Change-Id: I1c339953e2a95e957f9f04909968aae3142a42ba
Signed-off-by: Elisamara Aoki Gonçalves <elisamaraaoki.goncalves@windriver.com>
2024-11-08 15:27:36 +00:00

669 lines
26 KiB
ReStructuredText
Raw Blame History

.. _install-power-metrics-application-a12de3db7478:
=================================
Install Power Metrics Application
=================================
The Power Metrics app deploys two containers, cAdvisor and Telegraf that
collect metrics about hardware usage. This document describes the technical
preview of the Power Metrics functionality.
.. rubric:: |prereq|
For running power-metrics, your system must have the following drivers:
cpufreq kernel module
exposes per-CPU Frequency over sysfs
(``/sys/devices/system/cpu/cpu%d/cpufreq/scaling_cur_freq``)
msr kernel module
provides access to processor model specific registers over devfs
(``/dev/cpu/cpu%d/msr``)
intel-rapl module
exposes Intel Runtime Power Limiting metrics over sysfs
(``/sys/devices/virtual/powercap/intel-rapl``)
intel-uncore-frequency module
exposes Intel uncore frequency metrics over sysfs
(``/sys/devices/system/cpu/intel_uncore_frequency``)
Uncore events can only be loaded from the following cpu models:
.. list-table::
:widths: 6 25
:header-rows: 1
* - **Model number**
- **Processor name**
* - 0x55
- Intel Skylake-X
* - 0x6A
- Intel IceLake-X
* - 0x6C
- Intel IceLake-D
* - 0x47
- Intel Broadwell-G
* - 0x4F
- Intel Broadwell-X
* - 0x56
- Intel Broadwell-D
* - 0x8F
- Intel Sapphire Rapids X
* - 0xCF
- Intel Emerald Rapids X
Source: https://github.com/influxdata/telegraf/blob/master/plugins/inputs/intel_powerstat/README.md#supported-cpu-models
.. rubric:: |proc|
#. Upload the application.
.. code-block:: none
[sysadmin@controller-0 (keystone_admin)]$ system application-upload /usr/local/share/applications/helm/power-metrics-[version].tgz
#. Apply the application.
.. code-block:: none
[sysadmin@controller-0 (keystone_admin)]$ system application-apply power-metrics
#. Wait until Power Metrics is in applied state.
.. code-block:: none
[sysadmin@controller-0 (keystone_admin)]$ system application-show power-metrics
#. Assign a label to the node.
.. note::
A label must be assigned for the power-metrics to be enabled in a node.
.. code-block:: none
power-metrics:enabled
.. code-block:: none
[sysadmin@controller-0 (keystone_admin)]$ system host-label-assign <node-name> power-metrics=enabled
.. rubric:: |result|
The Power Metrics should be installed and both cAdvisor and Telegraf pods must
be up and running.
.. code-block:: none
sysadmin@controller-0:~$ kubectl get pods -n power-metrics
NAME READY STATUS RESTARTS AGE
cadvisor-v76zx 1/1 Running 0 26h
telegraf-mc6vd 1/1 Running 0 4d7h
It is possible to change some configurations via overrides.
--------
Telegraf
--------
Enable and disable Intel PMU metrics
------------------------------------
You can activate the Intel |PMU| plugin with the following command:
.. code-block:: none
[sysadmin@controller-0 ~(keystone_admin)]$ system helm-override-update power-metrics telegraf power-metrics --set pmu_enabled=true
+----------------+-------------------+
| Property | Value |
+----------------+-------------------+
| name | telegraf |
| namespace | power-metrics |
| user_overrides | pmu_enabled: true |
| | |
+----------------+-------------------+
Override Input Plugins
----------------------
You can change the default input plugins parameters by override.
The default plugin parameters include CPU and package metrics.
The list of available options for both CPU and package metrics can be found on
the powerstat documentation:
https://github.com/influxdata/telegraf/blob/master/plugins/inputs/intel_powerstat/README.md#configuration
.. note::
When overriding, you must inform both metrics parameters (CPU and package),
otherwise the plugin would stop collecting the missing metrics.
Example of overriding the powerstat plugin:
.. rubric:: |proc|
#. Update the input parameters.
.. code-block:: none
[sysadmin@controller-0 ~(keystone_admin)]$ cat telegraf-powerstat.yaml
config:
inputs:
# Default plugins to collect power-metrics data
- intel_powerstat:
cpu_metrics:
- "cpu_frequency"
- "cpu_busy_frequency"
- "cpu_temperature"
- "cpu_c0_state_residency"
- "cpu_c1_state_residency"
- "cpu_c6_state_residency"
- "cpu_busy_cycles"
package_metrics:
- "current_power_consumption"
- "current_dram_power_consumption"
- "thermal_design_power"
- "cpu_base_frequency"
- "uncore_frequency"
- intel_pmu:
event_definitions:
- "/etc/telegraf/events_definition.json"
core_events:
- events:
- INST_RETIRED.ANY
- linux_cpu:
metrics: ["cpufreq"]
#. Apply the override.
.. code-block:: none
[sysadmin@controller-0 ~(keystone_admin)]$ system helm-override-update power-metrics telegraf power-metrics --values telegraf-powerstat.yaml
+----------------+------------------------------------------------+
| Property | Value |
+----------------+------------------------------------------------+
| name | telegraf |
| namespace | power-metrics |
| user_overrides | config: |
| | inputs: |
| | - intel_powerstat: |
| | cpu_metrics: |
| | - cpu_frequency |
| | - cpu_busy_frequency |
| | - cpu_temperature |
| | - cpu_c0_state_residency |
| | - cpu_c1_state_residency |
| | - cpu_c6_state_residency |
| | - cpu_busy_cycles |
| | package_metrics: |
| | - current_power_consumption |
| | - current_dram_power_consumption |
| | - thermal_design_power |
| | - cpu_base_frequency |
| | - uncore_frequency |
| | - intel_pmu: |
| | event_definitions: |
| | - "/etc/telegraf/events_definition.json" |
| | core_events: |
| | - events: |
| | - INST_RETIRED.ANY |
| | - linux_cpu: |
| | metrics: ["cpufreq"] |
| | |
+----------------+------------------------------------------------+
#. Re-apply the application.
.. code-block:: none
[sysadmin@controller-0 ~(keystone_admin)]$ system application-apply power-metrics
.. note::
Power Metrics may increase the scheduling latency due to perf and |MSR|
readings. It was observed that there was a latency impact of around 3 µs on
average, plus spikes with significant increases in maximum latency values.
There was also an impact on the kernel processing time. Applications that
run with priorities at or above 50 in real time kernel isolated CPUs should
allow kernel services to avoid unexpected system behavior.
Configuration Requirement for Power Metrics and linux_cpu
---------------------------------------------------------
If the BIOS is not configured to delegate control to the operating system, the
``linux_cpu`` metrics may not function as expected. Remove ``linux_cpu`` to ensure that
power-metrics operate correctly. In this case, metrics generated by ``linux_cpu``
will not be available.
To verify that the BIOS is properly configured, a frequency driver should be
loaded in Linux. You can check this by running the :command:`cpupower frequency-info` command.
Example:
.. code-block:: none
sysadmin@controller-0:~$ cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 800 MHz - 3.60 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 800 MHz and 2.50 GHz.
The governor "performance" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 2.50 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
If there is no delegation from the BIOS to the operating system, the ``linux_cpu``
module may fail to function correctly. To enable power-metrics, it is necessary
to remove the ``linux_cpu`` module. In this scenario, the performance metrics
generated by the ``linux_cpu`` module will not be available.
Example:
.. code-block:: none
sysadmin@compute-0:~$ cpupower frequency-info
analyzing CPU 0:
no or unknown cpufreq driver is active on this CPU
CPUs which run at the same hardware frequency: Not Available
CPUs which need to have their frequency coordinated by software: Not Available
maximum transition latency: Cannot determine or is not supported.
Not Available
available cpufreq governors: Not Available
Unable to determine current policy
current CPU frequency: Unable to call hardware
current CPU frequency: Unable to call to kernel
boost state support:
Supported: yes
Active: yes
Intel Power Stat Configuration Behavior
---------------------------------------
This section describes the expected behavior for the [[inputs.intel_powerstat]]
configuration for different configuration scenarios:
- Empty configuration
When the ``platform_metrics`` parameter is set to an empty array, as shown
below, all the metrics should be restricted from being returned. This means, no
metrics will be provided in this configuration.
[[inputs.intel_powerstat]]
platform_metrics = []
- Default configuration
With either the default configuration or when the [[inputs.intel_powerstat]]
input is used without specifying platform_metrics, only the following metrics
should be enabled:
current_power_consumption
current_dram_power_consumption
thermal_design_power
This default behavior ensures that only the essential power consumption metrics
are collected.
- Specific platform metrics
If specific metrics are enabled using the following ``platform_metrics``
parameter, only the metrics specified in the ``platform_metrics`` array will be
returned. No other metrics will be included beyond the explicitly listed ones.
[[inputs.intel_powerstat]]
platform_metrics = ["cpu_base_frequency", ...]
Add Input Plugins
-----------------
You can add new plugins by overriding the inputs parameter.
Example of overriding the powerstat plugin:
#. Add the ``cpu_c3_state_residency`` metric to the ``intel_powerstat/cpu_metrics`` plugin.
.. code-block:: none
[sysadmin@controller-0 ~(keystone_admin)]$ cat telegraf-powerstat.yaml
config:
inputs:
# Default plugins to collect power-metrics data
- intel_powerstat:
cpu_metrics:
- "cpu_frequency"
- "cpu_busy_frequency"
- "cpu_temperature"
- "cpu_c0_state_residency"
- "cpu_c1_state_residency"
- "cpu_c3_state_residency"
- "cpu_c6_state_residency"
- "cpu_busy_cycles"
package_metrics:
- "current_power_consumption"
- "current_dram_power_consumption"
- "thermal_design_power"
- "cpu_base_frequency"
- "uncore_frequency"
- intel_pmu:
event_definitions:
- "/etc/telegraf/events_definition.json"
core_events:
- events:
- INST_RETIRED.ANY
- linux_cpu:
metrics: ["cpufreq"]
#. Apply the override.
.. code-block:: none
[sysadmin@controller-0 ~(keystone_admin)]$ system helm-override-update power-metrics telegraf power-metrics --values telegraf-powerstat.yaml
+----------------+------------------------------------------------+
| Property | Value |
+----------------+------------------------------------------------+
| name | telegraf |
| namespace | power-metrics |
| user_overrides | config: |
| | inputs: |
| | - intel_powerstat: |
| | cpu_metrics: |
| | - cpu_frequency |
| | - cpu_busy_frequency |
| | - cpu_temperature |
| | - cpu_c0_state_residency |
| | - cpu_c1_state_residency |
| | - cpu_c3_state_residency |
| | - cpu_c6_state_residency |
| | - cpu_busy_cycles |
| | package_metrics: |
| | - current_power_consumption |
| | - current_dram_power_consumption |
| | - thermal_design_power |
| | - cpu_base_frequency |
| | - uncore_frequency |
| | - intel_pmu: |
| | event_definitions: |
| | - "/etc/telegraf/events_definition.json" |
| | core_events: |
| | - events: |
| | - INST_RETIRED.ANY |
| | - linux_cpu: |
| | metrics: ["cpufreq"] |
| | |
+----------------+------------------------------------------------+
#. Re-apply the application.
.. code-block:: none
[sysadmin@controller-0 ~(keystone_admin)]$ system application-apply power-metrics
Remove Input Plugins
--------------------
You can remove plugins by overriding the inputs parameter.
#. Remove the ``linux_cpu`` plugin.
.. code-block:: none
[sysadmin@controller-0 ~(keystone_admin)]$ cat telegraf-powerstat.yaml
config:
inputs:
# Default plugins to collect power-metrics data
- intel_powerstat:
cpu_metrics:
- "cpu_frequency"
- "cpu_busy_frequency"
- "cpu_temperature"
- "cpu_c0_state_residency"
- "cpu_c1_state_residency"
- "cpu_c3_state_residency"
- "cpu_c6_state_residency"
- "cpu_busy_cycles"
package_metrics:
- "current_power_consumption"
- "current_dram_power_consumption"
- "thermal_design_power"
- "cpu_base_frequency"
- "uncore_frequency"
- intel_pmu:
event_definitions:
- "/etc/telegraf/events_definition.json"
core_events:
- events:
- INST_RETIRED.ANY
#. Apply the override.
.. code-block:: none
[sysadmin@controller-0 ~(keystone_admin)]$ system helm-override-update power-metrics telegraf power-metrics --values telegraf-powerstat.yaml
+----------------+------------------------------------------------+
| Property | Value |
+----------------+------------------------------------------------+
| name | telegraf |
| namespace | power-metrics |
| user_overrides | config: |
| | inputs: |
| | - intel_powerstat: |
| | cpu_metrics: |
| | - cpu_frequency |
| | - cpu_busy_frequency |
| | - cpu_temperature |
| | - cpu_c0_state_residency |
| | - cpu_c1_state_residency |
| | - cpu_c3_state_residency |
| | - cpu_c6_state_residency |
| | - cpu_busy_cycles |
| | package_metrics: |
| | - current_power_consumption |
| | - current_dram_power_consumption |
| | - thermal_design_power |
| | - cpu_base_frequency |
| | - uncore_frequency |
| | - intel_pmu: |
| | event_definitions: |
| | - "/etc/telegraf/events_definition.json" |
| | core_events: |
| | - events: |
| | - INST_RETIRED.ANY |
| | |
+----------------+------------------------------------------------+
#. Re-apply the application.
.. code-block:: none
[sysadmin@controller-0 ~(keystone_admin)]$ system application-apply power-metrics
Modify Telegraf Data Collection Interval
----------------------------------------
Telegraf report its metrics each 10 seconds, but you can modify this time
interval with the following command:
.. code-block:: none
system helm-override-update power-metrics telegraf power-metrics --set config.agent.interval=<time-interval>
--------
cAdvisor
--------
Enable or Disable cAdvisor
--------------------------
To enable or disable cAdvisor, use the following command:
.. code-block:: none
[sysadmin@controller-0 ~(keystone_admin)]$ system helm-override-update power-metrics cadvisor power-metrics --set cadvisor_enabled=true
+----------------+------------------------+
| Property | Value |
+----------------+------------------------+
| name | cadvisor |
| namespace | power-metrics |
| user_overrides | cadvisor_enabled: true |
| | |
+----------------+------------------------+
Reapply the power-metrics application and wait for the pod to restart.
.. code-block:: none
[sysadmin@controller-0 ~(keystone_admin)]$ system application-apply power-metrics
Enable and Disable Perf Events on cAdvisor
------------------------------------------
To enable or disable Perf Events on cAdvisor, use the following command:
.. code-block:: none
[sysadmin@controller-0 ~(keystone_admin)]$ system helm-override-update power-metrics cadvisor power-metrics --set perf_events=true
+----------------+-------------------+
| Property | Value |
+----------------+-------------------+
| name | cadvisor |
| namespace | power-metrics |
| user_overrides | perf_events: true |
| | |
+----------------+-------------------+
Finally, re-apply the power-metrics app, and wait until the pod restarts.
.. code-block:: none
system application-apply power-metrics
----------------------------
Remove the Power Metrics App
----------------------------
To remove the Power metrics app use the following command:
.. code-block:: none
system application-remove power-metrics
Then, use the following command to return the application to the uploaded state:
.. code-block:: none
system application-delete power-metrics
-----------------
Available Metrics
-----------------
With Power Metrics application, we have access to system and hardware level raw
data, enabling to visualize the power usage.
Power Metrics, by default, exposes the data collected from both, cAdvisor and
Telegraf, in the OpenMetrics format.
.. rubric:: **Thermal Design Power**
The Thermal Design Power, or TDP, is the maximum energy available, in watts,
for the processor. The metric name for checking the TDP is:
``powerstat_package_thermal_design_power_watts``.
.. rubric:: **Current Power Consumption**
The current power usage of the system in watts. The metric name for checking
power consumption is ``powerstat_package_current_power_consumption_watts``.
.. rubric:: **Current DRAM Power Consumption**
The current power usage of dram in the system in watts. The metric name for
checking DRAM Consumption is:
``powerstat_package_current_dram_power_consumption_watts``.
.. rubric:: **Current CPU Frequency**
The current CPU frequency of the of the processor. The metric name for
checking the CPU frequency is ``powerstat_core_cpu_frequency_mhz``.
.. rubric:: **CPU Base Frequency**
The base frequency (non-turbo) of the processor, it is the default speed of the
processor. The metric name for checking cpu base frequency is
``powerstat_package_cpu_base_frequency_mhz``.
.. rubric:: **Uncore Frequency**
The application reports the current, maximum, and minimum frequency. The uncore
frequency can be described as the frequency on a processor that is not actually
part of its processor core, like memory controller and cache controller.
You can check the current uncore frequency with the following metric name:
``powerstat_package_uncore_frequency_mhz_cur``, for maximum frequency metric
name is ``powerstat_package_uncore_frequency_limit_mhz_max``, and for minimum
the name ``powerstat_package_uncore_frequency_limit_mhz_min``.
.. rubric:: **Per-cpu minimum and maximum frequency**
The application reports the minimum and maximum frequency that each core of the
processor can achieve. It is possible to check the minimum frequency with the
metric name ``linux_cpu_cpuinfo_min_freq`` or ``linux_cpu_scaling_min_freq``,
and maximum with ``linux_cpu_cpuinfo_max_freq`` or
``linux_cpu_scaling_max_freq``.
.. rubric:: **Per-cpu busy frequency**
Busy frequency is the frequency of a core that has a high utilization. (confirm
this later). It is possible to see the busy frequency with the following metric
name ``powerstat_core_cpu_busy_frequency_mhz``.
.. rubric:: **Per-cpu percentage in C-State**
The application can report the time, in percent, that a core of the processor
spent in each c-state. c-State is the state of the core, in which it can reduce
its power consumption, the higher the c-state the higher the sleep state of
the core. We have in the power metrics the following c-states reports:
- C0 state, in this state, the core is executing normally, it is exposed as
``powerstat_core_cpu_c0_state_residency_percent``.
- C1 state, in this state, the core is active but it's not processing any
instructions, it can quickly go back to the C0 state, it is exposed as
``powerstat_core_cpu_c1_state_residency_percent``.
- C6 State, in this state the core is with its voltage reduced (or powered
off). This is the highest state. It takes a longer time to go to C0 state,
but the power saving is higher. It is exposed as
``powerstat_core_cpu_c6_state_residency_percent``.
.. rubric:: **Per-cpu current temperature**
The application reports the current temperature of each individual core from
the processor. The current temperature is exposed as the metric name
``powerstat_core_cpu_temperature_celsius``.
.. rubric:: **Container perf events total**
From cAdvisor it is reported the number of performance events that occurred in
a container, it is exposed as ``container_perf_events_total``.
.. rubric:: **Container perf events scaling ratio**
It also reports the scaling ratio, which calculates the ratio of performance
events in a container, it is exposed as
``container_perf_events_scaling_ration``.
View Git Blame Copy Permalink