61d9d93684
Fix yaml. Closes-bug: 2080002 Change-Id: Icb58878a62489c6f939ecdb8e447f9be22a768cc Signed-off-by: Elisamara Aoki Goncalves <elisamaraaoki.goncalves@windriver.com>
488 lines
15 KiB
ReStructuredText
488 lines
15 KiB
ReStructuredText
.. _data-streaming-accelerator-db88a67c930c:
|
|
|
|
==========================
|
|
Data Streaming Accelerator
|
|
==========================
|
|
|
|
.. rubric:: |context|
|
|
|
|
Intel® |DSA| is a high-performance data copy and transformation accelerator
|
|
integrated into Intel® processors starting with 4th Generation Intel® Xeon®
|
|
processors. It is targeted for optimizing streaming data movement and
|
|
transformation operations common with applications for high-performance
|
|
storage, networking, persistent memory, and various data processing
|
|
applications.
|
|
|
|
In |DSA| architecture, descriptors specify a work to be done by the device.
|
|
They contain the type of operation to be performed, data address and status
|
|
buffers.
|
|
|
|
A work queue is a queue on the device where the descriptors submitted by
|
|
software clients are stored until they are processed.
|
|
|
|
Intel |DSA| supports two kinds of work queues:
|
|
|
|
- |DWQ|: the work queue is owned by only a single client.
|
|
|
|
- |SWQ|: multiple clients can submit work to the queue. The engine is the
|
|
unit responsible for processing work. A group is a set of work queues and
|
|
engines.
|
|
|
|
.. rubric:: |prereq|
|
|
|
|
- BIOS
|
|
|
|
Intel |DSA| requires Intel Virtualization Technology for Directed I/O
|
|
(VT-d) to be enabled on BIOS. On some systems, you may also need to enable
|
|
|DSA| on Socket Configuration.
|
|
|
|
For example: **Socket Configuration** > **IIO Configuration** > **IOAT
|
|
Configuration** > **Sck0 IOAT Config** > **DSA**
|
|
|
|
- IDXD Driver
|
|
|
|
IDXD driver initialization can be checked using the :command:`dmesg`
|
|
command to print the kernel message buffer.
|
|
|
|
.. code-block:: none
|
|
|
|
sysadmin@controller-0:~$ dmesg | grep "idxd"
|
|
[ 11.094099] idxd 0000:f6:01.0: enabling device (0144 -> 0146)
|
|
[ 11.182431] idxd 0000:f6:01.0: Intel(R) Accelerator Device (v100)
|
|
|
|
- |DSA| Devices
|
|
|
|
Intel |DSA| |PCI| device ID is ``0x0b25``. The following command lists the
|
|
Intel |DSA| devices on the system.
|
|
|
|
.. code-block:: none
|
|
|
|
sysadmin@controller-0:~$ lspci | grep 0b25
|
|
f6:01.0 System peripheral: Intel Corporation Device 0b25
|
|
|
|
Install Intel Device Plugins Operator for Kubernetes
|
|
----------------------------------------------------
|
|
|
|
Intel Device Plugins Operator is a Kubernetes custom controller whose goal is
|
|
to serve the installation and lifecycle management of Intel device plugins for
|
|
Kubernetes. It provides a single point of control for |GPU|, |QAT|, |SGX|,
|
|
|FPGA|, |DSA| and |DLB| devices to a cluster administrators. The |DSA| plugin
|
|
discovers |DSA| work queues and presents them as a node resources.
|
|
|
|
This operator is provided via Intel Device Plugins StarlingX application
|
|
https://opendev.org/starlingx/app-intel-device-plugins.
|
|
|
|
.. rubric:: **Dependencies**
|
|
|
|
Intel Device Plugins Operator depends on node-feature-discovery StarlingX App.
|
|
|
|
.. rubric:: |proc|
|
|
|
|
#. Upload and apply ``node-feature-discovery`` app.
|
|
|
|
.. code-block:: none
|
|
|
|
$ system application-upload /usr/local/share/applications/helm/node-feature-discovery-24.09-<version>.tgz
|
|
$ system application-apply node-feature-discovery
|
|
|
|
#. Upload ``device-plugins-operator`` app.
|
|
|
|
.. code-block:: none
|
|
|
|
$ system application-upload /usr/local/share/applications/helm/intel-device-plugins-operator-24.09-<version>.tgz
|
|
|
|
#. Enable ``intel-device-plugins-dsa`` Helm chart.
|
|
|
|
.. code-block:: none
|
|
|
|
$ system helm-chart-attribute-modify --enabled true intel-device-plugins-operator intel-device-plugins-dsa intel-device-plugins-operator
|
|
|
|
#. Apply ``intel-device-plugins-operator`` app.
|
|
|
|
.. code-block:: none
|
|
|
|
$ system application-apply intel-device-plugins-operator
|
|
|
|
#. Confirm that dsa resources are available.
|
|
|
|
.. code-block:: none
|
|
|
|
$ kubectl get nodes -o go-template='{{range .items}}{{.metadata.name}}{{"\n"}}{{range $k,$v:=.status.allocatable}}{{" "}}{{$k}}{{": "}}{{$v}}{{"\n"}}{{end}}{{end}}' | grep '^\([^ ]\)\|\( dsa\)'
|
|
controller-0
|
|
dsa.intel.com/wq-user-shared: 40
|
|
|
|
Test Case Example
|
|
-----------------
|
|
|
|
The plugin can be tested by deploying a pod using the |VRAN| tools image
|
|
(``stx-debian-tools-dev``).
|
|
|
|
#. Create a yaml file for the test pod:
|
|
|
|
.. code-block:: yaml
|
|
|
|
$ cat << 'EOF' > dsa-accel-config-demo.yml
|
|
apiVersion: v1
|
|
kind: Pod
|
|
metadata:
|
|
name: dsa-accel-config-demo
|
|
labels:
|
|
app: dsa-accel-config-demo
|
|
spec:
|
|
containers:
|
|
- name: dsa-accel-config-demo
|
|
image: docker.io/starlingx/stx-debian-tools-dev:stx.10.0-v1.0.0
|
|
imagePullPolicy: "Always"
|
|
workingDir: "/usr/libexec/accel-config/test/"
|
|
command:
|
|
- "./dsa_user_test_runner.sh"
|
|
args:
|
|
- "--skip-config"
|
|
resources:
|
|
limits:
|
|
dsa.intel.com/wq-user-shared: 1
|
|
securityContext:
|
|
capabilities:
|
|
add:
|
|
["SYS_RAWIO"]
|
|
restartPolicy: Never
|
|
|
|
EOF
|
|
|
|
#. Apply the yaml file.
|
|
|
|
.. code-block:: none
|
|
|
|
$ kubectl apply -f dsa-accel-config-demo.yml
|
|
|
|
Review the job's log:
|
|
|
|
.. code-block:: none
|
|
|
|
$ kubectl logs dsa-accel-config-demo | tail
|
|
[ info] Checking Src & Dst buffers
|
|
[ info] compsts: 1
|
|
[ info] Checking All Tags
|
|
[ info] All Tags Validated
|
|
[ info] verifying task result for 0x5625182865b0
|
|
[ info] Checking Src & Dst buffers
|
|
[ info] compsts: 1
|
|
[ info] Checking All Tags
|
|
[ info] All Tags Validated
|
|
[ info] verifying task result for 0x562518286670
|
|
[ info] Checking Src & Dst buffers
|
|
[ info] compsts: 1
|
|
[ info] Checking All Tags
|
|
[ info] All Tags Validated
|
|
[ info] verifying task result for 0x562518286730
|
|
[ info] Checking Src & Dst buffers
|
|
[ info] compsts: 1
|
|
[ info] Checking All Tags
|
|
[ info] All Tags Validated
|
|
[ info] verifying task result for 0x5625182867f0
|
|
[ info] Checking Src & Dst buffers
|
|
[ info] compsts: 1
|
|
[ info] Checking All Tags
|
|
[ info] All Tags Validated
|
|
[ info] verifying task result for 0x5625182868b0
|
|
[ info] Checking Src & Dst buffers
|
|
[ info] compsts: 1
|
|
[ info] Checking All Tags
|
|
[ info] All Tags Validated
|
|
[ info] verifying task result for 0x562518286970
|
|
[ info] Checking Src & Dst buffers
|
|
[ info] compsts: 1
|
|
[ info] Checking All Tags
|
|
[ info] All Tags Validated
|
|
[ info] verifying task result for 0x562518286a30
|
|
[ info] Checking Src & Dst buffers
|
|
[ info] compsts: 1
|
|
[ info] Checking All Tags
|
|
[ info] All Tags Validated
|
|
[ info] verifying task result for 0x562518286af0
|
|
[ info] Checking Src & Dst buffers
|
|
[ info] compsts: 1
|
|
[ info] Checking All Tags
|
|
[ info] All Tags Validated
|
|
[ info] verifying task result for 0x562518286bb0
|
|
[ info] Checking Src & Dst buffers
|
|
[ info] compsts: 1
|
|
[ info] Checking All Tags
|
|
[ info] All Tags Validated
|
|
+ '[' --skip-config '!=' --skip-config ']'
|
|
|
|
If the pod did not successfully launch, possibly because it could not obtain
|
|
the |DSA| resource, it will be stuck in the Pending status:
|
|
|
|
.. code-block:: none
|
|
|
|
$ kubectl get pods
|
|
NAME READY STATUS RESTARTS AGE
|
|
dsa-accel-config-demo 0/1 Pending 0 7s
|
|
|
|
This can be verified by checking the Events of the pod:
|
|
|
|
.. code-block:: none
|
|
|
|
$ kubectl describe pod dsa-accel-config-demo | grep -A3 Events:
|
|
Events:
|
|
Type Reason Age From Message
|
|
---- ------ ---- ---- -------
|
|
Warning FailedScheduling 2m26s default-scheduler 0/1 nodes are available: 1 Insufficient dsa.intel.com/wq-user-dedicated, 1 Insufficient dsa.intel.com/wq-user-shared.
|
|
|
|
|
|
Customize the configuration
|
|
---------------------------
|
|
|
|
The default configuration uses shared queues for controller-0 node and
|
|
dedicated queues for the remaining nodes. Node specific configuration can be
|
|
passed by defining the config name with ``dsa-<node-name>.conf``.
|
|
|
|
The default ``DSA`` device configuration is as follow:
|
|
|
|
.. code-block:: yaml
|
|
|
|
$ cat << 'EOF' > dsa-override.yml
|
|
overrideConfig:
|
|
dsa.conf: |
|
|
[
|
|
{
|
|
"dev":"dsaX",
|
|
"read_buffer_limit":0,
|
|
"groups":[
|
|
{
|
|
"dev":"groupX.0",
|
|
"read_buffers_reserved":0,
|
|
"use_read_buffer_limit":0,
|
|
"read_buffers_allowed":8,
|
|
"grouped_workqueues":[
|
|
{
|
|
"dev":"wqX.0",
|
|
"mode":"dedicated",
|
|
"size":16,
|
|
"group_id":0,
|
|
"priority":10,
|
|
"block_on_fault":1,
|
|
"type":"user",
|
|
"name":"appX0",
|
|
"threshold":15
|
|
}
|
|
],
|
|
"grouped_engines":[
|
|
{
|
|
"dev":"engineX.0",
|
|
"group_id":0
|
|
},
|
|
]
|
|
},
|
|
{
|
|
"dev":"groupX.1",
|
|
"read_buffers_reserved":0,
|
|
"use_read_buffer_limit":0,
|
|
"read_buffers_allowed":8,
|
|
"grouped_workqueues":[
|
|
{
|
|
"dev":"wqX.1",
|
|
"mode":"dedicated",
|
|
"size":16,
|
|
"group_id":1,
|
|
"priority":10,
|
|
"block_on_fault":1,
|
|
"type":"user",
|
|
"name":"appX1",
|
|
"threshold":15
|
|
}
|
|
],
|
|
"grouped_engines":[
|
|
{
|
|
"dev":"engineX.1",
|
|
"group_id":1
|
|
},
|
|
]
|
|
},
|
|
{
|
|
"dev":"groupX.2",
|
|
"read_buffers_reserved":0,
|
|
"use_read_buffer_limit":0,
|
|
"read_buffers_allowed":8,
|
|
"grouped_workqueues":[
|
|
{
|
|
"dev":"wqX.2",
|
|
"mode":"dedicated",
|
|
"size":16,
|
|
"group_id":2,
|
|
"priority":10,
|
|
"block_on_fault":1,
|
|
"type":"user",
|
|
"name":"appX2",
|
|
"threshold":15
|
|
}
|
|
],
|
|
"grouped_engines":[
|
|
{
|
|
"dev":"engineX.2",
|
|
"group_id":2
|
|
},
|
|
]
|
|
},
|
|
{
|
|
"dev":"groupX.3",
|
|
"read_buffers_reserved":0,
|
|
"use_read_buffer_limit":0,
|
|
"read_buffers_allowed":8,
|
|
"grouped_workqueues":[
|
|
{
|
|
"dev":"wqX.3",
|
|
"mode":"dedicated",
|
|
"size":16,
|
|
"group_id":3,
|
|
"priority":10,
|
|
"block_on_fault":1,
|
|
"type":"user",
|
|
"name":"appX3",
|
|
"threshold":15
|
|
}
|
|
],
|
|
"grouped_engines":[
|
|
{
|
|
"dev":"engineX.3",
|
|
"group_id":3
|
|
},
|
|
]
|
|
},
|
|
]
|
|
}
|
|
]
|
|
EOF
|
|
|
|
|
|
The |DSA| device configuration can be customized via application overrides.
|
|
|
|
For example, the following config uses dedicated queues for all nodes:
|
|
|
|
.. code-block:: yaml
|
|
|
|
$ cat << 'EOF' > dsa-override.yml
|
|
overrideConfig:
|
|
dsa.conf: |
|
|
[
|
|
{
|
|
"dev":"dsaX",
|
|
"read_buffer_limit":0,
|
|
"groups":[
|
|
{
|
|
"dev":"groupX.0",
|
|
"read_buffers_reserved":0,
|
|
"use_read_buffer_limit":0,
|
|
"read_buffers_allowed":8,
|
|
"grouped_workqueues":[
|
|
{
|
|
"dev":"wqX.0",
|
|
"mode":"dedicated",
|
|
"size":16,
|
|
"group_id":0,
|
|
"priority":10,
|
|
"block_on_fault":1,
|
|
"type":"user",
|
|
"name":"appX0",
|
|
"threshold":15
|
|
}
|
|
],
|
|
"grouped_engines":[
|
|
{
|
|
"dev":"engineX.0",
|
|
"group_id":0
|
|
},
|
|
]
|
|
},
|
|
{
|
|
"dev":"groupX.1",
|
|
"read_buffers_reserved":0,
|
|
"use_read_buffer_limit":0,
|
|
"read_buffers_allowed":8,
|
|
"grouped_workqueues":[
|
|
{
|
|
"dev":"wqX.1",
|
|
"mode":"dedicated",
|
|
"size":16,
|
|
"group_id":1,
|
|
"priority":10,
|
|
"block_on_fault":1,
|
|
"type":"user",
|
|
"name":"appX1",
|
|
"threshold":15
|
|
}
|
|
],
|
|
"grouped_engines":[
|
|
{
|
|
"dev":"engineX.1",
|
|
"group_id":1
|
|
},
|
|
]
|
|
},
|
|
{
|
|
"dev":"groupX.2",
|
|
"read_buffers_reserved":0,
|
|
"use_read_buffer_limit":0,
|
|
"read_buffers_allowed":8,
|
|
"grouped_workqueues":[
|
|
{
|
|
"dev":"wqX.2",
|
|
"mode":"dedicated",
|
|
"size":16,
|
|
"group_id":2,
|
|
"priority":10,
|
|
"block_on_fault":1,
|
|
"type":"user",
|
|
"name":"appX2",
|
|
"threshold":15
|
|
}
|
|
],
|
|
"grouped_engines":[
|
|
{
|
|
"dev":"engineX.2",
|
|
"group_id":2
|
|
},
|
|
]
|
|
},
|
|
{
|
|
"dev":"groupX.3",
|
|
"read_buffers_reserved":0,
|
|
"use_read_buffer_limit":0,
|
|
"read_buffers_allowed":8,
|
|
"grouped_workqueues":[
|
|
{
|
|
"dev":"wqX.3",
|
|
"mode":"dedicated",
|
|
"size":16,
|
|
"group_id":3,
|
|
"priority":10,
|
|
"block_on_fault":1,
|
|
"type":"user",
|
|
"name":"appX3",
|
|
"threshold":15
|
|
}
|
|
],
|
|
"grouped_engines":[
|
|
{
|
|
"dev":"engineX.3",
|
|
"group_id":3
|
|
},
|
|
]
|
|
},
|
|
]
|
|
}
|
|
]
|
|
EOF
|
|
|
|
Apply the override file:
|
|
|
|
.. code-block:: none
|
|
|
|
$ system helm-override-update intel-device-plugins-operator intel-device-plugins-dsa intel-device-plugins-operator --values dsa-override.yaml
|
|
|
|
Apply ``intel-device-plugins-operator`` application:
|
|
|
|
.. code-block:: none
|
|
|
|
$ system application-apply intel-device-plugins-operator |