Kubernetes custom configuration support at Runtime (r8,dsR8)

Story: 2009766
Task: 47829

Signed-off-by: Elisamara Aoki Goncalves <elisamaraaoki.goncalves@windriver.com>
Change-Id: I32b9086daeccc6b970c58207ee4b91340a955933

doc/source/system_configuration/kubernetes/kubernetes-custom-configuration-31c1fd41857d.rst

@@ -19,7 +19,7 @@ Custom configuration includes:
     controllers,
 
 -   Configuring options on ``kube-controller-manager`` such as
--   ``node-monitor-period`` and ``pod-eviction-timeout``,
+    ``node-monitor-period`` and ``pod-eviction-timeout``,
 
 -   Configuring options on ``kube-scheduler`` such as feature gates,
 
@@ -30,7 +30,7 @@ Custom configuration includes:
 kube-apiserver configuration
 ----------------------------
 
-The Kubernetes API server validates and configures data for the api objects
+The Kubernetes API server validates and configures data for the API objects
 which include pods, services, replicationcontrollers, and others. The API
 Server services REST operations and provides the frontend to the cluster's
 shared state through which all other components interact.
@@ -52,6 +52,7 @@ Example usage:
      audit-policy-file: "/etc/kubernetes/audit-policy-file.yml"
      default-not-ready-toleration-seconds: "35"
      default-unreachable-toleration-seconds: "35"
+     feature-gates: "SCTPSupport=true,TTLAfterFinished=true,HugePageStorageMediumSize=true,RemoveSelfLink=false,MemoryManager=true"
      enable-admission-plugins: "NodeRestriction,PodNodeSelector"
      event-ttl: "20h"
 
@@ -59,7 +60,7 @@ Example usage:
 **Runtime configuration**
 
 To set, modify or delete a ``kube-apiserver`` parameter use the
-service-parameter add, modify or delete |CLI| command.
+``service-parameter add``, ``modify`` or ``delete`` |CLI| command.
 
 Example usage:
 
@@ -73,7 +74,7 @@ Example usage:
 
   .. note::
 
-    Parameter must not exists on service parameters, otherwise use
+    Parameter must not exist on service parameters, otherwise use
     :command:`modify` command.
 
 - Modify existing parameter
@@ -86,11 +87,15 @@ Example usage:
 
 - Delete parameter
 
-  #.  system service-parameter-list
+  .. code-block:: none
 
-  #.  copy parameter uuid to be deleted
+      system service-parameter-list
 
-  #.  system service-parameter-delete <uuid>
+  Copy parameter uuid to be deleted:
+
+  .. code-block:: none
+
+      system service-parameter-delete <uuid>
 
 
 -------------------------------------
@@ -113,7 +118,6 @@ To set or override a ``kube-controller-manager`` option, add the desired
 parameters to an ``controllermanager_extra_args`` section in the
 ``localhost.yml``.
 
-
 Example usage:
 
 .. code-block::
@@ -128,7 +132,7 @@ Example usage:
 **Runtime configuration**
 
 To set, modify or delete a ``kube-controller-manager`` parameter use the
-service-parameter add, modify or delete |CLI| command.
+``service-parameter add``, ``modify`` or ``delete`` |CLI| command.
 
 Example usage:
 
@@ -142,7 +146,7 @@ Example usage:
 
   .. note::
 
-    Parameter must not exists on service parameters, otherwise use
+    Parameter must not exist on service parameters, otherwise use
     :command:`modify` command.
 
 - Modify existing parameter
@@ -155,11 +159,15 @@ Example usage:
 
 - Delete parameter
 
-  #.  system service-parameter-list
+  .. code-block:: none
 
-  #.  copy parameter uuid to be deleted
+      system service-parameter-list
 
-  #.  system service-parameter-delete <uuid>
+  Copy parameter uuid to be deleted:
+
+  .. code-block:: none
+
+      system service-parameter-delete <uuid>
 
 
 ----------------------------
@@ -170,16 +178,16 @@ The Kubernetes scheduler is a control plane process which assigns Pods to
 Nodes. The scheduler determines which Nodes are valid placements for each Pod
 in the scheduling queue according to constraints and available resources. The
 scheduler then ranks each valid Node and binds the Pod to a suitable Node.
-Multiple different schedulers may be used within a cluster; kube-scheduler is
+Multiple different schedulers may be used within a cluster; ``kube-scheduler``
-the reference implementation.
+is the reference implementation.
 
 For a list of all configurable options of ``kube-scheduler``, see `kube-scheduler
 <https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/>`__.
 
 **Bootstrap configuration**
 
-To set or override a ``kube-scheduler`` option, add the desired parameters to an
+To set or override a ``kube-scheduler`` option, add the desired parameters to
-``scheduler_extra_args`` section in the ``localhost.yml`` .
+an ``scheduler_extra_args`` section in the ``localhost.yml``.
 
 Example usage:
 
@@ -191,7 +199,7 @@ Example usage:
 **Runtime configuration**
 
 To set, modify or delete a ``kube-controller-manager`` parameter use the
-service-parameter add, modify or delete |CLI| command.
+``service-parameter add``, ``modify`` or ``delete`` |CLI| command.
 
 Example usage:
 
@@ -205,7 +213,7 @@ Example usage:
 
   .. note::
 
-    Parameter must not exists on service parameters, otherwise use
+    Parameter must not exist on service parameters, otherwise use
     :command:`modify` command.
 
 - Modify existing parameter
@@ -218,11 +226,15 @@ Example usage:
 
 - Delete parameter
 
-  #.  system service-parameter-list
+  .. code-block:: none
 
-  #.  copy parameter uuid to be deleted
+      system service-parameter-list
 
-  #.  system service-parameter-delete <uuid>
+  Copy parameter uuid to be deleted:
+
+  .. code-block:: none
+
+      system service-parameter-delete <uuid>
 
 
 ----------------------
@@ -234,15 +246,12 @@ The kubelet is the primary "node agent" that runs on each node.
 For a list of all configurable  options, see `Kubelet Configuration (v1beta1)
 <https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/>`__.
 
-To set or override a kubelet option, add the desired parameters to an
+**Bootstrap configuration**
+
+To set or override a kubelet option, add the desired parameters to a
 ``kubelet_configurations`` section in the ``localhost.yml``.
 
 
-.. important::
-
-    Custom Kubelet configuration is not supported during runtime. This feature
-    will be supported in the next release.
-
 Example usage:
 
 .. code-block::
@@ -252,24 +261,88 @@ Example usage:
        MemoryManager: true
        HugePageStorageMediumSize: true
 
----------------------------------------------------------------------------------
+**Runtime configuration**
-apiserver_extra_volumes, controllermanager_extra_volumes, scheduler_extra_volumes
+
----------------------------------------------------------------------------------
+To set, modify or delete a kubelet parameter use the ``service-parameter add``,
+``modify`` or ``delete`` |CLI| command.
+
+The expected structure for existing field types are:
+
+- String, bool, meta/v1.Duration:
+
+  - No structure  defined, quotes may or may not be used.
+
+  - Ex.: ``cgroupDriver=cgroupfs`` or ``cgroupDriver="cgroupfs"``
+
+- int32,int64:
+
+  - No structure defined, quotes may or may not be used.
+
+  - Ex.: ``imageGCLowThresholdPercent=70`` or ``imageGCLowThresholdPercent="70"``
+
+- Array of strings (``[]string``):
+
+  - JSON-like format: ``'["string1","string2","stringN"]'``
+
+  - Ex.: ``clusterDNS='["10.96.0.10"]'``
+
+- ``map[string]string``:  json format.
+
+  - JSON-like format: ``'{"key_string1":"string1","key_string2":"string2","key_stringN":"stringN"}'``
+
+  - Ex: ``evictionHard='{"memory.available":"100Mi", "nodefs.available": "10%","nodefs.inodesFree": "6%", "imagefs.available": "2Gi"}'``
+
+Example usage:
+
+- Add new parameter
+
+  .. code-block:: none
+
+      system service-parameter-add kubernetes kubelet clusterDNS='["10.96.0.10"]'
+
+      system service-parameter-apply kubernetes
+
+  .. note::
+
+      Parameter must not exist on service parameters, otherwise use
+      :command:`modify`` command.
+
+- Modify existent parameter
+
+  .. code-block:: none
+
+      system service-parameter-modify kubernetes kubelet nodeStatusUpdateFrequency="5s"
+
+      system service-parameter-apply kubernetes
+
+- Delete parameter
+
+  .. code-block:: none
+
+      system service-parameter-list
+
+  Copy parameter uuid to be deleted:
+
+  .. code-block:: none
+
+      system service-parameter-delete <uuid>
+
+--------------------------------------------------------------------------------------
+kube-apiserver, kube-controller-manager and kube-scheduler extra-volumes configuration
+--------------------------------------------------------------------------------------
 
 Some options/parameters specified in ``apiserver_extra_args``,
-``controllermanager_extra_args`` and ``scheduler_extra_volumes`` refer to a
+``controller-manager_extra_args`` and ``scheduler extra-args`` refer to
-configuration file. The contents of these files are configured in the bootstrap
+configuration files or directories. Those referenced files or directories must
-overrides (``localhost.yml``) with the ``apiserver_extra_volumes``,
+be mounted as volumes on the corresponding control plane pod using the
-``controllermanager_extra_volumes`` and ``scheduler_extra_volumes``
+``extra-volume`` parameters.
-definitions.
 
+**Bootstrap configuration**
 
-.. important::
+To set or override an ``extra-volumes`` option, add the desired parameters to
-
+the corresponding ``extra-args`` section in the ``localhost.yml``, add the
-    Kubernetes custom configuration of extra-volumes for ``kube-apiserver``,
+desired ``extra-volume`` including the volume details and file contents (if
-    ``kube-controller-manager`` and ``kube-scheduler`` are not supported during
+corresponds).
-    runtime. This feature will be supported in the next release.
-
 
 For instance, if admission plugins are configured and need additional
 configuration, that configuration should be set in a specific file referenced
@@ -335,7 +408,7 @@ Example usage:
              namespace2: name-of-node-selector
 
 The example below enables kubernetes auditing which requires an
-audit-policy-file.yaml file to specify the details of what events should be
+``audit-policy-file.yaml`` file to specify the details of what events should be
 audited.
 
 Example usage:
@@ -362,6 +435,159 @@ Example usage:
        readOnly: false
        pathType: "DirectoryOrCreate"
 
+
+**Runtime configuration**
+
+To set, modify or delete an extra-volume parameter use the ``service-parameter
+add``, ``modify`` or ``delete`` |CLI| command.
+
+Valid extra-volume sections:
+
+- ``kube_apiserver_volumes``
+
+- ``kube_controller_manager_volumes``
+
+- ``kube_scheduler_volumes``
+
+Valid extra-volume parameter fields:
+
+- ``name``:
+
+  - Volume name.
+
+- ``hostPath``:
+
+  - Absolute path in node file system where the file or directory to mount is located.
+
+- ``mounthPath (opc)``:
+
+  - Absolute path in pod file system used to mount the file or directory.
+
+  - Default value: same as hostPath.
+
+- ``pathType (opc)``:
+
+  - The supported values are:
+
+    - DirectoryOrCreate: If nothing exists at the given path, an empty
+      directory will be created there as needed with permission set to 0755,
+      having the same group and ownership with Kubelet.
+
+    - File: A file must exist at the given path.
+
+  - Default value: File.
+
+- ``readOnly (opc)``:
+
+  - The supported values are: true or false.
+
+  - Default value: true.
+
+Valid input formats:
+
+- Pairs of strings separated by commas:
+
+  - Ex.: ``audit-log-dir=hostPath:/var/log/kubernetes/audit,readOnly:false,pathType:DirectoryOrCreate``
+
+- JSON format:
+
+  - Ex.: ``encryption-config='{"name": "encryption-config", "hostPath": "/etc/kubernetes/encryption-provider.yaml", "mountPath": "/etc/kubernetes/encryption-provider.yaml", "readOnly": true, "pathType":"File"}'``
+
+Example usage with two linked configuration files:
+
+An admission controller could be added to mitigates the problem when the API
+server gets flooded by requests to store new Events. The cluster admin can
+specify event rate limits by enabling the EventRateLimit admission controller
+and referencing an EventRateLimit configuration file.
+
+- Add new extra-args parameters
+
+  .. code-block:: none
+
+      system service-parameter-add kubernetes kube_apiserver admission-control-config-file=/etc/kubernetes/admission-control-config-file.yaml
+
+      system service-parameter-add kubernetes kube_apiserver enable-admission-plugins=EventRateLimit
+
+  .. note::
+
+      Parameter must not exist on service parameters, otherwise use
+      :command:`modify` command.
+
+- Add new extra-volume parameters
+
+  .. code-block:: none
+
+      system service-parameter-add kubernetes kube_apiserver_volumes admission-control-config-file=hostPath:/etc/kubernetes/admission-control-config-file.yaml
+
+      system service-parameter-add kubernetes kube_apiserver_volumes eventconfig=hostPath:/etc/kubernetes/eventconfig.yaml
+
+      system service-parameter-apply kubernetes
+
+  .. note::
+
+      Parameter must not exist on service parameters, otherwise use
+      :command:`modify` command.
+
+- Modify existent parameter
+
+  The configuration file name, for example, can be changed. During this
+  operation, the preloaded configuration file will be replaced.
+
+  .. code-block:: none
+
+      system service-parameter-modify kubernetes kube_apiserver_volumes admission-control-config-file=hostPath:/etc/kubernetes/new-admission-control-config-file.yaml
+
+      system service-parameter-apply kubernetes
+
+- Delete parameters
+
+  .. code-block:: none
+
+      system service-parameter-list
+
+  Copy parameter uuid to be deleted:
+
+  .. code-block:: none
+
+      system service-parameter-delete <uuid>
+
+  In the current example, if EventRateLimit is no longer needed, it should be
+  removed from the ``kube_apiserver enable-admission-plugins`` parameter,
+  either by changing its value or by removing the parameter. Then the
+  extra-volume ``kube_apiserver_volumes eventconfig`` parameter can be deleted.
+  If the configuration file is no longer needed, the ``kube_apserver
+  admission-control-config-file`` parameter can also be removed. Then the-extra
+  volume ``kube_apiserver_volumes connections-control-config-file`` can be
+  deleted.
+
+- Configuration Files Examples:
+
+  - ``admission-control-config-file.yaml``
+
+    .. code-block:: none
+
+        apiVersion: apiserver.config.k8s.io/v1
+        kind: AdmissionConfiguration
+        plugins:
+          - name: EventRateLimit
+            path: /etc/kubernetes/eventconfig.yaml
+
+  - ``eventconfig.yaml``
+
+    .. code-block:: none
+
+        apiVersion: eventratelimit.admission.k8s.io/v1alpha1
+        kind: Configuration
+        limits:
+          - type: Namespace
+            qps: 50
+            burst: 100
+            cacheSize: 2000
+          - type: User
+            qps: 10
+            burst: 50
+
+
 -----------------------------
 Complex Example configuration
 -----------------------------