diff --git a/doc/requirements.txt b/doc/requirements.txt index df537f714..9ee51d55e 100644 --- a/doc/requirements.txt +++ b/doc/requirements.txt @@ -2,4 +2,4 @@ sphinx>=1.6.1 sphinxcontrib-programoutput sphinxcontrib-httpdomain reno>=2.5.0 # Apache-2.0 -zuul-sphinx +zuul-sphinx>=0.3.0 diff --git a/doc/source/configuration.rst b/doc/source/configuration.rst index ff6fd54b6..865ab687a 100644 --- a/doc/source/configuration.rst +++ b/doc/source/configuration.rst @@ -1,5 +1,7 @@ .. _configuration: +.. default-domain:: zuul + Configuration ============= @@ -16,772 +18,969 @@ and ``providers`` sections:: providers: ... -.. note:: The builder daemon creates a UUID to uniquely identify itself and - to mark image builds in ZooKeeper that it owns. This file will be - named ``builder_id.txt`` and will live in the directory named by the - :ref:`images-dir` option. If this file does not exist, it will be - created on builder startup and a UUID will be created automatically. - The following sections are available. All are required unless otherwise indicated. -.. _webapp-conf: +.. attr:: webapp -webapp ------- + Define the webapp endpoint port and listen address -Define the webapp endpoint port and listen address. + .. attr:: port + :default: 8005 + :type: int -Example:: + The port to provide basic status information - webapp: - port: 8005 - listen_address: '0.0.0.0' + .. attr:: listen_address + :default: 0.0.0.0 -.. _elements-dir: + Listen address for web app -elements-dir ------------- +.. attr:: elements-dir + :example: /path/to/elements/dir + :type: str -If an image is configured to use diskimage-builder and glance to locally -create and upload images, then a collection of diskimage-builder elements -must be present. The ``elements-dir`` parameter indicates a directory -that holds one or more elements. + If an image is configured to use diskimage-builder and glance to locally + create and upload images, then a collection of diskimage-builder elements + must be present. The ``elements-dir`` parameter indicates a directory + that holds one or more elements. -Example:: +.. attr:: images-dir + :example: /path/to/images/dir + :type: str - elements-dir: /path/to/elements/dir + When we generate images using diskimage-builder they need to be + written to somewhere. The ``images-dir`` parameter is the place to + write them. -.. _images-dir: + .. note:: The builder daemon creates a UUID to uniquely identify + itself and to mark image builds in ZooKeeper that it + owns. This file will be named ``builder_id.txt`` and will + live in the directory named by the :attr:`images-dir` + option. If this file does not exist, it will be created on + builder startup and a UUID will be created automatically. -images-dir ----------- -When we generate images using diskimage-builder they need to be -written to somewhere. The ``images-dir`` parameter is the place to -write them. +.. attr:: build-log-dir + :example: /path/to/log/dir + :type: str -Example:: + The builder will store build logs in this directory. It will create + one file for each build, named `-.log`; for example, + `fedora-0000000004.log`. It defaults to ``/var/log/nodepool/builds``. - images-dir: /path/to/images/dir +.. attr:: build-log-retention + :default: 7 + :type: int -.. _build-log-dir: + At the start of each build, the builder will remove old build logs if + they exceed this value. This option specifies how many will be + kept (usually you will see one more, as deletion happens before + starting a new build). By default, the last 7 old build logs are + kept. -build-log-dir -------------- +.. attr:: zookeeper-servers + :type: list + :required: -The builder will store build logs in this directory. It will create -one file for each build, named `-.log`; for example, -`fedora-0000000004.log`. It defaults to ``/var/log/nodepool/builds``. + Lists the ZooKeeper servers uses for coordinating information between + nodepool workers. -Example:: + .. code-block:: yaml - build-log-dir: /path/to/log/dir + zookeeper-servers: + - host: zk1.example.com + port: 2181 + chroot: /nodepool -.. _build-log-retention: + Each entry is a dictionary with the following keys -build-log-retention -------------------- + .. attr:: host + :type: str + :example: zk1.example.com + :required: -At the start of each build, the builder will remove old build logs if -they exceed a certain number. This option specifies how many will be -kept (usually you will see one more, as deletion happens before -starting a new build). By default, the last 7 old build logs are -kept. + A zookeeper host -Example:: + .. attr:: port + :default: 2181 + :type: int - build-log-retention: 14 + Port to talk to zookeeper -zookeeper-servers ------------------ -Lists the ZooKeeper servers uses for coordinating information between -nodepool workers. Example:: + .. attr:: chroot + :type: int + :example: /nodepool - zookeeper-servers: - - host: zk1.example.com - port: 2181 - chroot: /nodepool + The ``chroot`` key, used for interpreting ZooKeeper paths + relative to the supplied root path, is also optional and has no + default. -The ``port`` key is optional (default: 2181). +.. attr:: labels + :type: list -The ``chroot`` key, used for interpreting ZooKeeper paths relative to -the supplied root path, is also optional and has no default. + Defines the types of nodes that should be created. Jobs should be + written to run on nodes of a certain label. Example -.. _labels: + .. code-block:: yaml -labels ------- - -Defines the types of nodes that should be created. Jobs should be -written to run on nodes of a certain label. Example:: - - labels: - - name: my-precise - max-ready-age: 3600 - min-ready: 2 - - name: multi-precise - min-ready: 2 - -**required** - - ``name`` - Unique name used to tie jobs to those instances. - -**optional** - - ``max-ready-age`` (int) - Maximum number of seconds the node shall be in ready state. If - this is exceeded the node will be deleted. A value of 0 disables this. - Defaults to 0. - - ``min-ready`` (default: 0) - Minimum number of instances that should be in a ready - state. Nodepool always creates more nodes as necessary in response - to demand, but setting ``min-ready`` can speed processing by - attempting to keep nodes on-hand and ready for immedate use. - ``min-ready`` is best-effort based on available capacity and is - not a guaranteed allocation. The default of 0 means that nodepool - will only create nodes of this label when there is demand. Set - to -1 to have the label considered disabled, so that no nodes will - be created at all. - -.. _maxholdage: - -max-hold-age ------------- - -Maximum number of seconds a node shall be in "hold" state. If -this is exceeded the node will be deleted. A value of 0 disables this. -Defaults to 0. - -This setting is applied to all nodes, regardless of label or provider. - -.. _diskimages: - -diskimages ----------- - -This section lists the images to be built using diskimage-builder. The -name of the diskimage is mapped to the :ref:`provider_diskimages` section -of the provider, to determine which providers should received uploads of each -image. The diskimage will be built in every format required by the -providers with which it is associated. Because Nodepool needs to know -which formats to build, if the diskimage will only be built if it -appears in at least one provider. - -To remove a diskimage from the system entirely, remove all associated -entries in :ref:`provider_diskimages` and remove its entry from `diskimages`. -All uploads will be deleted as well as the files on disk. - -Example configuration:: - - diskimages: - - name: ubuntu-precise - pause: False - rebuild-age: 86400 - elements: - - ubuntu-minimal - - vm - - simple-init - - openstack-repos - - nodepool-base - - cache-devstack - - cache-bindep - - growroot - - infra-package-needs - release: precise - username: zuul - env-vars: - TMPDIR: /opt/dib_tmp - DIB_CHECKSUM: '1' - DIB_IMAGE_CACHE: /opt/dib_cache - DIB_APT_LOCAL_CACHE: '0' - DIB_DISABLE_APT_CLEANUP: '1' - FS_TYPE: ext3 - - name: ubuntu-xenial - pause: True - rebuild-age: 86400 - formats: - - raw - - tar - elements: - - ubuntu-minimal - - vm - - simple-init - - openstack-repos - - nodepool-base - - cache-devstack - - cache-bindep - - growroot - - infra-package-needs - release: precise - username: ubuntu - env-vars: - TMPDIR: /opt/dib_tmp - DIB_CHECKSUM: '1' - DIB_IMAGE_CACHE: /opt/dib_cache - DIB_APT_LOCAL_CACHE: '0' - DIB_DISABLE_APT_CLEANUP: '1' - FS_TYPE: ext3 - - -**required** - - ``name`` - Identifier to reference the disk image in :ref:`provider_diskimages` - and :ref:`labels`. - -**optional** - - ``formats`` (list) - The list of formats to build is normally automatically created based on the - needs of the providers to which the image is uploaded. To build images even - when no providers are configured or to build additional formats which you - know you may need in the future, list those formats here. - - ``rebuild-age`` - If the current diskimage is older than this value (in seconds), - then nodepool will attempt to rebuild it. Defaults to 86400 (24 - hours). - - ``release`` - Specifies the distro to be used as a base image to build the image using - diskimage-builder. - - ``elements`` (list) - Enumerates all the elements that will be included when building the image, - and will point to the :ref:`elements-dir` path referenced in the same - config file. - - ``env-vars`` (dict) - Arbitrary environment variables that will be available in the spawned - diskimage-builder child process. - - ``pause`` (bool) - When set to True, nodepool-builder will not build the diskimage. - - ``username`` (string) - The username that a consumer should use when connecting to the node. Defaults - to ``zuul``. - -.. _provider: - -providers ---------- - -Lists the providers Nodepool should use. Each provider is associated to -a driver listed below. - -**required** - - ``name`` - - -**optional** - - ``driver`` - Default to *openstack* - - ``max-concurrency`` - Maximum number of node requests that this provider is allowed to handle - concurrently. The default, if not specified, is to have no maximum. Since - each node request is handled by a separate thread, this can be useful for - limiting the number of threads used by the nodepool-launcher daemon. - - -OpenStack driver -^^^^^^^^^^^^^^^^ - -Within each OpenStack provider the available Nodepool image types are defined -(see :ref:`provider_diskimages`). - -An OpenStack provider's resources are partitioned into groups called "pools" -(see :ref:`pools` for details), and within a pool, the node types which are -to be made available are listed (see :ref:`pool_labels` for -details). - -Example:: - - providers: - - name: provider1 - driver: openstack - cloud: example - region-name: 'region1' - rate: 1.0 - boot-timeout: 120 - launch-timeout: 900 - launch-retries: 3 - image-name-format: '{image_name}-{timestamp}' - hostname-format: '{label.name}-{provider.name}-{node.id}' - diskimages: - - name: trusty - meta: - key: value - key2: value - - name: precise - - name: devstack-trusty - pools: - - name: main - max-servers: 96 - availability-zones: - - az1 - networks: - - some-network-name - security-groups: - - zuul-security-group - labels: - - name: trusty - min-ram: 8192 - diskimage: trusty - console-log: True - - name: precise - min-ram: 8192 - diskimage: precise - - name: devstack-trusty - min-ram: 8192 - diskimage: devstack-trusty - - name: provider2 - driver: openstack - cloud: example2 - region-name: 'region1' - rate: 1.0 - image-name-format: '{image_name}-{timestamp}' - hostname-format: '{label.name}-{provider.name}-{node.id}' - diskimages: - - name: precise - meta: - key: value - key2: value - pools: - - name: main - max-servers: 96 - labels: - - name: trusty - min-ram: 8192 - diskimage: trusty - - name: precise - min-ram: 8192 - diskimage: precise - - name: devstack-trusty - min-ram: 8192 - diskimage: devstack-trusty - -**required** - - ``cloud`` - Name of a cloud configured in ``clouds.yaml``. - - The instances spawned by nodepool will inherit the default security group - of the project specified in the cloud definition in `clouds.yaml` (if other - values not specified). This means that when working with Zuul, for example, - SSH traffic (TCP/22) must be allowed in the project's default security group - for Zuul to be able to reach instances. - - More information about the contents of `clouds.yaml` can be found in - `the os-client-config documentation `_. - -**optional** - - ``boot-timeout`` - Once an instance is active, how long to try connecting to the - image via SSH. If the timeout is exceeded, the node launch is - aborted and the instance deleted. - - In seconds. Default 60. - - ``launch-timeout`` - - The time to wait from issuing the command to create a new instance - until that instance is reported as "active". If the timeout is - exceeded, the node launch is aborted and the instance deleted. - - In seconds. Default 3600. - - ``nodepool-id`` (deprecated) - - A unique string to identify which nodepool instances is using a provider. - This is useful if you want to configure production and development instances - of nodepool but share the same provider. - - Default None - - ``launch-retries`` - - The number of times to retry launching a server before considering the job - failed. - - Default 3. - - ``region-name`` - - ``hostname-format`` - Hostname template to use for the spawned instance. - Default ``{label.name}-{provider.name}-{node.id}`` - - ``image-name-format`` - Format for image names that are uploaded to providers. - Default ``{image_name}-{timestamp}`` - - ``rate`` - In seconds, amount to wait between operations on the provider. - Defaults to ``1.0``. - - ``clean-floating-ips`` - If it is set to True, nodepool will assume it is the only user of the - OpenStack project and will attempt to clean unattached floating ips that - may have leaked around restarts. - -.. _pools: - -pools -~~~~~ - -A pool defines a group of resources from an OpenStack provider. Each pool has a -maximum number of nodes which can be launched from it, along with a -number of cloud-related attributes used when launching nodes. - -Example:: - - pools: - - name: main - max-servers: 96 - availability-zones: - - az1 - networks: - - some-network-name - security-groups: - - zuul-security-group - auto-floating-ip: False - host-key-checking: True labels: - - name: trusty - min-ram: 8192 - diskimage: trusty - console-log: True - - name: precise - min-ram: 8192 - diskimage: precise - - name: devstack-trusty - min-ram: 8192 - diskimage: devstack-trusty - -**required** - - ``name`` - - -**optional** - - ``max-cores`` - Maximum number of cores usable from this pool. This can be used to limit - usage of the tenant. If not defined nodepool can use all cores up to the - quota of the tenant. - - ``max-servers`` - Maximum number of servers spawnable from this pool. This can be used to - limit the number of servers. If not defined nodepool can create as many - servers the tenant allows. - - ``max-ram`` - Maximum ram usable from this pool. This can be used to limit the amount of - ram allocated by nodepool. If not defined nodepool can use as much ram as - the tenant allows. - - ``ignore-provider-quota`` - Ignore the provider quota for this pool. Instead, only check against the - configured max values for this pool and the current usage based on stored - data. This may be useful in circumstances where the provider is incorrectly - calculating quota. - - ``availability-zones`` (list) - A list of availability zones to use. - - If this setting is omitted, nodepool will fetch the list of all - availability zones from nova. To restrict nodepool to a subset - of availability zones, supply a list of availability zone names - in this setting. - - Nodepool chooses an availability zone from the list at random - when creating nodes but ensures that all nodes for a given - request are placed in the same availability zone. - - ``networks`` (list) - Specify custom Neutron networks that get attached to each - node. Specify the name or id of the network as a string. - - ``security-groups`` (list) - Specify custom Neutron security groups that get attached to each - node. Specify the name or id of the security_group as a string. - - ``auto-floating-ip`` (bool) - Specify custom behavior of allocating floating ip for each node. - When set to False, nodepool-launcher will not apply floating ip - for nodes. When zuul instances and nodes are deployed in the same - internal private network, set the option to False to save floating ip - for cloud provider. The default value is True. - - ``host-key-checking`` (bool) - Specify custom behavior of validation of SSH host keys. When set to False, - nodepool-launcher will not ssh-keyscan nodes after they are booted. This - might be needed if nodepool-launcher and the nodes it launches are on - different networks. The default value is True. - -.. _provider_diskimages: - -diskimages -~~~~~~~~~~ - -Each entry in a provider's `diskimages` section must correspond to an -entry in :ref:`diskimages`. Such an entry indicates that the -corresponding diskimage should be uploaded for use in this provider. -Additionally, any nodes that are created using the uploaded image will -have the associated attributes (such as flavor or metadata). - -If an image is removed from this section, any previously uploaded -images will be deleted from the provider. - -Example configuration:: - - diskimages: - - name: precise - pause: False - meta: - key: value - key2: value - - name: windows - connection-type: winrm - connection-port: 5986 - -**required** - - ``name`` - Identifier to refer this image from :ref:`labels` and :ref:`diskimages` - sections. - -**optional** + - name: my-precise + max-ready-age: 3600 + min-ready: 2 + - name: multi-precise + min-ready: 2 + + Each entry is a dictionary with the following keys + + .. attr:: name + :type: string + :required: + + Unique name used to tie jobs to those instances. + + .. attr:: max-ready-age + :type: int + :default: 0 + + Maximum number of seconds the node shall be in ready state. If + this is exceeded the node will be deleted. A value of 0 disables + this. + + .. attr:: min-ready + :type: int + :default: 0 + + Minimum number of instances that should be in a ready + state. Nodepool always creates more nodes as necessary in response + to demand, but setting ``min-ready`` can speed processing by + attempting to keep nodes on-hand and ready for immedate use. + ``min-ready`` is best-effort based on available capacity and is + not a guaranteed allocation. The default of 0 means that nodepool + will only create nodes of this label when there is demand. Set + to -1 to have the label considered disabled, so that no nodes will + be created at all. + +.. attr:: max-hold-age + :type: int + :default: 0 + + Maximum number of seconds a node shall be in "hold" state. If this + is exceeded the node will be deleted. A value of 0 disables this. + + This setting is applied to all nodes, regardless of label or + provider. + +.. attr:: diskimages + :type: list + + This section lists the images to be built using + diskimage-builder. The name of the diskimage is mapped to the + :attr:`providers.openstack.diskimages` section of the provider, to + determine which providers should received uploads of each image. + The diskimage will be built in every format required by the + providers with which it is associated. Because Nodepool needs to + know which formats to build, if the diskimage will only be built if + it appears in at least one provider. + + To remove a diskimage from the system entirely, remove all + associated entries in :attr:`providers.openstack.diskimages` and + remove its entry from `diskimages`. All uploads will be deleted as + well as the files on disk. + + .. code-block:: yaml + + diskimages: + - name: ubuntu-precise + pause: False + rebuild-age: 86400 + elements: + - ubuntu-minimal + - vm + - simple-init + - openstack-repos + - nodepool-base + - cache-devstack + - cache-bindep + - growroot + - infra-package-needs + release: precise + username: zuul + env-vars: + TMPDIR: /opt/dib_tmp + DIB_CHECKSUM: '1' + DIB_IMAGE_CACHE: /opt/dib_cache + DIB_APT_LOCAL_CACHE: '0' + DIB_DISABLE_APT_CLEANUP: '1' + FS_TYPE: ext3 + - name: ubuntu-xenial + pause: True + rebuild-age: 86400 + formats: + - raw + - tar + elements: + - ubuntu-minimal + - vm + - simple-init + - openstack-repos + - nodepool-base + - cache-devstack + - cache-bindep + - growroot + - infra-package-needs + release: precise + username: ubuntu + env-vars: + TMPDIR: /opt/dib_tmp + DIB_CHECKSUM: '1' + DIB_IMAGE_CACHE: /opt/dib_cache + DIB_APT_LOCAL_CACHE: '0' + DIB_DISABLE_APT_CLEANUP: '1' + FS_TYPE: ext3 + + Each entry is a dictionary with the following keys + + .. attr:: name + :type: string + :required: + + Identifier to reference the disk image in + :attr:`providers.openstack.diskimages` and :attr:`labels`. + + .. attr:: formats + :type: list + + The list of formats to build is normally automatically created + based on the needs of the providers to which the image is + uploaded. To build images even when no providers are configured + or to build additional formats which you know you may need in the + future, list those formats here. + + .. attr:: rebuild-age + :type: int + :default: 86400 + + If the current diskimage is older than this value (in seconds), + then nodepool will attempt to rebuild it. Defaults to 86400 (24 + hours). + + .. attr:: release + :type: string + + Specifies the distro to be used as a base image to build the image using + diskimage-builder. + + .. attr:: elements + :type: list + + Enumerates all the elements that will be included when building the image, + and will point to the :attr:`elements-dir` path referenced in the same + config file. + + .. attr:: env-vars + :type: dict + + Arbitrary environment variables that will be available in the spawned + diskimage-builder child process. + + .. attr:: pause + :type: bool + :default: False + + When set to True, ``nodepool-builder`` will not build the diskimage. + + .. attr:: username + :type: string + :default: zuul + + The username that a consumer should use when connecting to the + node. + + +.. attr:: providers + :type: list + + Lists the providers Nodepool should use. Each provider is associated to + a driver listed below. + + Each entry is a dictionary with the following keys + + .. attr:: name + :type: string + :required: + + Name of the provider + + .. attr:: max-concurrency + :type: int + :default: 0 + + Maximum number of node requests that this provider is allowed to + handle concurrently. The default, if not specified, is to have + no maximum. Since each node request is handled by a separate + thread, this can be useful for limiting the number of threads + used by the nodepool-launcher daemon. + + .. attr:: driver + :type: string + :default: openstack + + The driver type. + + .. value:: openstack + + For details on the extra options required and provided by the + OpenStack driver, see the separate section + :attr:`providers.openstack` + + .. value:: static + + For details on the extra options required and provided by the + static driver, see the separate section + :attr:`providers.static` + + +OpenStack Driver +---------------- + +.. attr:: providers.openstack + + Specifying the ``openstack`` driver for a provider adds the + following keys to the :attr:`providers` configuration. Note for + documentation purposes the option names are prefixed + ``providers.openstack`` to disambiguate from other drivers, but the + ``openstack`` is not required in the configuration (e.g. below + ``providers.openstack.cloud`` refers to the ``cloud`` key in the + ``providers`` section when the ``openstack`` driver is selected). + + An OpenStack provider's resources are partitioned into groups + called "pools" (see :attr:`providers.openstack.pools` for details), + and within a pool, the node types which are to be made available + are listed (see :attr:`providers.openstack.pools.labels` for + details). + + Within each OpenStack provider the available Nodepool image types + are defined (see :attr:`providers.openstack.diskimages`). + + .. code-block:: yaml + + providers: + - name: provider1 + driver: openstack + cloud: example + region-name: 'region1' + rate: 1.0 + boot-timeout: 120 + launch-timeout: 900 + launch-retries: 3 + image-name-format: '{image_name}-{timestamp}' + hostname-format: '{label.name}-{provider.name}-{node.id}' + diskimages: + - name: trusty + meta: + key: value + key2: value + - name: precise + - name: devstack-trusty + pools: + - name: main + max-servers: 96 + availability-zones: + - az1 + networks: + - some-network-name + security-groups: + - zuul-security-group + labels: + - name: trusty + min-ram: 8192 + diskimage: trusty + console-log: True + - name: precise + min-ram: 8192 + diskimage: precise + - name: devstack-trusty + min-ram: 8192 + diskimage: devstack-trusty + - name: provider2 + driver: openstack + cloud: example2 + region-name: 'region1' + rate: 1.0 + image-name-format: '{image_name}-{timestamp}' + hostname-format: '{label.name}-{provider.name}-{node.id}' + diskimages: + - name: precise + meta: + key: value + key2: value + pools: + - name: main + max-servers: 96 + labels: + - name: trusty + min-ram: 8192 + diskimage: trusty + - name: precise + min-ram: 8192 + diskimage: precise + - name: devstack-trusty + min-ram: 8192 + diskimage: devstack-trusty + + .. attr:: cloud + :type: string + :required: + + Name of a cloud configured in ``clouds.yaml``. + + The instances spawned by nodepool will inherit the default + security group of the project specified in the cloud definition + in `clouds.yaml` (if other values not specified). This means + that when working with Zuul, for example, SSH traffic (TCP/22) + must be allowed in the project's default security group for Zuul + to be able to reach instances. + + More information about the contents of `clouds.yaml` can be + found in `the os-client-config documentation + `_. + + .. attr:: boot-timeout + :type: int seconds + :default: 60 + + Once an instance is active, how long to try connecting to the + image via SSH. If the timeout is exceeded, the node launch is + aborted and the instance deleted. + + .. attr:: launch-timeout + :type: int seconds + :default: 3600 + + The time to wait from issuing the command to create a new instance + until that instance is reported as "active". If the timeout is + exceeded, the node launch is aborted and the instance deleted. + + .. attr:: nodepool-id + :type: string + :default: None + + *Deprecated* + + A unique string to identify which nodepool instances is using a + provider. This is useful if you want to configure production + and development instances of nodepool but share the same + provider. + + .. attr:: launch-retries + :type: int + :default: 3 + + The number of times to retry launching a server before + considering the job failed. + + .. attr:: region-name + :type: string + + The region name if the provider cloud has multiple regions. + + .. attr:: hostname-format + :type: string + :default: {label.name}-{provider.name}-{node.id} + + Hostname template to use for the spawned instance. + + .. attr:: image-name-format + :type: string + :default: {image_name}-{timestamp} + + Format for image names that are uploaded to providers. + + .. attr:: rate + :type: int seconds + :default: 1 + + In seconds, amount to wait between operations on the provider. + + .. attr:: clean-floating-ips + :type: bool + :default: True + + If it is set to True, nodepool will assume it is the only user of + the OpenStack project and will attempt to clean unattached + floating ips that may have leaked around restarts. + + .. attr:: diskimages + :type: list + + Each entry in a provider's `diskimages` section must correspond + to an entry in :attr:`diskimages`. Such an entry indicates that + the corresponding diskimage should be uploaded for use in this + provider. Additionally, any nodes that are created using the + uploaded image will have the associated attributes (such as + flavor or metadata). - ``pause`` (bool) - When set to True, nodepool-builder will not upload the image to the - provider. + If an image is removed from this section, any previously uploaded + images will be deleted from the provider. - ``config-drive`` (boolean) - Whether config drive should be used for the image. Defaults to unset which - will use the cloud's default behavior. + .. code-block:: yaml - ``meta`` (dict) - Arbitrary key/value metadata to store for this server using the Nova - metadata service. A maximum of five entries is allowed, and both keys and - values must be 255 characters or less. - - ``connection-type`` (string) - The connection type that a consumer should use when connecting to the - node. For most diskimages this is not necessary. However when creating - Windows images this could be 'winrm' to enable access via ansible. - - ``connection-port`` (int) - The port that a consumer should use when connecting to the - node. For most diskimages this is not necessary. This defaults to 22 for - ssh and 5986 for winrm. - -.. _provider_cloud_images: - -cloud-images -~~~~~~~~~~~~ - -Each cloud-image entry in :ref:`labels` refers to an entry in this section. -This is a way for modifying launch parameters of the nodes (currently only -config-drive). + diskimages: + - name: precise + pause: False + meta: + key: value + key2: value + - name: windows + connection-type: winrm + connection-port: 5986 -Example configuration:: + Each entry is a dictionary with the following keys - cloud-images: - - name: trusty-external - config-drive: False - - name: windows-external - connection-type: winrm - connection-port: 5986 + .. attr:: name + :type: string + :required: -**required** + Identifier to refer this image from + :attr:`providers.openstack.pools.labels` and + :attr:`diskimages` sections. - ``name`` - Identifier to refer this cloud-image from :ref:`labels` section. - Since this name appears elsewhere in the nodepool configuration - file, you may want to use your own descriptive name here and use - one of ``image-id`` or ``image-name`` to specify the cloud image - so that if the image name or id changes on the cloud, the impact - to your Nodepool configuration will be minimal. However, if - neither of those attributes are provided, this is also assumed to - be the image name or ID in the cloud. + .. attr:: pause + :type: bool + :default: False -**optional** + When set to True, nodepool-builder will not upload the image + to the provider. - ``config-drive`` (boolean) - Whether config drive should be used for the cloud image. Defaults to - unset which will use the cloud's default behavior. + .. attr:: config-drive + :type: bool + :default: unset - ``image-id`` (str) - If this is provided, it is used to select the image from the cloud - provider by ID, rather than name. Mutually exclusive with ``image-name``. + Whether config drive should be used for the image. Defaults to + unset which will use the cloud's default behavior. - ``image-name`` (str) - If this is provided, it is used to select the image from the cloud - provider by this name or ID. Mutually exclusive with ``image-id``. + .. attr:: meta + :type: dict - ``username`` (str) - The username that a consumer should use when connecting to the node. + Arbitrary key/value metadata to store for this server using + the Nova metadata service. A maximum of five entries is + allowed, and both keys and values must be 255 characters or + less. - ``connection-type`` (str) - The connection type that a consumer should use when connecting to the - node. For most diskimages this is not necessary. However when creating - Windows images this could be 'winrm' to enable access via ansible. + .. attr:: connection-type + :type: string - ``connection-port`` (int) - The port that a consumer should use when connecting to the - node. For most diskimages this is not necessary. This defaults to 22 - for ssh and 5986 for winrm. + The connection type that a consumer should use when connecting + to the node. For most diskimages this is not + necessary. However when creating Windows images this could be + ``winrm`` to enable access via ansible. -.. _pool_labels: + .. attr:: connection-port + :type: int + :default: 22 / 5986 -labels -~~~~~~ + The port that a consumer should use when connecting to the + node. For most diskimages this is not necessary. This defaults + to 22 for ssh and 5986 for winrm. + + .. attr:: cloud-images + :type: list + + Each entry in this section must refer to an entry in the + :attr:`labels` section. + + .. code-block:: yaml + + cloud-images: + - name: trusty-external + config-drive: False + - name: windows-external + connection-type: winrm + connection-port: 5986 + + Each entry is a dictionary with the following keys + + .. attr:: name + :type: string + :required: + + Identifier to refer this cloud-image from :attr:`labels` + section. Since this name appears elsewhere in the nodepool + configuration file, you may want to use your own descriptive + name here and use one of ``image-id`` or ``image-name`` to + specify the cloud image so that if the image name or id + changes on the cloud, the impact to your Nodepool + configuration will be minimal. However, if neither of those + attributes are provided, this is also assumed to be the image + name or ID in the cloud. + + .. attr:: config-drive + :type: bool + :default: unset + + Whether config drive should be used for the cloud + image. Defaults to unset which will use the cloud's default + behavior. + + .. attr:: image-id + :type: str + + If this is provided, it is used to select the image from the + cloud provider by ID, rather than name. Mutually exclusive + with :attr:`providers.openstack.cloud-images.image-name` + + .. attr:: image-name + :type: str + + If this is provided, it is used to select the image from the + cloud provider by this name or ID. Mutually exclusive with + :attr:`providers.openstack.cloud-images.image-id` + + .. attr:: username + :type: str + + The username that a consumer should use when connecting to the + node. + + .. attr:: connection-type + :type: str + + The connection type that a consumer should use when connecting + to the node. For most diskimages this is not + necessary. However when creating Windows images this could be + 'winrm' to enable access via ansible. + + .. attr:: connection-port + :type: int + :default: 22/ 5986 + + The port that a consumer should use when connecting to the + node. For most diskimages this is not necessary. This defaults + to 22 for ssh and 5986 for winrm. + + .. attr:: pools + :type: list + + A pool defines a group of resources from an OpenStack + provider. Each pool has a maximum number of nodes which can be + launched from it, along with a number of cloud-related attributes + used when launching nodes. + + .. code-block:: yaml + + pools: + - name: main + max-servers: 96 + availability-zones: + - az1 + networks: + - some-network-name + security-groups: + - zuul-security-group + auto-floating-ip: False + host-key-checking: True + labels: + - name: trusty + min-ram: 8192 + diskimage: trusty + console-log: True + - name: precise + min-ram: 8192 + diskimage: precise + - name: devstack-trusty + min-ram: 8192 + diskimage: devstack-trusty + + Each entry is a dictionary with the following keys + + .. attr:: name + :type: string + :required: + + Pool name + + .. attr:: max-cores + :type: int + + Maximum number of cores usable from this pool. This can be used + to limit usage of the tenant. If not defined nodepool can use + all cores up to the quota of the tenant. + + .. attr:: max-servers + :type: int + + Maximum number of servers spawnable from this pool. This can + be used to limit the number of servers. If not defined + nodepool can create as many servers the tenant allows. + + .. attr:: max-ram + :type: int + + Maximum ram usable from this pool. This can be used to limit + the amount of ram allocated by nodepool. If not defined + nodepool can use as much ram as the tenant allows. + + .. attr:: ignore-provider-quota + :type: bool + :default: False + + Ignore the provider quota for this pool. Instead, only check + against the configured max values for this pool and the + current usage based on stored data. This may be useful in + circumstances where the provider is incorrectly calculating + quota. + + .. attr:: availability-zones + :type: list + + A list of availability zones to use. + + If this setting is omitted, nodepool will fetch the list of + all availability zones from nova. To restrict nodepool to a + subset of availability zones, supply a list of availability + zone names in this setting. + + Nodepool chooses an availability zone from the list at random + when creating nodes but ensures that all nodes for a given + request are placed in the same availability zone. + + .. attr:: networks + :type: list + + Specify custom Neutron networks that get attached to each + node. Specify the name or id of the network as a string. + + .. attr:: security-groups + :type: list + + Specify custom Neutron security groups that get attached to + each node. Specify the name or id of the security_group as a + string. + + .. attr:: auto-floating-ip + :type: bool + :default: True + + Specify custom behavior of allocating floating ip for each + node. When set to False, ``nodepool-launcher`` will not apply + floating ip for nodes. When zuul instances and nodes are + deployed in the same internal private network, set the option + to False to save floating ip for cloud provider. + + .. attr:: host-key-checking + :type: bool + :default: True + + Specify custom behavior of validation of SSH host keys. When + set to False, nodepool-launcher will not ssh-keyscan nodes + after they are booted. This might be needed if + nodepool-launcher and the nodes it launches are on different + networks. The default value is True. + + .. attr:: labels + :type: list -Each entry in a pool`s `labels` section indicates that the -corresponding label is available for use in this pool. When creating -nodes for a label, the flavor-related attributes in that label's -section will be used. + Each entry in a pool`s `labels` section indicates that the + corresponding label is available for use in this pool. When + creating nodes for a label, the flavor-related attributes in that + label's section will be used. -Example configuration:: + .. code-block:: yaml - labels: - - name: precise - min-ram: 8192 - flavor-name: 'something to match' - console-log: True + labels: + - name: precise + min-ram: 8192 + flavor-name: 'something to match' + console-log: True -**required** + Each entry is a dictionary with the following keys - ``name`` - Identifier to refer this image from :ref:`labels` and :ref:`diskimages` - sections. + .. attr:: name + :type: str + :required: -**one of** + Identifier to refer this image; from :attr:`labels` and + :attr:`diskimages` sections. - ``diskimage`` - Refers to provider's diskimages, see :ref:`provider_diskimages`. + .. attr:: diskimage + :type: str + :required: - ``cloud-image`` - Refers to the name of an externally managed image in the cloud that already - exists on the provider. The value of ``cloud-image`` should match the - ``name`` of a previously configured entry from the ``cloud-images`` section - of the provider. See :ref:`provider_cloud_images`. + Refers to provider's diskimages, see + :attr:`providers.openstack.diskimages`. Mutually exclusive + with :attr:`providers.openstack.pools.labels.cloud-image` -**at least one of** + .. attr:: cloud-image + :type: str + :required: + + Refers to the name of an externally managed image in the + cloud that already exists on the provider. The value of + ``cloud-image`` should match the ``name`` of a previously + configured entry from the ``cloud-images`` section of the + provider. See :attr:`providers.openstack.cloud-images`. + Mutually exclusive with + :attr:`providers.openstack.pools.labels.diskimage` - ``flavor-name`` - Name or id of the flavor to use. If ``min-ram`` is omitted, it - must be an exact match. If ``min-ram`` is given, ``flavor-name`` will - be used to find flavor names that meet ``min-ram`` and also contain - ``flavor-name``. + .. attr:: flavor-name + :type: str - ``min-ram`` - Determine the flavor to use (e.g. ``m1.medium``, ``m1.large``, - etc). The smallest flavor that meets the ``min-ram`` requirements - will be chosen. + Name or id of the flavor to use. If + :attr:`providers.openstack.pools.labels.min-ram` is + omitted, it must be an exact match. If + :attr:`providers.openstack.pools.labels.min-ram` is given, + ``flavor-name`` will be used to find flavor names that meet + :attr:`providers.openstack.pools.labels.min-ram` and also + contain ``flavor-name``. -**optional** + One of :attr:`providers.openstack.pools.labels.min-ram` or + :attr:`providers.openstack.pools.labels.flavor-name` must + be specified. - ``boot-from-volume`` (bool) - If given, the label for use in this pool will create a volume from the - image and boot the node from it. + .. attr:: min-ram + :type: int - Default: False + Determine the flavor to use (e.g. ``m1.medium``, + ``m1.large``, etc). The smallest flavor that meets the + ``min-ram`` requirements will be chosen. - ``key-name`` - If given, is the name of a keypair that will be used when booting each - server. + One of :attr:`providers.openstack.pools.labels.min-ram` or + :attr:`providers.openstack.pools.labels.flavor-name` must + be specified. - ``console-log`` (default: False) - On the failure of the ssh ready check, download the server console log to - aid in debuging the problem. + .. attr:: boot-from-volume + :type: bool + :default: False - ``volume-size`` - When booting an image from volume, how big should the created volume be. + If given, the label for use in this pool will create a + volume from the image and boot the node from it. - In gigabytes. Default 50. + .. attr:: key-name + :type: string - ``instance-properties`` (default: None) - A dictionary of key/value properties to set when booting each - server. These properties become available via the ``meta-data`` - on the active server (e.g. within - ``config-drive:openstack/latest/meta_data.json``) + If given, is the name of a keypair that will be used when + booting each server. + .. attr:: console-log + :type: bool + :default: False -Static driver -^^^^^^^^^^^^^ + On the failure of the ssh ready check, download the server + console log to aid in debuging the problem. -The static provider driver is used to define static nodes. Nodes are also -partitioned into groups called "pools" (see :ref:`static_nodes` for details). + .. attr:: volume-size + :type: int gigabytes + :default: 50 -.. NOTE:: + When booting an image from volume, how big should the + created volume be. - Although you may define more than one pool, it is essentially useless to do - so since a node's ``name`` must be unique across all pools. + .. attr:: instance-properties + :type: dict + :default: None -Example:: + A dictionary of key/value properties to set when booting + each server. These properties become available via the + ``meta-data`` on the active server (e.g. within + ``config-drive:openstack/latest/meta_data.json``) - providers: - - name: static-rack - driver: static - pools: - - name: main - nodes: - - name: trusty.example.com - labels: trusty-static - host-key: fake-key - timeout: 13 - connection-port: 22022 - username: zuul - max-parallel-jobs: 1 -.. _static_nodes: +Static Provider +--------------- -static nodes -~~~~~~~~~~~~ +.. attr:: providers.static + :type: list -Each entry in a pool's nodes section indicates a static node and it's -corresponding label. + The static provider driver is used to define static nodes. -**required** + Note for documentation purposes the option names are prefixed + ``providers.static`` to disambiguate from other drivers, but the + ``openstack`` is not required in the configuration (e.g. below + ``providers.static.pools`` refers to the ``pools`` key in the + ``providers`` section when the ``static`` driver is selected). - ``name`` - The hostname or ip address of the static node. This must be unique - across all nodes defined within the configuration file. + .. code-block:: yaml - ``labels`` (list) - The list of labels associated with the node. + providers: + - name: static-rack + driver: static + pools: + - name: main + nodes: + - name: trusty.example.com + labels: trusty-static + host-key: fake-key + timeout: 13 + connection-port: 22022 + username: zuul + max-parallel-jobs: 1 -**optional** + .. attr:: pools + :type: list - ``username`` - The username nodepool will use to validate it can connect to the node. - Default to *zuul* + Each entry in a pool's nodes section indicates a static node and + it's corresponding label. - ``timeout`` - The timeout in second before the ssh ping is considered failed. - Default to *5* seconds + .. note:: - ``host-key`` - The ssh host key of the node. + Although you may define more than one pool, it is essentially + useless to do so since a node's ``name`` must be unique + across all pools. - ``connection-type`` (string) - The connection type that a consumer should use when connecting to the - node. Should be set to either 'ssh' or 'winrm'. Defaults to 'ssh'. + Each entry is a dictionary with entries as follows - ``connection-port`` (int) - The port that a consumer should use when connecting to the node. - For most nodes this is not necessary. This defaults to 22 when - ``connection-type`` is 'ssh' and 5986 when it is 'winrm'. + .. attr:: name + :type: str + :required: - ``max-parallel-jobs`` - The number of jobs that can run in parallel on this node, default to *1*. + The hostname or ip address of the static node. This must be + unique across all nodes defined within the configuration file. + + .. attr:: labels + :type: list + :required: + + The list of labels associated with the node. + + .. attr:: username + :type: str + :default: zuul + + The username nodepool will use to validate it can connect to the + node. + + .. attr:: timeout + :type: int + :default: 5 + + The timeout in second before the ssh ping is considered failed. + + .. attr:: host-key + :type: str + + The ssh host key of the node. + + .. attr:: connection-type + :type: string + + The connection type that a consumer should use when connecting + to the node. + + .. value:: winrm + + .. value:: ssh + + .. attr:: connection-port + :type: int + :default: 22 / 5986 + + The port that a consumer should use when connecting to the node. + For most nodes this is not necessary. This defaults to 22 when + ``connection-type`` is 'ssh' and 5986 when it is 'winrm'. + + .. attr:: max-parallel-jobs + :type: int + :default: 1 + + The number of jobs that can run in parallel on this node. diff --git a/doc/source/operation.rst b/doc/source/operation.rst index 41950d44e..d14083321 100644 --- a/doc/source/operation.rst +++ b/doc/source/operation.rst @@ -222,7 +222,7 @@ is the same. Web interface ------------- -If configured (see :ref:`webapp-conf`), a ``nodepool-launcher`` +If configured (see :attr:`webapp-conf`), a ``nodepool-launcher`` instance can provide a range of end-points that can provide information in text and ``json`` format. Note if there are multiple launchers, all will provide the same information.