doc: Update testing documentation

The information on how to run functional tests was badly out of date.
Correct it.

Change-Id: I7e77e2a96ab15f7dede01344cb03b0e61c8589bd
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>

create_yaml.sh

@@ -1,31 +0,0 @@
-#!/bin/bash
-#
-# NOTE(thowe): There are some issues with OCC envvars that force us to do
-# this for now.
-#
-mkdir -p ~/.config/openstack/
-FILE=~/.config/openstack/clouds.yaml
-export OS_IDENTITY_API_VERSION=3 # force v3 identity
-echo 'clouds:' >$FILE
-echo '  test_cloud:' >>$FILE
-env | grep OS_ | tr '=' ' ' | while read k v
-do
-  k=$(echo $k | sed -e 's/OS_//')
-  k=$(echo $k | tr '[A-Z]' '[a-z]')
-  case "$k" in
-  region_name|*_api_version)
-    echo "    $k: $v" >>$FILE
-  esac
-done
-echo "    auth:" >>$FILE
-env | grep OS_ | tr '=' ' ' | while read k v
-do
-  k=$(echo $k | sed -e 's/OS_//')
-  k=$(echo $k | tr '[A-Z]' '[a-z]')
-  case "$k" in
-  region_name|*_api_version)
-    ;;
-  *)
-    echo "      $k: $v" >>$FILE
-  esac
-done

doc/source/contributor/clouds.yaml

@@ -1,19 +1,37 @@
 clouds:
-  test_cloud:
+  devstack:
-    region_name: RegionOne
     auth:
-      auth_url: http://xxx.xxx.xxx.xxx:5000/v2.0/
+      auth_url: http://xxx.xxx.xxx.xxx/identity
-      username: demo
+      password: password
-      password: secrete
+      project_domain_id: default
       project_name: demo
-  rackspace:
+      user_domain_id: default
-    cloud: rackspace
+      username: demo
+    identity_api_version: '3'
+    region_name: RegionOne
+    volume_api_version: '3'
+  devstack-admin:
     auth:
-      username: joe
+      auth_url: http://xxx.xxx.xxx.xxx/identity
-      password: joes-password
+      password: password
-      project_name: 123123
+      project_domain_id: default
-    region_name: IAD
+      project_name: admin
+      user_domain_id: default
+      username: admin
+    identity_api_version: '3'
+    region_name: RegionOne
+    volume_api_version: '3'
+  devstack-alt:
+    auth:
+      auth_url: http://xxx.xxx.xxx.xxx/identity
+      password: password
+      project_domain_id: default
+      project_name: alt_demo
+      user_domain_id: default
+      username: alt_demo
+    identity_api_version: '3'
+    region_name: RegionOne
+    volume_api_version: '3'
 example:
-  image_name: fedora-20.x86_64
+  image_name: cirros-0.5.2-x86_64-disk
   flavor_name: m1.small
-  network_name: private

doc/source/contributor/testing.rst

@@ -3,25 +3,27 @@ Testing
 
 The tests are run with `tox <https://tox.readthedocs.org/en/latest/>`_ and
 configured in ``tox.ini``. The test results are tracked by
-`testr <https://testrepository.readthedocs.org/en/latest/>`_ and configured
+`stestr <https://stestr.readthedocs.io/en/latest/>`_ and configured
-in ``.testr.conf``.
+in ``.stestr.conf`` and via command line options passed to the ``stestr``
+executable when it's called by ``tox``.
+
 
 Unit Tests
 ----------
 
-Run
+Running tests
-***
+~~~~~~~~~~~~~
 
 In order to run the entire unit test suite, simply run the ``tox`` command
 inside of your source checkout. This will attempt to run every test command
-listed inside of ``tox.ini``, which includes Python 3.8, and a PEP 8 check.
+listed inside of ``tox.ini``, which includes Python 3.x, and a PEP 8 check.
 You should run the full test suite on all versions before
 submitting changes for review in order to avoid unexpected failures in the
 continuous integration system.::
 
-   (sdk3)$ tox
+   $ tox
    ...
-   py38: commands succeeded
+   py3: commands succeeded
    pep8: commands succeeded
    congratulations :)
 
@@ -29,8 +31,13 @@ During development, it may be more convenient to run a subset of the tests
 to keep test time to a minimum. You can choose to run the tests only on one
 version. A step further is to run only the tests you are working on.::
 
-   (sdk3)$ tox -e py38                # Run run the tests on Python 3.8
+   # Run run the tests on Python 3.9
-   (sdk3)$ tox -e py38 TestContainer  # Run only the TestContainer tests on 3.8
+   $ tox -e py39
+   # Run only the compute unit tests on Python 3.9
+   $ tox -e py39 openstack.tests.unit.compute
+   # Run only the tests in a specific file on Python 3.9
+   $ tox -e py39 -- -n openstack/tests/unit/compute/test_version.py
+
 
 Functional Tests
 ----------------
@@ -41,17 +48,43 @@ public clouds but first and foremost they must be run against OpenStack. In
 practice, this means that the tests should initially be run against a stable
 branch of `DevStack <https://docs.openstack.org/devstack/latest/>`_.
 
-os-client-config
+Configuration
-****************
+~~~~~~~~~~~~~
 
-To connect the functional tests to an OpenStack cloud we use
+To connect the functional tests to an OpenStack cloud we require a
-`os-client-config <http://opendev.org/openstack/os-client-config>`_.
+``clouds.yaml`` file, as discussed in :doc:`/user/config/configuration`.
-To setup os-client-config create a ``clouds.yaml`` file in the root of your
+You can place this ``clouds.yaml`` file in the root of your source checkout or
-source checkout.
+in one of the other standard locations, ``$HOME/.config/openstack`` or
+``/etc/openstack``.
+
+There must be at least three clouds configured, or rather three accounts
+configured for the one cloud. These accounts are:
+
+- An admin account, which defaults to ``devstack-admin`` but is configurable
+  via the ``OPENSTACKSDK_OPERATOR_CLOUD`` environment variable,
+- A user account, which defaults to ``devstack`` but is configurable
+  via the ``OPENSTACKSDK_DEMO_CLOUD`` environment variable, and
+- An alternate user account, which defaults to ``devstack-demo`` but is
+  configurable via the ``OPENSTACKSDK_DEMO_CLOUD_ALT`` environment variable
+
+In addition, you must indicate the names of the flavor and image that should be
+used for tests. These can be configured via ``functional.flavor_name`` and
+``functional.image_name`` settings in the ``clouds.yaml`` file.
+
+Finally, you can configure the timeout for tests using the
+``OPENSTACKSDK_FUNC_TEST_TIMEOUT`` environment variable (defaults to 300
+seconds). Some test modules take specific timeout values. For example, all
+tests in ``openstack.tests.functional.compute`` will check for the
+``OPENSTACKSDK_FUNC_TEST_TIMEOUT_COMPUTE`` environment variable before checking
+for ``OPENSTACKSDK_FUNC_TEST_TIMEOUT``.
+
+.. note::
+
+    Recent versions of DevStack will configure a suitable ``clouds.yaml`` file
+    for you, which will be placed at ``/etc/openstack/clouds.yaml``.
 
 This is an example of a minimal configuration for a ``clouds.yaml`` that
-connects the functional tests to a DevStack instance. Note that one cloud
+connects the functional tests to a DevStack instance.
-under ``clouds`` must be named ``test_cloud``.
 
 .. literalinclude:: clouds.yaml
    :language: yaml
@@ -59,29 +92,16 @@ under ``clouds`` must be named ``test_cloud``.
 Replace ``xxx.xxx.xxx.xxx`` with the IP address or FQDN of your DevStack
 instance.
 
-You can also create a ``~/.config/openstack/clouds.yaml`` file for your
+Running tests
-DevStack cloud environment using the following commands. Replace
+~~~~~~~~~~~~~
-``DEVSTACK_SOURCE`` with your DevStack source checkout.::
 
-   (sdk3)$ source DEVSTACK_SOURCE/accrc/admin/admin
+Functional tests are also run against multiple Python versions. In order to run
-   (sdk3)$ ./create_yaml.sh
+the entire functional test suite against the default Python 3 version in your
+environment, run the ``tox -e functional`` command inside of your source
+checkout. This will attempt to run every tests in the
+``openstack/tests/functional`` directory. For example::
 
-Run
+   $ tox -e functional
-***
-
-Functional tests are run against both Python 2 and 3. In order to run the
-entire functional test suite, run the ``tox -e functional`` and
-``tox -e functional3`` command inside of your source checkout. This will
-attempt to run every test command under ``/openstack/tests/functional/``
-in the source tree. You should run the full functional test suite before
-submitting changes for review in order to avoid unexpected failures in
-the continuous integration system.::
-
-   (sdk3)$ tox -e functional
    ...
    functional: commands succeeded
    congratulations :)
-   (sdk3)$ tox -e functional3
-   ...
-   functional3: commands succeeded
-   congratulations :)

tox.ini

@@ -25,7 +25,7 @@ commands =
     stestr run {posargs}
     stestr slowest
 
-[testenv:functional]
+[testenv:functional{,-py36,-py37,-py38,-py39}]
 # Some jobs (especially heat) takes longer, therefore increase default timeout
 # This timeout should not be smaller, than the longest individual timeout
 setenv =