da047b3db7
The commit adds contributing page for easier orientation in our projects. Change-Id: Iff202064e3fe1b4651f73e22ec0733f191c3ecaf
69 lines
3.4 KiB
ReStructuredText
Executable File
69 lines
3.4 KiB
ReStructuredText
Executable File
Interoperability Guideline Testing
|
|
==================================
|
|
|
|
Testing against 2017.01 until present Capabilities
|
|
--------------------------------------------------
|
|
|
|
Tempest can be run standalone, or under a test runner such as refstack-client.
|
|
We recommend running under refstack-client.
|
|
|
|
The test names of the capabilities are derived from a recent release of
|
|
Tempest, from the time of capability approval. Keep in mind that Tempest
|
|
is under active development, and tests may move. If you're not seeing
|
|
full coverage, please consider reverting back to a Tempest SHA that more
|
|
closely matches the capability release date. The git SHA of Tempest that was
|
|
known to be working at the time the Guideline was approved is listed in the
|
|
Guideline JSON document itself (just search for "git-sha"). Please contact
|
|
RefStack team at <openstack-discuss@lists.openstack.org> for assistance if
|
|
needed. Use [interop] and/or [refstack] tags in the subject of the email.
|
|
|
|
It's important to run a recent version of Tempest, as some bugs might have
|
|
been fixed just recently. refstack-client uses Tempest from a specific commit
|
|
(see setup_env script or README), which is intentionally older than master to
|
|
avoid compatibility issues with older stable releases of OpenStack.
|
|
|
|
Network provisioning have been fixed. Some tests are still flagged due to
|
|
outstanding bugs in the Tempest library, particularly tests that require SSH.
|
|
We are working on correcting these bugs upstream. Please note that
|
|
although some tests are flagged because of bugs, there is still an
|
|
expectation that the capabilities covered by the tests are available.
|
|
|
|
In addition to testing required capabilities, we are also interested
|
|
in collecting data on which API tests are being passed by production clouds.
|
|
This information will be very useful in determining which capabilities will be
|
|
used to define future releases. For that reason, we ask that you run all
|
|
tests rather than just the required subset when submitting results to
|
|
the OpenStack Foundation.
|
|
|
|
It is important to note that you MUST NOT modify the Tempest tests in any
|
|
way. Modifying the tests means that Capability being tested is validated
|
|
in a different way on your cloud than it is on other clouds, which voids
|
|
any guarantee of interoperability. If you're having problems passing
|
|
all required tests, please contact RefStack team at
|
|
<openstack-discuss@lists.openstack.org> for assistance or consider filing a
|
|
request to have the tests flagged.
|
|
|
|
Please refer to `HACKING <../hacking.rst>`_ for information on valid
|
|
reasons to flag a test and how to file a flag request. Results from
|
|
modified tests cannot be accepted as valid for trademark licensing
|
|
purposes.
|
|
|
|
Recommended Test Procedure
|
|
##########################
|
|
|
|
Follow steps mentioned in the refstack-client's documentation:
|
|
https://docs.opendev.org/openinfra/refstack-client/latest/readme.html
|
|
|
|
* Follow 'Environment setup' section to clone and install refstack-client
|
|
|
|
* Follow steps in 'Usage' section in order to run tests against a desired
|
|
OpenStack Marketing program.
|
|
|
|
* Review the test results, and when you're satisfied, upload it to RefStack
|
|
server (steps in the 'Usage' section) then send them to interop@openstack.org
|
|
|
|
* Every effort should be made to pass all of the required tests, but you
|
|
will want to compare any lists of failed tests to the list of flagged tests.
|
|
The refstack server will automatically grade tests results, taking
|
|
into account flagged tests.
|