Fix issues with novajoin functional test

1. Set parameters to set flavor and image for baremetal
2. add and reorganize timeouts/retry code
3. fix changed call to verify compact services
4. fix user to ssh into the image

Change-Id: I80cc2e0cd8abc5d860b22349276ef4a40282e4f2

doc/source/readme.rst

@@ -35,42 +35,45 @@ This section describes how to set up these tests on an RDO system.
 
 To set up tempest on the RDO system, use dnf to install tempest:
 
+::
+
     $ sudo dnf install -y openstack-tempest python-devel
 
 Also install the novajoin-tempest-plugin.
 
+::
+
     $ sudo dnf install python-devel gcc
-
     $ git clone https://github.com/vakwetu/novajoin_tempest_plugin.git
-
     $ cd novajoin_tempest_plugin
-
     $ sudo python setup.py install
 
 Prepare a working directory to run the tempest tests.
 
+::
+
     $ cd ~
-
     $ tempest init tempest_run
-
     $ cd tempest_run
 
 The discovery command detailed below only appears to work for v2 of the
 keystone API.  Copy the stackrc file and convert the OS_AUTH_URL to a v2
 version.
 
-    $ cp /home/stack/stackrc .
+::
 
+    $ cp /home/stack/stackrc .
     $ vi stackrc (convert to v2 by changing as follows:
       OS_AUTH_URL=https://192.168.24.2:13000/v2.0
       OS_IDENTITY_API_VERSION='2'
-
     $ source ./stackrc
 
 There is currently a bug in the code used to discover and generate
 tempest config files if the volume service (cinder) is not installed.
 To work around this, comment out the line:
 
+::
+
     #check_volume_backup_service(clients.volume_service, conf, services)
 
 at around line 203 in the file
@@ -79,6 +82,8 @@ at around line 203 in the file
 Run the discovery command to generate the required tempest config file
 under ./etc/tempest.
 
+::
+
     $ discover-tempest-config --verbose \
       --image http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img \
       --out etc/tempest.conf --debug --create identity.uri $OS_AUTH_URL \
@@ -89,20 +94,21 @@ The tests need credentials to connect to the IPA server.  In our tests, we
 have used the keytab for the novajoin user.  Copy this keytab and set the
 appropriate ownership and permissions for the user executing the plugin.
 
-    $ sudo cp /etc/novajoin/krb5.keytab /home/stack/krb5.keytab
+::
 
+    $ sudo cp /etc/novajoin/krb5.keytab /home/stack/krb5.keytab
     $ sudo chown stack: /home/stack/krb5.keytab
 
 Add the following directives to the [validation] stanza in the generated
 tempest configuration file in ./etc/tempest.conf to configure the ssh
 client.
 
+::
+
     $ vi tempest/tempest.conf
 
        [validation]
-
        connect_method = fixed
-
        network_for_ssh = ctlplane
 
 
@@ -116,6 +122,9 @@ Some of these are described below.  All of these config directives would be
 specified under a [novajoin] stanza in ./etc/tempest.conf.
 
 NovajoinGroup = [
+    cfg.StrOpt('flavor_tag',
+               default='vm',
+               help='Flavor tag to use in novajoin enrollment tests'),
     cfg.StrOpt('keytab',
                default='/home/stack/novajoin.keytab',
                help='Keytab to connect to IPA as the novajoin user'),
@@ -133,6 +142,43 @@ NovajoinGroup = [
                help='Undercloud short host name'
                )
 
+Tempest Configuration for TripleO
+---------------------------------
+
+The tempest tests for a tripleo environment are typically run on the
+undercloud and use novajoin in the undercloud to generate baremetal nodes
+(like the controllers and computes).
+
+In this case, we have made the following tempest configuration work:
+
+::
+
+    [novajoin]
+    tripleo_controllers = controller-0,controller-1,controller-2
+    tripleo_computes = compute-0,compute-1
+    tripleo_undercloud = undercloud-0
+    flavor_tag = baremetal
+
+    [validation]
+    connect_method = fixed
+    network_for_ssh = ctlplane
+    image_ssh_user = fedora
+
+    [image]
+    image_path = http://foo.example.com/path-to-image.qcow2
+    region = regionOne
+    http_image = http://foo.example.com/path-to-image.qcow2
+
+Some things to note:
+
+- The flavor_tag is set to either 'vm' or 'baremetal'.  In this case,
+  the novajoin enrollment test will try to create test servers on baremetal
+  nodes.  This nodes should already be provisioned by ironic.
+- The image_path should point to the image to use when creating baremetal
+  servers.  The image_ssh_user needs to correspond to the correct default
+  user.  For Fedora images, for instance, this is 'fedora'.  Ideally, the
+  image should already have the ipa-client package installed.
+
 
 Running the tests
 -----------------
@@ -143,6 +189,8 @@ the requested services and hosts are created.  The test also confirms that
 the instances and services are appropriately deleted when the instance
 is deleted.  To run these tests,
 
+::
+
     $ tempest run --regex test_novajoin_enrollment
 
 The tests in test_tripleo_deployment should be run on the undercloud in a
@@ -152,6 +200,8 @@ and services have been created in the IPA server.  In addition, it
 confirms that the certificates requested by Heat are tracked by certmonger
 on the overcloud nodes.
 
+::
+
     $ tempest run --regex test_tripleo_deployment
 
 The tests in test_tripleo_tls should be run on the undercloud in a TLS enabled
@@ -159,4 +209,6 @@ tripleo deployment.  These tests verify that all services have TLS connections
 on all external and internal connections using the openssl client to attempt
 TLS connections.
 
+::
+
     $ tempest run --regex test_tripleo_tls

novajoin_tempest_plugin/config.py

@@ -25,6 +25,9 @@ novajoin_group = cfg.OptGroup(
     title="Novajoin test plugin settings")
 
 NovajoinGroup = [
+    cfg.StrOpt('flavor_tag',
+               default='vm',
+               help='Flavor tag to use in novajoin enrollment tests'),
     cfg.StrOpt('keytab',
                default='/home/stack/novajoin.keytab',
                help='Keytab to connect to IPA as the novajoin user'),

novajoin_tempest_plugin/tests/scenario/novajoin_manager.py

@@ -12,6 +12,7 @@
 #    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
 #    License for the specific language governing permissions and limitations
 #    under the License.
+import functools
 import json
 import six
 import subprocess
@@ -55,24 +56,31 @@ class NovajoinScenarioTest(manager.ScenarioTest):
         super(NovajoinScenarioTest, cls).setup_clients()
         cls.ipa_client = ipa_client.IPAClient()
 
+    def retry_with_timeout(func):
+        @functools.wraps(func)
+        def wrapper_retry_with_timeout(*args, **kwargs):
+            start = int(time.time())
+            timeout = 300
+            result = func(*args, **kwargs)
+            while not result and (int(time.time()) - start < timeout):
+                time.sleep(30)
+                result = func(*args, **kwargs)
+            assert result
+        return wrapper_retry_with_timeout
+
+    @retry_with_timeout
     def verify_host_registered_with_ipa(self, host, add_domain=True):
         if add_domain:
             host = self.add_domain_to_host(host)
         result = self.ipa_client.find_host(host)
-        self.assertTrue(result['count'] > 0)
+        return result['count'] > 0
 
+    @retry_with_timeout
     def verify_host_not_registered_with_ipa(self, host, add_domain=True):
         if add_domain:
             host = self.add_domain_to_host(host)
         result = self.ipa_client.find_host(host)
-        start = int(time.time())
+        return result['count'] == 0
-        host_count = result['count']
-        timeout = 300
-        while (host_count > 0) and (int(time.time()) - start < timeout):
-            time.sleep(30)
-            result = self.ipa_client.find_host(host)
-            host_count = result['count']
-        self.assertFalse(result['count'] > 0)
 
     def add_domain_to_host(self, host):
         host = '{host}.{domain}'.format(
@@ -80,34 +88,30 @@ class NovajoinScenarioTest(manager.ScenarioTest):
             domain=self.ipa_client.domain)
         return host
 
+    @retry_with_timeout
     def verify_host_has_keytab(self, host, add_domain=True):
         if add_domain:
             host = self.add_domain_to_host(host)
         result = self.ipa_client.show_host(host)['result']
-        start = int(time.time())
+        return result['has_keytab']
-        keytab_status = result['has_keytab']
-        timeout = 300
-        while not keytab_status and (int(time.time()) - start < timeout):
-            time.sleep(30)
-            result = self.ipa_client.show_host(host)['result']
-            keytab_status = result['has_keytab']
-        self.assertTrue(keytab_status)
 
+    @retry_with_timeout
     def verify_service_created(self, service, host):
         service_principal = self.get_service_principal(host, service)
         result = self.ipa_client.find_service(service_principal)
-        self.assertTrue(result['count'] > 0)
+        return result['count'] > 0
 
+    @retry_with_timeout
     def verify_service_managed_by_host(self, service, host):
         service_principal = self.get_service_principal(host, service)
-        result = self.ipa_client.service_managed_by_host(service_principal,
+        return self.ipa_client.service_managed_by_host(service_principal,
-                                                         host)
+                                                       host)
-        self.assertTrue(result)
 
+    @retry_with_timeout
     def verify_service_deleted(self, service, host):
         service_principal = self.get_service_principal(host, service)
         result = self.ipa_client.find_service(service_principal)
-        self.assertFalse(result['count'] > 0)
+        return result['count'] == 0
 
     def verify_compact_services_deleted(self, services, host):
         for (service, networks) in services.items():
@@ -115,18 +119,13 @@ class NovajoinScenarioTest(manager.ScenarioTest):
                 subhost = '{host}.{network}.{domain}'.format(
                     host=host, network=network, domain=self.ipa_client.domain
                 )
-                service_principal = self.get_service_principal(
+                self.verify_service_deleted(service, subhost)
-                    subhost, service)
-                result = self.ipa_client.find_service(service_principal)
-                self.assertFalse(result['count'] > 0)
 
     def verify_managed_services_deleted(self, services):
         for principal in services:
             service = principal.split('/', 1)[0]
             host = principal.split('/', 1)[1]
-            service_principal = self.get_service_principal(host, service)
+            self.verify_service_deleted(service, host)
-            result = self.ipa_client.find_service(service_principal)
-            self.assertFalse(result['count'] > 0)
 
     def get_service_cert(self, service, host):
         service_principal = self.get_service_principal(host, service)
@@ -163,10 +162,11 @@ class NovajoinScenarioTest(manager.ScenarioTest):
         result = self.execute_on_controller(user, hostip, cmd)
         self.assertTrue('track: yes' in result)
 
+    @retry_with_timeout
     def verify_cert_revoked(self, serial):
         # verify that the given certificate has been revoked
         result = self.ipa_client.show_cert(serial)['result']
-        self.assertTrue(result['revoked'])
+        return result['revoked']
 
     def get_compact_services(self, metadata):
         # compact key-per-service

novajoin_tempest_plugin/tests/scenario/test_novajoin_enrollment.py

@@ -23,9 +23,25 @@ import ast
 
 CONF = config.CONF
 LOG = logging.getLogger(__name__)
-USER = 'cloud-user'
+USER = CONF.validation.image_ssh_user
 NETWORK = 'ctlplane'
 
+FLAVORS = {
+    'baremetal': {'ram': 4096,
+                  'vcpus': 3,
+                  'disk': 30,
+                  'specs': {
+                      "capabilities:boot_option": "local",
+                      "capabilities:profile": "ironic",
+                      "resources:CUSTOM_IRONIC": '1'}},
+    'vm': {'ram': 4096,
+           'vcpus': 1,
+           'disk': 40,
+           'specs': {
+               "capabilities:boot_option": "local",
+               "capabilities:profile": "compute"}}
+}
+
 
 class ServerTest(novajoin_manager.NovajoinScenarioTest):
 
@@ -44,20 +60,16 @@ class ServerTest(novajoin_manager.NovajoinScenarioTest):
     def resource_setup(cls):
         super(ServerTest, cls).resource_setup()
 
-    def _create_flavor(self, flavor_name):
+    def _create_flavor(self, flavor_name, tag):
-        specs = {"capabilities:boot_option": "local",
-                 "capabilities:profile": "compute"}
         flv_id = data_utils.rand_int_id(start=1000)
-        ram = 4096
+        flavor = FLAVORS[tag]
-        vcpus = 1
-        disk = 40
         self.flavors_client.create_flavor(name=flavor_name,
-                                          ram=ram,
+                                          ram=flavor['ram'],
-                                          vcpus=vcpus,
+                                          vcpus=flavor['vcpus'],
-                                          disk=disk,
+                                          disk=flavor['disk'],
                                           id=flv_id)['flavor']
         self.flavors_client.set_flavor_extra_spec(flv_id,
-                                                  **specs)
+                                                  **flavor['specs'])
         return flv_id
 
     def _create_image(self, name, properties={}):
@@ -71,11 +83,13 @@ class ServerTest(novajoin_manager.NovajoinScenarioTest):
 
     def _verify_host_and_services_are_enrolled(
             self, server_name, server_id, keypair):
+        server_details = self.servers_client.show_server(server_id)['server']
+        ip = self.get_server_ip(server_details)
+
         self.verify_host_registered_with_ipa(server_name)
         self.verify_host_has_keytab(server_name)
 
         # Verify compact services are created
-
         metadata = self.servers_client.list_server_metadata(server_id
                                                             )['metadata']
         services = metadata['compact_services']
@@ -83,6 +97,7 @@ class ServerTest(novajoin_manager.NovajoinScenarioTest):
         self.verify_compact_services(
             services=self.compact_services,
             host=server_name,
+            host_ip=ip
         )
 
         # Verify managed services are created
@@ -91,9 +106,6 @@ class ServerTest(novajoin_manager.NovajoinScenarioTest):
         self.verify_managed_services(self.managed_services)
 
         # Verify instance created above is ipaclient
-        server_details = self.servers_client.show_server(server_id
-                                                         )['server']
-        ip = self.get_server_ip(server_details)
         self.verify_host_is_ipaclient(ip, USER, keypair)
 
     def _verify_host_and_services_are_not_enrolled(self, server_name):
@@ -109,7 +121,9 @@ class ServerTest(novajoin_manager.NovajoinScenarioTest):
         networks = self.networks_client.list_networks(name=NETWORK)
         net_id = networks['networks'][0]['id']
         flavor_name = data_utils.rand_name('flv_metadata_in_instance')
-        flavor_id = self._create_flavor(flavor_name)
+        flavor_id = self._create_flavor(flavor_name,
+                                        CONF.novajoin.flavor_tag)
+
         image_name = data_utils.rand_name('img_metadata_in_instance')
         image_id = self._create_image(image_name)
         keypair = self.create_keypair()
@@ -136,14 +150,12 @@ class ServerTest(novajoin_manager.NovajoinScenarioTest):
         networks = self.networks_client.list_networks(name=NETWORK)
         net_id = networks['networks'][0]['id']
         flavor_name = data_utils.rand_name('flv_metadata_in_image')
-        flavor_id = self._create_flavor(flavor_name)
+        flavor_id = self._create_flavor(flavor_name,
-        image_name = data_utils.rand_name('metadata_in_image')
+                                        CONF.novajoin.flavor_tag)
+        image_name = data_utils.rand_name('img_metadata_in_image')
         properties = {"ipa_enroll": "True"}
         image_id = self._create_image(image_name, properties)
         keypair = self.create_keypair()
-        f = open('/tmp/priv.key', 'w')
-        f.write(keypair['private_key'])
-        f.close()
         instance_name = data_utils.rand_name("novajoin")
         metadata = {"compact_services":
                     "{\"HTTP\": [\"ctlplane\", \"internalapi\"]}",