Add initial files for jenkins, zuul, nodepool

Depends on nodepool https://review.openstack.org/#/c/189762/

README.md

@@ -0,0 +1,53 @@
+# Example project-config repository
+
+This is an example project-config repository for use as a
+starting point to setup a 3rd party CI account.
+
+## Steps to begin customization
+
+The project-config repository is intended to contain custom configurations
+needed by each CI system.
+
+## Customize Zuul
+
+The zuul layout configuration is located in `zuul/layout.yaml`. You can find
+the full configuration details in the [Zuul manual](http://docs.openstack.org/infra/zuul/).
+
+1. Change 'myvendor' in the 'recheck' command to your CI's name.
+
+2. Configure the e-mail addresses for merge-failures and job notification.
+
+3. By default, the project zuul triggers on is `openstack-dev/ci-sandbox`.
+   After testing your CI system update this section to include other projects.
+   You are encouraged to use the 'silent' pipeline until your jobs are stable.
+
+## Customize Nodepool
+
+The nodepool configuration is located in `nodepool/nodepool.yaml`. You can
+find the full configuration details in the [Nodepool manual](http://docs.openstack.org/infra/nodepool/).
+There are a few configuration that need to be updated.
+
+1. There are some user names and passwords that need to be configured.
+
+2. Select a 'random time' for your nodepool images to be built in the
+   `image-update` property. By having 3rd party systems use different
+   times will help reduce the spike load on OpenStack's Git servers.
+
+3. Setup an intial set of nodepool scripts and elements. Start by cloning
+   OpenStack's [project-config](https://git.openstack.org/cgit/openstack-infra/project-config/)
+   and copy the contents of that repo's `nodepool/elements` to your repo's
+   `nodepool/elements`. Optionally do the same for the `nodepool/scripts`
+   folder. You may have to change these elements to work in your environment.
+   If so, see this [README](http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/README.rst)
+   for help.
+
+## Customize Jenkins Jobs
+
+Adjust the jenkins jobs in `jenkins/jobs/` to your needs. You can find the full configuration details in the
+[Jenkins Job Builder manual](http://docs.openstack.org/infra/jenkins-job-builder/)
+
+1. Change the value of the `$PUBLISH_HOST` to the host (without https:// prefix) you will publish
+   job artifacts to. This is also known as the Log server. You can set one up using [this script]
+
+
+

jenkins/jobs/README.md

@@ -0,0 +1,2 @@
+You can find many more examples of jobs in
+[openstack-infra/project-config](http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs)

jenkins/jobs/defaults.yaml

@@ -0,0 +1,20 @@
+- defaults:
+    name: global
+    description: |
+      <p><b>This job is managed by puppet and will be overwritten.</b></p>
+
+      <p><b>Do not edit this job through the web</b></p>
+    project-type: freestyle
+    concurrent: true
+
+    wrappers:
+      - timeout:
+          timeout: 30
+          fail: true
+      - timestamps
+
+    logrotate:
+      daysToKeep: 7 
+      numToKeep: -1
+      artifactDaysToKeep: -1
+      artifactNumToKeep: -1

jenkins/jobs/dsvm-cinder-driver.yaml

@@ -0,0 +1,75 @@
+- job-template:
+    name: 'dsvm-tempest-my-cinder-driver'
+    node: '{node}'
+
+    wrappers:
+      - timeout:
+          timeout: 125
+          fail: true
+      - timestamps
+
+    builders:
+      - link-logs
+      - net-info
+      - devstack-checkout
+      - shell: |
+          #!/bin/bash -xe
+
+          function pre_test_hook {{
+              echo "Install thirdparty client libraries"
+              #TODO: update your client here if needed, otherwise delete
+              sudo -H pip install mydriverclient
+              echo "Configure the local.conf file to properly setup hp lefthand driver in cinder.conf"
+              cat <<EOF >$BASE/new/devstack/local.conf
+
+          [[post-config|\$CINDER_CONF]]
+          [DEFAULT]
+          enabled_backends=mybackend
+          default_volume_type=myvolumetype
+
+          [mybackend]
+          # add you driver configuration here
+          volume_driver=cinder.volume.drivers.MyDriver
+          volume_backend_name=mybackend
+
+          # Use post-extra because the tempest configuration file is
+          # overwritten with the .sample after post-config.
+          [[post-extra|\$TEMPEST_CONFIG]]
+          [volume]
+          storage_protocol=iSCSI
+          vendor_name=MyVendor
+          EOF
+
+              echo "Configure localrc file to properly setup CINDER_ENABLED_BACKENDS"
+              cat <<EOF >>$BASE/new/devstack/localrc
+          CINDER_ENABLED_BACKENDS=mybackend:myvolumetype
+          EOF
+
+          }}
+
+          export -f pre_test_hook
+
+          # To keep our CINDER_ENABLED_BACKENDS configuration in localrc
+          export KEEP_LOCALRC=true
+
+          export PYTHONUNBUFFERED=true
+          export DEVSTACK_GATE_TIMEOUT=120
+          export DEVSTACK_GATE_TEMPEST=1
+          export DEVSTACK_GATE_TEMPEST_REGEX="volume"
+
+          # Let's use the http protocol instead of git protocol
+          export GIT_BASE=https://git.openstack.org
+
+          if [ -z $ZUUL_PROJECT ]; then
+              export ZUUL_PROJECT=openstack-dev/ci-sandbox
+          fi
+          if [ -z $ZUUL_BRANCH ]; then
+              export ZUUL_BRANCH=master
+          fi
+
+          cp devstack-gate/devstack-vm-gate-wrap.sh ./safe-devstack-vm-gate-wrap.sh
+          ./safe-devstack-vm-gate-wrap.sh
+
+    publishers:
+      - devstack-logs
+      - console-log

jenkins/jobs/examples.yaml

@@ -0,0 +1,50 @@
+- job-template:
+    name: 'noop-check-communication'
+    node: '{node}'
+
+    builders:
+      - shell: |
+          #!/bin/bash -xe
+          echo "Hello world, this is the {vendor} Testing System"
+      - link-logs
+
+    publishers:
+      - devstack-logs
+      - console-log
+
+
+- job-template:
+    name: 'dsvm-tempest-full'
+    node: '{node}'
+
+    wrappers:
+      - timeout:
+          timeout: 185  # Timeout in *minutes*
+          fail: true  # A job run that exceeds the timeout will cause a failure
+      - timestamps
+
+    builders:
+      - net-info
+#      - devstack-checkout
+      - devstack-checkout-http
+      - shell: |
+          #!/bin/bash -xe
+          if [ -z $ZUUL_PROJECT ]; then
+              export ZUUL_PROJECT=openstack-dev/sandbox
+          fi
+          if [ -z $ZUUL_BRANCH ]; then
+              export ZUUL_BRANCH=master
+          fi
+          export PYTHONUNBUFFERED=true
+          export DEVSTACK_GATE_TIMEOUT=180
+          export DEVSTACK_GATE_TEMPEST=1
+
+          export DEVSTACK_GATE_TEMPEST_REGEX="volume.api"
+
+          cp devstack-gate/devstack-vm-gate-wrap.sh ./safe-devstack-vm-gate-wrap.sh
+          ./safe-devstack-vm-gate-wrap.sh
+      - link-logs
+
+    publishers:
+      - devstack-logs
+      - console-log

jenkins/jobs/macros-common.yaml

@@ -0,0 +1,95 @@
+- builder:
+    name: devstack-checkout
+    builders:
+      - shell: |
+          #!/bin/bash -xe
+          if [[ ! -e devstack-gate ]]; then
+              git clone git://git.openstack.org/openstack-infra/devstack-gate
+          else
+              cd devstack-gate
+              git remote set-url origin git://git.openstack.org/openstack-infra/devstack-gate
+              git remote update
+              git reset --hard
+              if ! git clean -x -f ; then
+                  sleep 1
+                  git clean -x -f
+              fi
+              git checkout master
+              git reset --hard remotes/origin/master
+              if ! git clean -x -f ; then
+                  sleep 1
+                  git clean -x -f
+              fi
+              cd ..
+          fi
+
+- builder:
+    name: devstack-checkout-http
+    builders:
+      - shell: |
+          #!/bin/bash -xe
+          if [[ ! -e devstack-gate ]]; then
+              git clone http://git.openstack.org/openstack-infra/devstack-gate
+          else
+              cd devstack-gate
+              git remote set-url origin http://git.openstack.org/openstack-infra/devstack-gate
+              git remote update
+              git reset --hard
+              if ! git clean -x -f ; then
+                  sleep 1
+                  git clean -x -f
+              fi
+              git checkout master
+              git reset --hard remotes/origin/master
+              if ! git clean -x -f ; then
+                  sleep 1
+                  git clean -x -f
+              fi
+              cd ..
+          fi
+
+- builder:
+    name: link-logs
+    builders:
+      - shell: |
+          #!/bin/sh
+          # TODO: Update these links if using a different gerrit server
+          echo "Triggered by: https://review.openstack.org/$ZUUL_CHANGE patchset $ZUUL_PATCHSET"
+
+          # TODO: Update this link to point to your log server
+          echo "Detailed logs: http://<your-log-server>/$LOG_PATH/"
+
+- publisher:
+    name: console-log
+    publishers:
+      - scp:
+          site: 'LogServer'
+          files:
+            - target: 'logs/$LOG_PATH'
+              copy-console: true
+              copy-after-failure: true
+
+- publisher:
+    name: devstack-logs
+    publishers:
+      - scp:
+          site: 'LogServer'
+          files:
+            - target: 'logs/$LOG_PATH'
+              source: 'logs/**'
+              keep-hierarchy: true
+              copy-after-failure: true
+
+- builder:
+    name: net-info
+    builders:
+      - shell: |
+          #!/bin/sh
+          export PATH=$PATH:/sbin
+          echo "Network interface addresses..."
+          ip address show
+          echo "Network routing tables..."
+          ip route show
+          ip -6 route show
+          echo "Network neighbors..."
+          ip neighbor show

jenkins/jobs/projects.yaml

@@ -0,0 +1,12 @@
+- project:
+    name: sandbox
+    github-org: openstack-dev
+    node: master
+    vendor: myvendor
+
+    jobs:
+        - noop-check-communication
+        - dsvm-tempest-full:
+            node: 'devstack_slave || devstack-precise-check || d-p-c'
+        - dsvm-tempest-my-cinder-driver:
+            node: 'd-p-c'

nodepool/elements/README.md

@@ -0,0 +1,3 @@
+Empty elements. Use these elements as a starting point:
+[openstack-infra/project-config](http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements)
+

nodepool/nodepool.yaml

@@ -0,0 +1,58 @@
+script-dir: /etc/nodepool/scripts
+elements-dir: /etc/nodepool/elements
+images-dir: /opt/nodepool_dib
+
+cron:
+  cleanup: '*/1 * * * *'
+  check: '*/15 * * * *'
+# TODO: Please choose a random hour for nodepool image updates.
+# This will help reduce the load on upstream git farms & mirros where all 3rd
+# party ci systems start building images at the same time.
+# Doing so is easy: update the first '17' below with a random number between 0 to 23
+# This references the hour of the day when images will be built.
+  image-update: '0 0 * * *'
+
+zmq-publishers:
+  - tcp://localhost:8888
+
+gearman-servers:
+  - host: 127.0.0.1
+
+labels:
+  - name: d-p-c
+    image: dpc
+    min-ready: 1
+    providers:
+      - name: local_01
+
+diskimages:
+    - name: dpc
+      elements:
+        - ubuntu
+        - vm
+        - openstack-repos
+        - puppet
+        - nodepool-base
+        - node-devstack
+      release: trusty
+      env-vars:
+        TMPDIR: /opt/dib_tmp
+        DIB_IMAGE_CACHE: /opt/dib_cache
+
+providers:
+  - name: local_01
+#TODO: Update the provider username, password, and authurl
+    username: '<%= provider_username %>'
+    password: '<%= provider_password %>'
+    auth-url: 'http://<openstack cloud>:5000/v2.0'
+    project-name: 'admin'
+    max-servers: 2
+    images:
+      - name: dpc
+        min-ram: 8192
+        diskimage: dpc
+        username: jenkins
+        private-key: '/home/nodepool/.ssh/id_rsa'
+
+targets:
+  - name: jenkins1

nodepool/scripts/README.md

@@ -0,0 +1,5 @@
+These scripts will be copied to the nodepool slave image's
+/etc/nodepool/scripts folder.
+
+The are optional. You can reference these:
+[openstack-infra/project-config](http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/scripts)

zuul/layout.yaml

@@ -0,0 +1,79 @@
+includes:
+    #TODO: Rename this file. These functions are what enables single use nodes in nodepool.
+  - python-file: openstack_functions.py
+
+pipelines:
+  - name: check
+    description: Newly uploaded patchsets enter this pipeline to receive an initial +/-1 Verified vote from Jenkins.
+    failure-message: "Build failed. For 3rd party CI contact info: https://wiki.openstack.org/wiki/ThirdPartySystems"
+    manager: IndependentPipelineManager
+    trigger:
+      gerrit:
+        - event: patchset-created
+        - event: change-restored
+        # TODO: Change "myvendor" below to your vendor's name to add a custom
+        # recheck trigger that runs the check pipeline jobs when someone
+        # adds a comment to a review that says "recheck myvendor".
+        - event: comment-added
+          comment: (?i)^(Patch Set [0-9]+:)?( [\w\\+-]*)*(\n\n)?\s*recheck myvendor\s*$
+    success:
+      gerrit:
+        verified: 1
+      smtp:
+        #TODO: Update these to real e-mail addresses
+        to: third_party_ci@example.com
+        from: zuul@example.com
+        subject: 'Silent check of {change.project} {change.number},{change.patchset} passed'
+    failure:
+      gerrit:
+        verified: -1
+      smtp:
+        #TODO: Update these to real e-mail addresses
+        to: third_party_ci@example.com
+        from: zuul@example.com
+        subject: 'Silent check of {change.project} {change.number},{change.patchset} failed'
+    merge-failure:
+      smtp:
+        #TODO: Update these to real e-mail addresses
+        to: third_party_ci@example.com
+        from: zuul@example.com
+        subject: Upstream change {change} has a merge failure
+
+
+  - name: silent
+    description: Newly uploaded patchsets enter this pipeline to check jobs whose results are NOT to be posted (because e.g. they are not yet stable)
+    manager: IndependentPipelineManager
+    trigger:
+      gerrit:
+        - event: patchset-created
+        - event: change-restored
+        - event: comment-added
+          comment: (?i)^(Patch Set [0-9]+:)?( [\w\\+-]*)*(\n\n)?\s*recheck myvendor\s*$
+    success:
+      smtp:
+        #TODO: Update these to real e-mail addresses
+        to: third_party_ci@example.com
+        from: zuul@example.com
+        subject: 'Silent check of {change.project} {change.number},{change.patchset} passed'
+    failure:
+      smtp:
+        #TODO: Update these to real e-mail addresses
+        to: third_party_ci@example.com
+        from: zuul@example.com
+        subject: 'Silent check of {change.project} {change.number},{change.patchset} failed'
+
+
+jobs:
+  - name: ^dsvm-tempest.*$
+    parameter-function: single_use_node
+
+projects:
+  - name: openstack-dev/ci-sandbox
+    check:
+      # Remove this after successfully verifying communication with upstream
+      # and seeing a posted successful review.
+      - noop-check-communication
+    silent:
+      # Uncomment this job when you have a jenkins slave running and want to
+      # test a full Tempest run within devstack.
+      - dsvm-tempest-full

zuul/openstack_functions.py

@@ -0,0 +1,38 @@
+# Copyright 2013 OpenStack Foundation
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+
+
+def set_log_url(item, job, params):
+    if hasattr(item.change, 'refspec'):
+        path = "%s/%s/%s/%s" % (
+            params['ZUUL_CHANGE'][-2:], params['ZUUL_CHANGE'],
+            params['ZUUL_PATCHSET'], params['ZUUL_PIPELINE'])
+    elif hasattr(item.change, 'ref'):
+        path = "%s/%s/%s" % (
+            params['ZUUL_NEWREV'][:2], params['ZUUL_NEWREV'],
+            params['ZUUL_PIPELINE'])
+    else:
+        path = params['ZUUL_PIPELINE']
+    params['BASE_LOG_PATH'] = path
+    params['LOG_PATH'] = path + '/%s/%s' % (job.name,
+                                            params['ZUUL_UUID'][:7])
+
+
+def single_use_node(item, job, params):
+    set_log_url(item, job, params)
+    params['OFFLINE_NODE_WHEN_COMPLETE'] = '1'
+
+
+def reusable_node(item, job, params):
+    set_log_url(item, job, params)