584 Commits

Author SHA1 Message Date
David Moreau-Simard
d75f5d2bf2 Refactor "emit-ara-html" into a new role: "ara-report"
We are going to provide support the ARA sqlite middleware which
provides the ability for an ARA web application to dynamically load
databases, eliminating the need for HTML generation.

HTML generation does not scale very well: it's orders of magnitude
larger than the database file containing the data and it also takes
a significant amount of time to generate the report.

It would be awkward to add dynamic database things into a role called
"emit-ara-html" and so I took the opportunity to refactor things a
little bit and make the role less prone to failures.

Note that this role was developed with the nested Ansible use case in
mind -- a job running it's own copy of Ansible with ARA should be able
to use this role to generate the report or save the database.

This role will be tested with 'base-test' first and it is expected to
make the 'emit-ara-html' role obsolete if everything works well.

Change-Id: Idedc0bfa1f0f89356b795fb9e2a16f9421a2dc18
Depends-On: I3b10c93b4902a9b45e23c227863e472697f662ef
2018-03-19 15:45:34 +00:00
Zuul
d28c326b36 Merge "Parameterize the test-setup role" 2018-03-08 19:08:24 +00:00
Zuul
eeae72ef66 Merge "Add known hosts for all users" 2018-03-08 10:05:17 +00:00
Paul Belanger
7e69f2f8ea
Fix lint issue with emit-job-header role
This is breaking project-config, and some reason we didn't catch it in
testing.  We should look into why that is.

Change-Id: I994484cdb28297fdff8c52ccf7455733584cd8ae
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-03-07 10:20:00 -05:00
Andrea Frittoli
870298bf54 Add known hosts for all users
The role today only adds known hosts for the zuul user.
This replicates what d-g does today by adding a default known_hosts
file which applies to all users.

We csnnot remove the original task that adds for the zuul user since
there's a unit test in openstack-zuul-jons that expects that.

Depends-on: https://review.openstack.org/550425/

Change-Id: I54d4d975fa54cbb6df7dbed525211ab5c2cd6754
2018-03-07 11:03:24 +00:00
Andreas Jaeger
71bc02db76 Fix emit-job-header
This currently fails with "Error 'nodepool' is undefined".

REmove nodepool line completely for now.

Change-Id: I134079a6118c8825fa91920c0f12c2d2789361a5
2018-03-07 11:53:31 +01:00
Zuul
f8fd1b0a92 Merge "Add Ansible version to job header" 2018-03-07 09:19:15 +00:00
Zuul
8ce392b66e Merge "Revert "Revert "Add zuul.{pipeline,nodepool.provider,executor.hostname} to job header""" 2018-03-07 09:19:14 +00:00
Andreas Jaeger
54077ba1e1 Fix condition in version-from-git
This should be the negation of the condition above, not exactly the same
but double negated.

Change-Id: I24e5f73480c83777f8dcb5ca71283afe5565212f
2018-03-07 08:25:18 +00:00
Andreas Jaeger
9ff13740f8 Fix version-from-git syntax
Fix "Syntax Error while loading YAML" with proper ordering of block.

Change-Id: I8a59856dbc91bec03adf193128ca6cc0ff7a7b54
2018-03-07 07:51:45 +01:00
Zuul
08724c3134 Merge "Add arm64 support to mirror setup" 2018-03-07 05:28:26 +00:00
Zuul
6188f949a9 Merge "Handle repos with no tags in version-from-git" 2018-03-07 04:06:23 +00:00
Ian Wienand
44bc52a539 Add arm64 support to mirror setup
Load the variables from an architecture specific file first --
ansible's ansible_architecture fact for arm64 is aarch64 (it's all
confusing, Ubuntu calls it arm64).  Add this file that sets up the
repo as "ubuntu-ports", which is our mirror of ports.ubuntu.com

Change-Id: Ieb50ae29c7f822e831ef1e32fdd03f74a93e298a
2018-03-07 13:49:26 +11:00
Zuul
c8b2ecf59a Merge "Add role for fetching javascript built content" 2018-03-06 21:58:17 +00:00
Monty Taylor
35da394f53
Add role for fetching javascript built content
Built content and source tarballs are different. Have a role that
handles uploading built content.

Append -content to the tarball so that a project can publish both a
source tarball and a built content tarball without stepping on each
other.

Change-Id: I33dff8080018bda281a00e648468de80d3ecd679
2018-03-06 13:33:10 -06:00
Zuul
f57fe26cdd Merge "Revert "Revert "Deploy the build ssh key to auth_keys""" 2018-03-06 19:18:45 +00:00
Zuul
90b3f4e6be Merge "Fix the copy-build-sshkey role" 2018-03-06 19:18:44 +00:00
Monty Taylor
a966445c16 Handle repos with no tags in version-from-git
The version-from-git role doesn't properly handle tag-less repos. Update
it to handle that case too.

Change-Id: I358037fd34f5dc202a6761b9e6f81bbe7870f3f2
2018-03-06 19:14:06 +00:00
Andrea Frittoli
d4ee2b5dc6 Revert "Revert "Deploy the build ssh key to auth_keys""
After fixing the copy-build-sshkey role, using it again in
multinode job pre.

This reverts commit ace4db4b04fd1eb71411d079524fded8189a72aa.

Change-Id: Id59216ce7dd62ff604ddf2c2c1cf27c4dca0deb0
2018-03-06 18:51:30 +00:00
David Moreau Simard
85c7ff6989 Fix the copy-build-sshkey role
Run the first task in the block, with become_user since the zuul
user may not access a generic use .ssh folder.

Depends-on: https://review.openstack.org/550091

Change-Id: I4ce120412079fe92502eee5310a03664aa55d5ce
2018-03-06 18:26:26 +00:00
Zuul
fd35b1b9aa Merge "ara: check for return code instead of succeed" 2018-03-05 16:02:09 +00:00
Zuul
0bbe3d31cb Merge "bindep: use shell instead of command with executable" 2018-03-05 16:02:07 +00:00
Zuul
979d6a74ac Merge "Allow updates from untrusted mirrors" 2018-03-05 16:02:04 +00:00
Monty Taylor
72fe550764 Add Ansible version to job header
As we look towards updating the version of Ansible zuul is running with,
letting people know what version it is would be friendly.

Change-Id: I09cd990af5365a8da3c339eede128166a76b35dd
2018-03-04 23:11:35 +00:00
David Moreau-Simard
ce176a5b6f Revert "Revert "Add zuul.{pipeline,nodepool.provider,executor.hostname} to job header""
The nodepool provider variable is not always available, for example
when running only on localhost, so default it to "N/A".

Additionally, we don't want these tasks to run multiple times if we
are running a multi-node job so use "run_once: yes" to prevent
printing the header more than once.

This reverts commit 27755ba83f83858f3dec3c23c3ab5101ea93a692.

Depends-On: Ia79a793200fcb89161783ff641b85106936084b5
Change-Id: Id9117935d9c53268ea893796154af03db21ee3a7
2018-03-04 23:11:19 +00:00
Jens Harbott
9861bc1640 Allow updates from untrusted mirrors
The default behaviour of apt has been changed, see [0]. When using
unsigned mirrors, we need to set the "trusted" option explicitly in
order to allow updates from these.

[0] https://manpages.debian.org/stretch/apt/apt-secure.8.en.html

Depends-On: https://review.openstack.org/#/c/536615/
Change-Id: I1a6818b022cd34c8899179c36cae962a6c8ec5ed
2018-03-03 06:49:27 +00:00
Dirk Mueller
294e9cbe1d Set wheels-mirror for openSUSE Tumbleweed
Unfortunately ansible_distribution is "openSUSE Tumbleweed" on
tumbleweed, and the whitespace in there is harmful. as there
isn't a version with tumbleweed, we can just replace
distribution/version with "opensuse-tumbleweed".

Change-Id: I21da4b5f851ec864dfb3cbf8ea6713bc12cda58f
2018-03-02 16:48:49 +01:00
Zuul
1d50e386cf Merge "Remove process-test-results" 2018-03-01 20:16:14 +00:00
Michel Peterson
724a520adf Fix lookup of subunit files in fetch-subunit-output
The role allows override of the zuul_work_dir variable, however in the
step where it retrieves the test results is the only step where it
prepends the directory with a fixed path. This breaks the step if the
zuul_work_dir given is a fully qualified path.

This patch defaults the zuul_work_dir to a fully qualified directory and
standarizes the usage zuul_work_dir across the role.

Change-Id: I62bc7571f625586a0d1b660f896e0c7cbaeae6e1
2018-02-28 14:33:35 +02:00
Michel Peterson
5c281b5494 Remove process-test-results
The role is an almost duplicate of fetch-subunit-output.

This patch favors the latter, since it's the more used role.

Depends-On: https://review.openstack.org/548394
Change-Id: I2dc7e6a0ead585601debf0f772fee25c96155448
2018-02-27 21:08:57 +02:00
Dirk Mueller
48099a5347 Fix Tumbleweed mirroring url path typo
Last revision of I700ba2279dc7aadc7051cad53898e1de9007a08d
of course had a typo compared to what was tested successfully before.

Change-Id: I6f1053d69650ee6235089e61e7f19f8927d8e8f5
2018-02-27 15:20:18 +01:00
Andrea Frittoli
ace4db4b04 Revert "Deploy the build ssh key to auth_keys"
This commits starts using the copy-build-sshkey role, however
the role is buggy, so we should first revert this, then use
reapply this after it's tested.

This reverts commit 4bdb01e60a5085c65f0a80e8f8bed743e7043b5d.

Change-Id: Ibbcfd4a0a9c37ce819cd2ad14109e9cdd20b3a1e
2018-02-26 15:06:08 +00:00
Zuul
bbf98b459b Merge "Setup repositories for openSUSE Tumbleweed" 2018-02-26 13:35:00 +00:00
Zuul
85bc67c592 Merge "Deploy the build ssh key to auth_keys" 2018-02-26 12:20:38 +00:00
Andrea Frittoli
4bdb01e60a Deploy the build ssh key to auth_keys
Deploy the build ssh key to the authorized_keys for root user
on all hosts in a multinode environment, to allow passwordless
ssh between nodes.

Change-Id: I3b1a8ce825d093a886f6dee2c1c8d49204939274
2018-02-26 11:56:27 +00:00
Dirk Mueller
7172bd5dc8 Setup repositories for openSUSE Tumbleweed
Tumbleweed as a rolling release model distribution does not
have a sensible release version and is kept distinct from the
classing "release + update" distributions that are called "Leap".

Conditionalize the repository setup for tumbleweed properly,
the base path is different and it does not have a releaseversion
component (the release version is always changing, it is
the snapshot date in the form YYYYMMDD).

Change-Id: I700ba2279dc7aadc7051cad53898e1de9007a08d
2018-02-25 21:15:09 +01:00
Zuul
5c0bb18336 Merge "Add abstract attribute to base jobs" 2018-02-23 18:00:44 +00:00
Andreas Jaeger
aee28f884e Add abstract attribute to base jobs
Add abstract attribute to base jobs, these should be inherited and not
used as is.

Change-Id: If43e99c0954acf776997d8399b9ea40966b362f6
2018-02-23 17:54:06 +01:00
Tristan Cacqueray
834358541d ara: check for return code instead of succeed
In Ansible-2.4, the |succeed behavior changed and always return True
when task uses 'failed_when: false'.
Also uses "type -p" instead of "which" that is used in other roles.

Change-Id: Iddcff07a40dc6c36a1a1c2101a7822697cdbbc34
2018-02-22 06:56:29 +00:00
Tristan Cacqueray
8a6b1215c3 bindep: use shell instead of command with executable
As of Ansible 2.4, the parameter 'executable' is
no longer supported with the 'command' module.

Also check for return code since it the |succeed changed
and now always returns True when using failed_when: false.

Change-Id: I4c33db7f3d6fd613a6144f7520054cbcf50684e1
2018-02-22 06:40:11 +00:00
Andrea Frittoli (andreaf)
ac60bc714b Improve file renaming in stage-output
We only rename the extensions for files specified in zuul_copy_output
but we should rename files in folders specified in zuul_copy_output as
well. In addition we need to rename files or folders that start with a
'.' which would otherwise not be visible once uploaded to logs.

Change-Id: Id55ab51019d10d5ccbc6531549758ac71ad276be
2018-02-16 08:03:56 +00:00
Zuul
46fb37f8bb Merge "Change the list of extensions to a dict" 2018-02-09 07:47:46 +00:00
Andrea Frittoli (andreaf)
f7fd2fce61 Change the list of extensions to a dict
The stage-output role accepts a list of extensions to be replaced as txt
so that they can be seen directly from within a web browser.
Using a list means that children jobs cannot add to the parent setting,
only overwrite. Adding support for dicts while preserving, at least for
now, support for lists - to avoid breaking existing implementations.

This reverts change 844b3333aae02d596061386b5a1a2fbc821950be.

Change-Id: I5052c873f0a5da7ecaa38627e8900af86bbec2f9
Needed-By: https://review.openstack.org/539686
Needed-By: https://review.openstack.org/539708
Needed-By: https://review.openstack.org/539854
2018-02-08 18:52:13 +00:00
Zuul
97c55dbd37 Merge "Revert "Test Use item.checkout from zuul.projects when mirroring"" 2018-02-07 05:30:03 +00:00
Zuul
6cfe94a80a Merge "Use item.checkout from zuul.projects when mirroring" 2018-02-05 20:59:26 +00:00
James E. Blair
bd1ab3e8bf Revert "Test Use item.checkout from zuul.projects when mirroring"
This reverts commit c185af7449c50cded02c5c370a467015acc20395.

Change-Id: I070a295cfe418604040e8d9a5174bc88644b8972
Depends-On: https://review.openstack.org/540949
2018-02-05 12:58:51 -08:00
Monty Taylor
4d2df638d1 Use item.checkout from zuul.projects when mirroring
We need to set the remote HEAD of the repos after we mirror them. We
have been doing that by looking at git status on the local repos to find
what their head has been set to. However, we have 'checkout' in
zuul.projects now, so we can really just use that.

Change-Id: I251da11b7767793278c3f08dcf64aa008e759669
Depends-On: https://review.openstack.org/540948
2018-02-05 11:21:44 -08:00
James E. Blair
64048189a4 Fix error in test-mirror-workspace-git-repos
Change-Id: I32fe15c1e62a9e4800457af80e25758afa988803
2018-02-05 11:16:45 -08:00
James E. Blair
c185af7449 Test Use item.checkout from zuul.projects when mirroring
This creates a test role for change https://review.openstack.org/540934
that we can use in base-test.

Change-Id: I23b468b9440c6a4ec7e4c34f351b1934e0f63be0
2018-02-05 09:31:45 -08:00
Doug Hellmann
2943ce05c9 add debug info to mirror-workspace-git-repos
Sometimes the task to determine the local HEAD of projects we've
checked out is failing in a way that we can't parse. Add a step before
the one that fails to show the full git status output to help with
debugging.

Change-Id: I095489249593442e1e9a3b346aed1ecf4135825f
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-02-05 10:06:39 -05:00