37 Commits

Author SHA1 Message Date
Jan Kundrát
834733bbe5 Let scripts return some useful message about failure
This patch exports a new env var for the script, the $TH_RESULT_FILE. If
the launched script writes something into the file name specified in
that env var and if the content of that file doesn't appear to indicate
a successful build, the content of the file is used as summary message
sent back to Zuul.

Originally, this patch attempted to do a similar thing through reading
back stuff from the shell's output log and reporting the last line from
that output, but the two major disadvantages were the inclusion of
timestamps in the log output, and Python's enrichment of the log line
with timestamps. I pondered writing a custom Python's logging.Handler
which simply remembers the last message, but the required changes in
utils.execute_to_log appeared rather messy to my untrained eye.

The main driver behind this change is a need to differentiate from hard
build failures (where we don't get any build artifacts because the build
simply failed) from "mere" test failures. Yes, our (KDE) tests are still
sloppy, and we have a fair amount of broken test still around, which is
why this difference matters to us. The result field in Zuul is more or
less free-form, so this patch does not impose any restrictions about its
format, except the "cannot start with 'SUCCESS'" because that one is
indeed very, very special.

It can be seen in action at http://ci-logs.kde.flaska.net/matrix.html .

Change-Id: I48c29d2566da12b02dcf27a551a058ecc4a303d4
2015-06-12 01:50:40 +02:00
Joshua Hesketh
f265245cce Fix envvars to be strings
Change-Id: Ia777482a44cbcc3199fb71cd0712cd498e652730
2015-06-05 20:14:38 +10:00
Joshua Hesketh
326fd8a230 Make git more verbose for debugging
Change-Id: I05ebe9e58bacc324dbe4430ef9627dd81c14f023
2015-06-05 12:37:44 +10:00
Joshua Hesketh
bad0b1fda2 Set env so git global has $HOME
Currently setting git config fails because the env isn't complete.

Change-Id: I95cabe87b9ad99cafb62b1f3595122f8f9c34649
2015-06-04 14:13:46 +10:00
Joshua Hesketh
a4b178d1f1 Fix hacking and requirements
Change-Id: Ie7a36b0a7957f102b6a1195cdc7c0403cdd9545a
2015-06-04 14:13:46 +10:00
Joshua Hesketh
5ef62278f3 Harden git cloning
Change-Id: Ie2659a265c5198c106c85f1ac0cf938ece29b286
2015-05-01 11:59:36 +10:00
Joshua Hesketh
9b2c312aa3 Add extra debugging info on git failure
Fetching from git sometimes fails. Log the IP in these cases
so we can investigate the git.o.o logs

Change-Id: I09994cf0706670c44412f5d6da97c67329b40d1b
2015-02-26 13:30:41 +11:00
Joshua Hesketh
f4b2038732 Send results back on failure
Change-Id: I2ffb64c4c96ab5d600e6381a74bf0c5080618ea3
2015-02-24 17:35:07 +11:00
Joshua Hesketh
91f4ff6594 Retry git cloning 3 times
Change-Id: I045aea5407fa291e25b476f91117aae6a03602f1
2015-02-24 16:21:24 +11:00
Joshua Hesketh
05baf01cf1 Improve log uploading and index generation
Turbo-hipster now produces different levels of logs for each job. These
need to be uploaded an in some cases have indexes generated for (eg
swift has no directory listing).

Support for zuul's swift instructions still need updating.

Change-Id: I572c8edfc856bb33998d1cfa0a8d31d274ab1bef
2014-12-03 15:02:46 +11:00
Joshua Hesketh
aae9786f1e Have real_db_upgrade store logs in results dir
This way the publisher will upload all of the logs and the individual
plugin doesn't need to handle pushing the files.

Change-Id: Ib13fd149828118518a115e473b4bae06d592b28e
2014-12-02 15:19:14 +11:00
Joshua Hesketh
62721fbb68 Update config.yaml to define jobs rather than plugins
Pay off some technical debt and make config.yaml define jobs rather than
plugins.

Change-Id: Ib4aa17649c02ab246d31515c8073230e8259cc4e
2014-12-02 13:24:30 +11:00
Joshua Hesketh
d5d7a21ed0 Improve how jobs log
Improve the logging for jobs in turbo-hipster so that if something fails
in a plugin (for example the db migration checks) it is logged and uploaded
appropriately.

This causes multiple logs per job.

Change-Id: I01e73ea418defbc0c1abd3b2b4357a816ddf99c3
2014-12-02 13:23:23 +11:00
Jan Kundrát
3e3deef7b8 Export job name to the shell script, too
self.job_name is something like "shell_script-${hostname}-${number}",
and job_arguments doesn't contain anything like a job name, either.

Change-Id: I775609990c740c29016abe89c5dce9ab81ca61b9
2014-11-28 10:13:05 +01:00
Jan Kundrát
2221d4835b Send debugging info about this program to Zuul
This will make sure that correct data appear in the output of `zuul show
running-jobs`.

Change-Id: If8a57e89cb23645ad7cc17f408c3b57253ccc945
2014-11-28 10:12:13 +01:00
Jan Kundrát
977a65422a Pass all zuul arguments to the job's shell script
The idea is that Zuul passes a lot of useful information, such as the
project name, target branch etc., as ZUUL_* options within the JSON job
data. Let's pass all of these variables to the launched shell script.
This will make it possible to e.g. have a single script that can do the
"right thing" for different projects with distinct build steps, etc.

Change-Id: I6f57ed19e9eef1b987fe2493d409b3ee51af22b6
2014-11-27 23:16:36 +01:00
Joshua Hesketh
1c8d2dfdc0 Update gerrit-git-prep to infra's
This provides better support for zuul merger parameters

Change-Id: I9b8f5b03a682374c67532784adfbf94176316013
2014-05-05 15:50:48 +10:00
Joshua Hesketh
6e20b16702 Send back final work_data if job fails
If a job has failed due to an uncaught exception we should send back
a work_data packet to give zuul more information about the failure.

Change-Id: Ifbf3037134ecd129e1b020bcca193ba092349c29
2014-04-09 13:11:09 +10:00
Joshua Hesketh
96adb287f8 Allow jobs to shutdown turbo-hipster
This lets jobs turn off and exit turbo-hipster once they are done.
This is useful for when using nodepool or when a job leaves the
environment dirty and we can't run more jobs on this worker.

Change-Id: I823be4196a5bf9ca92a14d9caf26163398a9434c
2014-04-03 10:42:49 +11:00
Joshua Hesketh
9cd2f933a4 Register worker functions once we are connected
This also re-registers once a server appears ensuring reconnection is
properly set up.

Change-Id: I9b915482494d1b273991cb8bdd1b6ad963cf25f2
2014-03-05 16:49:26 +11:00
Joshua Hesketh
38a1718c50 Stop all tasks on shutdown
This stops all of the working jobs when turbo-hipster is closing down

Change-Id: Ic160f75bd8222b233f81d269e4c035f95be11e47
2014-03-05 15:10:51 +11:00
Joshua Hesketh
15e1e6977e Make gerrit-git-fetch params configurable
Change-Id: I793153a63a82a98e90b0ecc2404e8d6596f2be48
2014-01-31 13:59:25 +11:00
Joshua Hesketh
ae4ef1eaa4 Only push the shell output log for now
Change-Id: Iaf7872d370b547fe25394e571f6d89f3f6d5e7c5
2014-01-31 13:13:27 +11:00
Joshua Hesketh
3cda79ff66 Send the work data result as soon as we know it
Change-Id: I40b46e19d298f18b3ca3cf031612ae94b3874bfa
2014-01-31 13:10:29 +11:00
Joshua Hesketh
9900554992 set the results_set_name to the filepath
Change-Id: I320b103a8957bcd74b1aea1e61722f9ed04e72a2
2014-01-30 16:37:06 +11:00
Joshua Hesketh
48de067f62 Calculate result in work_data
Change-Id: I999cc7c3e79225ae03764e621e77a5c105888970
2014-01-30 16:33:22 +11:00
Joshua Hesketh
b5f99b6158 Fix the result check
Change-Id: I0d190978280f48a244d651f354e88d9725eac9b5
2014-01-30 16:03:19 +11:00
Joshua Hesketh
3918c26c68 Fix log file to be uploaded
Change-Id: I3a8236d90817695c83cf0ae30ca19a39ab62d1dd
2014-01-30 15:47:30 +11:00
Joshua Hesketh
1f2d1a27d5 Fix order of steps and pass log file to the git prep script
Change-Id: I361ba7fc0827c11509de611841c484e38159f637
2014-01-30 15:41:53 +11:00
Joshua Hesketh
235b13dab5 Fix extra paramater for do_job_check
Change-Id: I8abccccd4b95924200eb11a7d29d1e6cde224938
2014-01-30 04:32:29 +00:00
Joshua Hesketh
221ae7442a Follow zuul's instructions to upload via swift
Change-Id: Ib6a5b0701f53d2edef8bc617dbc5a2812d5bb056
2014-01-22 16:10:30 +11:00
Joshua Hesketh
6055f3187c Add results uploader to generic ShellTask
Change-Id: I2ed0269bf6b47b13a001774a441b94a4746103f1
2014-01-22 15:04:54 +11:00
Joshua Hesketh
c73328c444 Continue prepping generic shell execution
Change-Id: Ie4c8b849f9ed2d80c09104a8b2e0a30f143b2cbb
2014-01-20 11:28:52 +11:00
Joshua Hesketh
81f87ed6d6 More refactoring of jobs
Move more of db_upgrade into a generic shell script executor.

Change-Id: I71d4e4ac800d1b7dd8f66d00b2ef757d5c39d0ab
2014-01-20 11:28:20 +11:00
Joshua Hesketh
9177876e16 Begin adding in a generic shell script runner
Change-Id: Ic4c657c91b6fc8ce12df7c5179a955b758d5ba08
2014-01-20 11:27:39 +11:00
Joshua Hesketh
6224554d06 Replace _do_next_step with a decorator
Change-Id: I2d44d97d7061f333a128dcfb3ee4124a89aafc22
2014-01-20 11:27:26 +11:00
Joshua Hesketh
e76a0dd790 Move common task functions into a model
Change-Id: If39a70edeaf5736dc84750367678bdb7cbe05506
2014-01-18 14:08:59 +11:00