opendev/system-config
Code Issues Proposed changes

Merge "Fixup some details in the zuul doc"

Browse Source
This commit is contained in:
Zuul 2021-10-22 16:48:02 +00:00 committed by Gerrit Code Review
commit f2384b8b91
1 changed files with 35 additions and 35 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

70
doc/source/zuul.rst
View File

@ -13,9 +13,9 @@ At a Glance
:Hosts: :Hosts:
* https://zuul.opendev.org * https://zuul.opendev.org
* zuul*.openstack.org * zuul*.opendev.org
* ze*.openstack.org * ze*.opendev.org
* zm*.openstack.org * zm*.opendev.org
:Configuration: :Configuration:
* :config:`zuul/main.yaml` * :config:`zuul/main.yaml`
* :config:`zuul.d` * :config:`zuul.d`
@ -26,12 +26,12 @@ At a Glance
:Resources: :Resources:
* `Zuul Reference Manual`_ * `Zuul Reference Manual`_
:Chat: :Chat:
* #zuul on OFTC * ``#zuul:opendev.org`` on Matrix
Overview Overview
======== ========
The OpenStack project uses a number of pipelines in Zuul: The OpenDev project uses a number of pipelines in Zuul:
**check** **check**
Newly uploaded patchsets enter this pipeline to receive an initial Newly uploaded patchsets enter this pipeline to receive an initial
@ -97,7 +97,7 @@ Zuul has three main subsystems:
* Zuul Executors * Zuul Executors
* Zuul Web * Zuul Web
that in OpenStack's deployment depend on four 'external' systems: that in OpenDev's deployment depend on four 'external' systems:
* Nodepool * Nodepool
* Zookeeper * Zookeeper
@ -111,7 +111,7 @@ The Zuul Scheduler and gear are all co-located on a single host,
referred to by the ``zuul.opendev.org`` CNAME in DNS. referred to by the ``zuul.opendev.org`` CNAME in DNS.
Zuul is stateless, so the server does not need backing up. However Zuul is stateless, so the server does not need backing up. However
zuul talks through git and ssh so you will need to manually check ssh Zuul talks through git and ssh so you will need to manually check ssh
host keys as the zuul user. host keys as the zuul user.
e.g.:: e.g.::
@ -122,7 +122,7 @@ e.g.::
The Zuul Scheduler talks to Nodepool using Zookeeper and distributes work to The Zuul Scheduler talks to Nodepool using Zookeeper and distributes work to
the executors using gear. the executors using gear.
OpenStack's Zuul installation is also configured to write job results into OpenDev's Zuul installation is also configured to write job results into
a MySQL database via the SQL Reporter plugin. The database for that is a a MySQL database via the SQL Reporter plugin. The database for that is a
Rackspace Cloud DB and is configured in the ``mysql`` entry of the Rackspace Cloud DB and is configured in the ``mysql`` entry of the
``zuul_connection_secrets`` entry for the ``zuul-scheduler`` group. ``zuul_connection_secrets`` entry for the ``zuul-scheduler`` group.
@ -131,12 +131,12 @@ Executors
--------- ---------
The Zuul Executors are a horizontally scalable set of servers named The Zuul Executors are a horizontally scalable set of servers named
ze*.openstack.org. They perform git merging operations for the scheduler ``ze*.opendev.org``. They perform git merging operations for the scheduler
and execute Ansible playbooks to actually run jobs. and execute Ansible playbooks to actually run jobs.
Our jobs are configured to upload as much information as possible along with Our jobs are configured to upload as much information as possible along with
their logs, but if there is an error which can not be diagnosed in that their logs, but if there is an error which can not be diagnosed in that
manner, logs are available in the executor-debug log file on manner, logs are available in the ``executor-debug`` log file on
the executor host. You may use the Zuul build UUID to track the executor host. You may use the Zuul build UUID to track
assignment of a given job from the Zuul scheduler to the Zuul executor assignment of a given job from the Zuul scheduler to the Zuul executor
used by that job. used by that job.
@ -148,8 +148,8 @@ Web
--- ---
Zuul Web is a horizontally scalable service. It is currently running colocated Zuul Web is a horizontally scalable service. It is currently running colocated
with the scheduler on zuul.openstack.org. Zuul Web provides live console with the scheduler on ``zuul.opendev.org``. Zuul Web provides live console
streaming and will be the home of various web dashboards such as the status streaming and is the home of various web dashboards such as the status
page. page.
Zuul Web is stateless so is safe to restart, however restarting it will result Zuul Web is stateless so is safe to restart, however restarting it will result
@ -161,8 +161,8 @@ Restarting Zuul Services
Currently the safest way to restart the Zuul scheduler is to restart all Currently the safest way to restart the Zuul scheduler is to restart all
services at the same time. The reason for this is that if the scheduler is services at the same time. The reason for this is that if the scheduler is
restarted but executors are not then the executors and scheduler can get out restarted, but executors are not, then the executors and scheduler can get out
of sync with each other. Note that restarting zuul web or a single executor of sync with each other. Note that restarting Zuul Web or a single executor
should continue to be safe as noted above, but this process should generally should continue to be safe as noted above, but this process should generally
be preferred. be preferred.
@ -173,29 +173,29 @@ releases or have other important changes that are about to land.
Since Zuul is stateless, some work needs to be done to save and then Since Zuul is stateless, some work needs to be done to save and then
re-enqueue patches when restarts are done. To accomplish this, start by re-enqueue patches when restarts are done. To accomplish this, start by
running the zuul-changes script to save the check and gate queues:: running the ``zuul-changes`` script to save the check and gate queues::
root@zuul02# ~root/zuul-changes.py https://zuul.opendev.org >queues-$(date +%Y%m%d).sh root@zuul02# ~root/zuul-changes.py https://zuul.opendev.org >queues-$(date +%Y%m%d).sh
This script will be executed when Zuul is up and running again to restore The resulting script will be executed when Zuul is up and running again to restore
the previous queue contents. the previous queue contents.
One other thing to consider before restarting all zuul services is you may One other thing to consider before restarting all zuul services is you may
want to update all of the zuul docker images. This can be useful if restarting want to update all of the zuul docker images. This can be useful if restarting
Zuul to correct a bug that was fixed in the Zuul codebase. To do this run Zuul to correct a bug that was fixed in the Zuul codebase. To do this run
the zuul_pull.yaml playbook from bridge:: the ``zuul_pull.yaml`` playbook from bridge::
root@bridge# ansible-playbook -f 20 /home/zuul/src/opendev.org/opendev/system-config/playbooks/zuul_pull.yaml root@bridge# ansible-playbook -f20 /home/zuul/src/opendev.org/opendev/system-config/playbooks/zuul_pull.yaml
Once ready to restart all Zuul services you will want to run the Once ready to restart all Zuul services you will want to run the
zuul_restart.yaml playbook from bridge to do this:: ``zuul_restart.yaml`` playbook from bridge to do this::
root@bridge# ansible-playbook -f20 /home/zuul/src/opendev.org/opendev/system-config/playbooks/zuul_restart.yaml root@bridge# ansible-playbook -f20 /home/zuul/src/opendev.org/opendev/system-config/playbooks/zuul_restart.yaml
Once this playbook is done running the services will have been restarted, but Once this playbook is done running, the services will have been restarted, but
the Zuul system still needs to load its configs before it is ready to do work. the Zuul system still needs to load its configs before it is ready to do work.
The `root <https://zuul.opendev.org/>`_ of the Zuul dashboard will show you The `root <https://zuul.opendev.org/>`_ of the Zuul dashboard will show you
loaded tenants. Once all tenants show up on this page it is safe to proceed loaded tenants. Once all tenants show up on this page, it is safe to proceed
with re-enqueing changes to pipelines with the script we generated earlier. with re-enqueing changes to pipelines with the script we generated earlier.
Note that the OpenStack tenant takes the most time. If you wait for it to Note that the OpenStack tenant takes the most time. If you wait for it to
show up in the dashboard you should be ready to go. You can double check show up in the dashboard you should be ready to go. You can double check
@ -229,12 +229,12 @@ ZooKeeper. To construct this you will need to know the Zuul connection name
and full project name. The connection name in the example above is 'gerrit', and full project name. The connection name in the example above is 'gerrit',
replace it with the appropriate connection name for the project you are looking replace it with the appropriate connection name for the project you are looking
at. Next is the unique project name. In the example above we start with at. Next is the unique project name. In the example above we start with
`opendev/system-config` and split it on `/`. Everything before the first `/` ``opendev/system-config`` and split it on ``/``. Everything before the first ``/``
is the next component of our name in this case, `opendev`. Then we take the is the next component of our name in this case, ``opendev``. Then we take the
entire name `opendev/system-config` and URL encode it to get entire name ``opendev/system-config`` and URL encode it to get
`opendev%2Fsystem-config` which becomes our last component. ``opendev%2Fsystem-config`` which becomes our last component.
Save the output of this jq command to a file `secret.pem`. Then extract the Save the output of this jq command to a file ``secret.pem``. Then extract the
secret ciphertext from the job configuration to remove surrounding secret ciphertext from the job configuration to remove surrounding
YAML (there is no need to recombine split lines) and run the following YAML (there is no need to recombine split lines) and run the following
command to decrypt:: command to decrypt::
@ -242,8 +242,6 @@ command to decrypt::
cat ciphertext.txt | sed 's/^ *//' | base64 -d | sudo openssl rsautl -decrypt -oaep -inkey \ cat ciphertext.txt | sed 's/^ *//' | base64 -d | sudo openssl rsautl -decrypt -oaep -inkey \
secret.pem secret.pem
.. _zuul_github_projects:
Debugging Problems Debugging Problems
------------------ ------------------
@ -253,13 +251,13 @@ daunting to start as Zuul's logs are quite verbose. The good news is that once
you learn a few tricks those verbose logs become quite the powerful tool. you learn a few tricks those verbose logs become quite the powerful tool.
Often the best place to start is grepping the Zuul scheduler debug log for Often the best place to start is grepping the Zuul scheduler debug log for
the pipeline entry identifier (eg change number, tag, or rev sha1):: the pipeline entry identifier (eg change number, tag, or ref sha1)::
you@zuul02$ grep 123456 /var/log/zuul/debug.log you@zuul02$ grep 123456 /var/log/zuul/debug.log
you@zuul02$ grep c6229660cda0af42ecd5afbe7fefdb51136a0436 /var/log/zuul/debug.log you@zuul02$ grep c6229660cda0af42ecd5afbe7fefdb51136a0436 /var/log/zuul/debug.log
In many of these log lines you'll see Zuul event IDs like In many of these log lines you'll see Zuul event IDs like
`[e: 1718628fe39643e1bd6a88a9a1477b4f]` this ID identifies the event that ``[e: 1718628fe39643e1bd6a88a9a1477b4f]``. This ID identifies the event that
triggered Zuul to take action for these changes and is logged through all triggered Zuul to take action for these changes and is logged through all
the Zuul services. It can be very powerful to do a grep on this event ID and the Zuul services. It can be very powerful to do a grep on this event ID and
trace through the actions that the scheduler took for this event:: trace through the actions that the scheduler took for this event::
@ -272,10 +270,12 @@ ID to grep for actions related to this even on the executor::
you@ze01$ grep 1718628fe39643e1bd6a88a9a1477b4f /var/log/zuul/executor-debug.log you@ze01$ grep 1718628fe39643e1bd6a88a9a1477b4f /var/log/zuul/executor-debug.log
As you trace through the logs related to a change or event ID you can look for As you trace through the logs related to a change or event ID you can look for
`ERROR` or `Traceback` messages to try and identify the underlying source of ``ERROR`` or ``Traceback`` messages to try and identify the underlying source of
the problem. Note that `Traceback` messages are not prefixed with the event the problem. Note that ``Traceback`` messages are not prefixed with the event
ID which means you'll have to grep with additional context, for example using ID which means you'll have to grep with additional context, for example using
`grep -B20 -A20`. ``grep -B20 -A20``.
.. _zuul_github_projects:
GitHub Projects GitHub Projects
=============== ===============
@ -291,7 +291,7 @@ navigate to the `OpenDev Zuul`_ app in the GitHub UI. From there they can
click "Install", then choose the project or organization they want to install click "Install", then choose the project or organization they want to install
the App on. the App on.
The repository then needs to be added to the `zuul/main.yaml` file before Zuul The repository then needs to be added to the ``zuul/main.yaml`` file before Zuul
can be configured to actually run jobs on it. can be configured to actually run jobs on it.
Information about the configuration of the OpenDev Zuul App itself can be Information about the configuration of the OpenDev Zuul App itself can be
@ -299,4 +299,4 @@ found on the :ref:`github` page at :ref:`openstack_zuul_app`.
.. _OpenDev Zuul: https://github.com/apps/opendev-zuul .. _OpenDev Zuul: https://github.com/apps/opendev-zuul
.. _Zuul Reference Manual: https://zuul-ci.org/docs/zuul .. _Zuul Reference Manual: https://zuul-ci.org/docs/zuul
.. _Zuul Status Page: http://zuul.opendev.org .. _Zuul Status Page: https://zuul.opendev.org