The kolla docker/podman workers already have support for ignoring when
containers are missing. This patch just exposes that to the CLI with the
flag ``--ignore-missing``.
Change-Id: Ie74ea3ea66380063cf67f2e0edb2a0e160d89cd7
In many cases we use the kolla_address filter to look up the IP address
of the current host or another host on a particular network interface.
This filter uses the host's facts to determine the IP, meaning that we
must have gathered facts for the host, even if it is outside of a
requested --limit. This is a limitation, since it requires that all
hosts must be reachable, even if we are not directly configuring them.
Most instances of this cross-host fact referencing involve a controller,
since they host clustered services. The only instance found to affect
compute nodes is in the prometheus role, where Prometheus server needs
to know the IP address of all targets in its scrape configs.
If we are able to specify the address of the scrape targets as a static
variable such as a host variable, then facts would not be required for
compute nodes outside of the --limit.
Removing the requirement to have facts for all compute nodes has
benefits for performance (gathering facts for all hosts can take a long
time) and fault tolerance (we can operate when some compute hosts are
unreachable).
This change modifies the kolla_address filter to accept an optional
override_var argument which can be used to specify the name of a host
variable that may override the returned IP address. This is used in the
Prometheus server configuration to allow specifying the IP address used
by Prometheus server when collecting metrics from exporter using
a 'prometheus_target_address' host variable. If specified, this takes
precedence over the API interface address currently used. This makes it
possible to statically override prometheus_target_address and avoid the
cross-host fact reference.
This is not a complete solution because it is not yet possible to skip
the cross-host fact gathering step.
Partial-Bug: #2041860
Change-Id: I207ca56362de00d8ec578333eab9e1a72e7bcd19
This allows operators quickly diagnose all containers across
all hosts by running kolla-ansible check. It returns a list
of containers that are missing, not running or in unhealthy
state for each OpenStack service.
Change-Id: I36119ccdeb264aa3de928ec2254d6ff4cc955bfb
Implements: blueprint check-containers
Co-Authored-By: Roman Krček <roman.krcek@tietoevry.com>
From [1]:
If check is true, and the process exits with a non-zero
exit code, a CalledProcessError exception will be raised.
Attributes of that exception hold the arguments, the exit
code, and stdout and stderr if they were captured.
[1]: https://docs.python.org/3/library/subprocess.html
Closes-Bug: #2089173
Change-Id: I8cf38c2f7db1493e7303e94c212251fbeafaced3
I have no clue how it worked previously in CI, but now
it's using default path to the inventory - which does
not exist.
In addition to that, type=int in cliff will default to
None, so the check had to be rewritten - because we
always did cert expiry check instead of generating them.
Change-Id: I84d71558c2666ba2cfa47054f782d25ff0c1f691
When installing kolla-ansible with `pip install ./kolla-ansible`, pip
always creates a direct_url.json file, even when not using an editable
installation.
We see this behaviour with Python 3.12, while direct_url.json is only
created for editable installations on Python 3.9, which was used when
this code was initially developed for Kayobe.
When using a regular (non-editable) installation, this would make
kolla-ansible invoke site.yml from the source directory instead of the
virtualenv installation, causing a failure to load Ansible collections:
Invalid plugin FQCN (ansible.utils.ipaddr): unable to locate collection ansible.utils
Fix by returning the source URL only if dir_info.editable is True.
Change-Id: Icdc2cedaa6a6e3a6b4351b1f4369e2e8b3a2dc97
Moving the CLI to python allows for easier
maintenance and larger feature-set.
This patch introduces a few breaking changes!
The changes stem the nature of the cliff package.
- the order of parameters must be
kolla-ansible <action> <arguments>
- mariadb_backup and mariadb_recovery now are
mariadb-backup and mariadb-recovery
Closes-bug: #1589020
Signed-off-by: Roman Krček <roman.krcek@tietoevry.com>
Change-Id: I9749b320d4f5eeec601a055b597dfa7d8fb97ce2
wrong use of a f-string when no variable is templated
Signed-off-by: Sven Kieske <kieske@osism.tech>
Change-Id: I4ef5147eacef32ed93c21d44bf23b664adf1eb91
Ansible passes port as a string - therefore matching does not work
and we get https://nova_url:443/v2.1
Closes-Bug: #2063434
Change-Id: I76cce7f491c77b6b788d29c8644e87055f5cfff0
When kolla VIP address is changed the cell0 database connection is
now updated to the new address.
Closes-bug: #1915302
Change-Id: I35be54efb5aaa230702d0cebaae04f1e64c3bca3
Signed-off-by: Roman Krček <roman.krcek@tietoevry.com>
This avoids the need to use a proxy, or some other means, to connect to
Prometheus. This is disabled by default and can be enabled by setting
enable_prometheus_server_external to true.
Change-Id: Ia0af044ff436c2a204b357750a16ff49fcdfec45
Unlike other methods such as resolve(), get() does not return an Undefined object, but None.
This removes 4 ansible-lint warnings in various files calling kolla_address.
Closes-Bug: #2038281
Change-Id: I591a50512a954210f951c40a350ed4b9e1fc48ae
Use case: exposing single external https frontend and
load balancing services using FQDNs.
Support different ports for internal and external endpoints.
Introduced kolla_url filter to normalize urls like:
- https://magnum.external:443/v1
- http://magnum.external:80/v1
Change-Id: I9fb03fe1cebce5c7198d523e015280c69f139cd0
Co-Authored-By: Jakub Darmach <jakub@stackhpc.com>
The kolla-genpwd, kolla-mergepwd, kolla-readpwd and kolla-writepwd
commands now creates or updates passwords.yml with correct
permissions. Also they display warning message about incorrect
permissions.
Closes-Bug: #2018338
Change-Id: I4b50053ced9150499d1d09fd4a0ec2e243cf938b
Signed-off-by: Maksim Malchuk <maksim.malchuk@gmail.com>
Moves Hashi Vault client login to use `auth.approle.login` as
current method is being deprecated in the next release.
```
DeprecationWarning: Call to deprecated function 'auth_approle'.
This method will be removed in version '0.12.0' Please use
the 'login' method on the 'hvac.api.auth_methods.approle'
class moving forward.
client.auth_approle(vault_role_id, vault_secret_id)
```
Change-Id: Ie5c1ebe99c8508336cc10944fdaa742ad7d1d85e
This patch adds loadbalancer-config role
which is "wrapper" around haproxy-config
and proxysql-config role which will be added
in follow-up patches.
Change-Id: I64d41507317081e1860a94b9481a85c8d400797d
Kolla environment currently uses haproxy
to fullfill HA in mariadb. This patch
is switching haproxy to proxysql if enabled.
This patch is also replacing mariadb's user
'haproxy' with user 'monitor'. This replacement
has two reasons:
- Use better name to "monitor" galera claster
as there are two services using this user
(HAProxy, ProxySQL)
- Set password for monitor user as it's
always better to use password then not use.
Previous haproxy user didn't use password
as it was historically not possible with
haproxy and mariadb-clustercheck wasn't
implemented.
Depends-On: https://review.opendev.org/c/openstack/kolla/+/769385
Depends-On: https://review.opendev.org/c/openstack/kolla/+/765781
Depends-On: https://review.opendev.org/c/openstack/kolla/+/850656
Change-Id: I0edae33d982c2e3f3b5f34b3d5ad07a431162844
This change introduces automated configuration of firewalld and adds
a new filter for extracting services from the project_services dict.
the filter selects any enabled services and their haproxy element
and returns them so they can be iterated over.
This commit also enables automated configuration of firewalld from enabled
openstack services and adds them to the defined zone and reloads the
system firewall.
Change-Id: Iea3680142711873984efff2b701347b6a56dd355
From:
(kolla) 13:11 (s) marcin@puchatek:kolla-ansible$ kolla-genpwd
Traceback (most recent call last):
File "/home/marcin/.virtualenvs/kolla/bin/kolla-genpwd", line 8, in <module>
sys.exit(main())
File "/home/marcin/.virtualenvs/kolla/lib/python3.10/site-packages/kolla_ansible/cmd/genpwd.py", line 135, in main
genpwd(passwords_file, length, uuid_keys, ssh_keys, blank_keys,
File "/home/marcin/.virtualenvs/kolla/lib/python3.10/site-packages/kolla_ansible/cmd/genpwd.py", line 59, in genpwd
with open(passwords_file, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/etc/kolla/passwords.yml'
To:
(kolla) 13:17 (s) marcin@puchatek:kolla-ansible$ kolla-genpwd
ERROR: Passwords file "/etc/kolla/passwords.yml" is missing
Change-Id: I18a9559daeb3d124a03dcb735ebb01a2cf24f617
This key can be used by users in networking-generic-switch
scenario instead of adding cleartext password in ml2_conf.ini.
Change-Id: I10003e6526a55a97f22678ab81c411e4645c5157
this adds back the ability to configure
the rabbitmq/erlang kernel network interface
which was removed in https://review.opendev.org/#/c/584427/
seemingly by accident.
Closes-Bug: 1900160
Change-Id: I6f00396495853e117429c17fadfafe809e322a31
The contextfilter decorator was deprecated in jinja2 3.0.0, and has been
dropped in 3.1.0. This results in the following warning, and failed
attempts to use filters:
[WARNING]: Skipping plugin (filters.py) as it seems to be invalid:
module 'jinja2' has no attribute 'contextfilter'
This change switches to use the pass_context decorator. The minimum
version of Jinja2 is raised to 3 to ensure pass_context is present.
Change-Id: I649dd6211d3ae72b9539bc44652ef8cf5d579777
This commit adds two new cli commands to allow an operator
to read and write passwords into a configured Hashicorp Vault
KV.
Change-Id: Icf0eaf7544fcbdf7b83f697cc711446f47118a4d
By default, Ansible injects a variable for every fact, prefixed with
ansible_. This can result in a large number of variables for each host,
which at scale can incur a performance penalty. Ansible provides a
configuration option [0] that can be set to False to prevent this
injection of facts. In this case, facts should be referenced via
ansible_facts.<fact>.
This change updates all references to Ansible facts within Kolla Ansible
from using individual fact variables to using the items in the
ansible_facts dictionary. This allows users to disable fact variable
injection in their Ansible configuration, which may provide some
performance improvement.
This change disables fact variable injection in the ansible
configuration used in CI, to catch any attempts to use the injected
variables.
[0] https://docs.ansible.com/ansible/latest/reference_appendices/config.html#inject-facts-as-vars
Change-Id: I7e9d5c9b8b9164d4aee3abb4e37c8f28d98ff5d1
Partially-Implements: blueprint performance-improvements
this patchset has implemented:
- network (lb-mgmt-net)
- security groups and rules (used by amphora and health manager)
- amphora flavor (used by amphora)
- nova keypair (used by amphora at the time of debugging)
Add a octavia_amp_listen_port variable which used by amphora
Add amp_image_owner_id in octavia.conf
Implements: blueprint implement-automatic-deploy-of-octavia
Co-Authored-By: zhangchun <zhangchun@yovole.com>
Depends-On: https://review.opendev.org/652030
Change-Id: I67009d046925cfc02c1e0073c80085c1471975f6
Currently we generate multiple fluentd configuration files for inputs,
filters, formatters and outputs.
These are then included from the main td-agent.conf configuration file.
With a large number of hosts, this can take a long time to template.
Benchmarking of templating is available at [1].
This change switches to a single fluentd configuration file, with the
include done locally. For the default template files included with Kolla
Ansible we use Jinja includes, but this does not work with templates in
a different directory. We therefore use the Ansible template lookup
plugin, which has a slightly higher overhead than a jinja include, but
far lower than generating multiple templates. This should drastically
improve the performance of this task.
[1] https://github.com/stackhpc/ansible-scaling/blob/master/doc/template.md
Partially-Implements: blueprint performance-improvements
Change-Id: Ia8623be0aa861fea3e54d2c9e1c971dfd8e3afa9
The kolla-genpwd and kolla-mergepwd commands can be used to manipulate
the kolla passwords.yml file. The format is a YAML encoded dict of
password variable names to their values. If the format is not a dict,
the error messages are unhelpful. In particular, this can happen if the
file is encrypted e.g. via Ansible Vault.
For kolla-genpwd:
AttributeError: 'NoneType' object has no attribute 'items'
For kolla-mergepwd:
AttributeError: 'NoneType' object has no attribute 'update'
This change adds a more friendly message.
Change-Id: I27f0835b904e05006ae401adf383090322e1b891
Closes-Bug: #1880220
This includes some lightweight refactoring to avoid code
duplication.
This patch is made to be backportable to Train.
We now include Ansible in testing since Ussuri so the comments
about the bool filter are wrong.
Change-Id: Ia2e0f7f24988763bacfeafefb7977021f5949f4e
Closes-bug: #1848941
W503 and W504 are incompatible and we need to choose one of them.
Existing codes follows W503, so we disable W504.
Change-Id: Ic745e956dd332eb0fa49b93c1e6acb12f8a7f26c
In Ibecac60d1417269bbe25a280996ca9de6e6d018f, the services in the common
role were marked as being mapped to the 'all' group, since the
'service_mapped_to_host' filter expects every service definition to have
either a 'group' or 'host_in_groups' field. While this allows the filter
to pass the common services without error, it will not actually show
them as being mapped to any hosts. This is because the filter uses the
'group_names' variable, which contains all of the groups that a host
belongs to, except the default 'all' group.
This change fixes the issue by returning True from
service_mapped_to_host when the service's group is 'all'.
Change-Id: I39c8416f5d30a535c1743f9c43434b7d2a382196
Related-Bug: #1868596
The repo is Python 3 now, so update hacking to version 3.0 which
supports Python 3.
Fix problems found by updated hacking version.
Remove hacking and friends from lower-constraints, they are not needed
during installation.
Change-Id: I7ef5ac8a89e94f5da97780198619b6facc86ecfe
Now that py2 is gone, oslotest dropped dependency on mock and will
soon affect Ussuri CI [1], let's use unittest.mock built in py3.
This also fixes py38 jobs and proactively prevents py36 and py37
failing due to [1]. This is because we never included mock in
test-requirements (but in lower-constraints where it does not
really belong at all) and instead relied on oslotest to bring
it in.
[1] https://review.opendev.org/716322
Change-Id: I30e82e2d87418272a71c7ee089a8acdaf8872158
The service_mapped_to_host filter is used to check if a service is
mapped to a host, based on the group for the service or its
host_in_groups attribute if one exists. We check if the service's group
is in the 'groups' list. However, to get the list of groups to which a
host belongs, we should use the 'group_names' list.
This filter is currently only used in neutron IPv6 module loading, so
the effects are minimal.
Change-Id: I37409ca8d273b0426df0a648db222dc5432e738a
Closes-Bug: #1868285
Kolla-Ansible Ceph deployment mechanism has been deprecated in Train [1].
This change removes the Ansible code and associated CI jobs.
[1]: https://review.opendev.org/669214
Change-Id: Ie2167f02ad2f525d3b0f553e2c047516acf55bc2
