performa/os-faults
Code Issues Proposed changes

Update documentation and examples

Browse Source 
Change-Id: Ia6943edb541ab80e8dc7c6f878179e36caeaf5dc
This commit is contained in:
Ilya Shakhat 2017-11-03 11:40:32 +01:00
parent 622b43d040
commit b8e4a30385
19 changed files with 224 additions and 99 deletions

133
README.rst

@ -18,7 +18,7 @@ IPMI driver).
Installation
------------
Reqular installation::
Regular installation::
pip install os-faults
@ -31,18 +31,73 @@ please use the following command to install os-faults with extra dependencies::
Configuration
-------------
The cloud deployment configuration schema is an extension to the cloud config
used by the `os-client-config <https://github.com/openstack/os-client-config>`_
library:
The cloud deployment configuration is specified in JSON/YAML format or Python dictionary.
The library operates with 2 types of objects:
* `service` - is a software that runs in the cloud, e.g. `nova-api`
* `nodes` - nodes that host the cloud, e.g. a server with a hostname
Example 1. DevStack
~~~~~~~~~~~~~~~~~~~
Connection to DevStack can be specified using the following YAML file:
.. code-block:: yaml
cloud_management:
driver: devstack
args:
address: devstack.local
username: stack
private_key_file: cloud_key
iface: enp0s8
OS-Faults library will connect to DevStack by address `devstack.local` with user `stack`
and SSH key located in file `cloud_key`. Default networking interface is specified with
parameter `iface`. Note that user should have sudo permissions (by default DevStack user has them).
DevStack driver is responsible for service discovery. For more details please refer
to driver documentation: http://os-faults.readthedocs.io/en/latest/drivers.html#devstack-systemd-devstackmanagement
Example 2. An OpenStack with services and power management
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
An arbitrary OpenStack can be handled too with help of `universal` driver.
In this example os-faults is used as Python library.
.. code-block:: python
cloud_config = {
'cloud_management': {
'driver': 'devstack',
'args': {
'address': 'devstack.local',
'username': 'root',
'driver': 'universal',
},
'node_discover': {
'driver': 'node_list',
'args': [
{
'ip': '192.168.5.127',
'auth': {
'username': 'root',
'private_key_file': 'openstack_key',
}
},
{
'ip': '192.168.5.128',
'auth': {
'username': 'root',
'private_key_file': 'openstack_key',
}
}
]
},
'services': {
'memcached': {
'driver': 'system_service',
'args': {
'service_name': 'memcached',
'grep': 'memcached',
}
}
},
'power_managements': [
@ -52,52 +107,41 @@ library:
'connection_uri': 'qemu+unix:///system',
}
},
{
'driver': 'ipmi',
'args': {
'mac_to_bmc': {
'aa:bb:cc:dd:ee:01': {
'address': '55.55.55.55',
'username': 'foo',
'password': 'bar',
}
}
}
}
]
}
Establish a connection to the cloud and verify it:
The config contains all OpenStack nodes with credentials and all
services. OS-Faults will automatically figure out the mapping between services
and nodes. Power management configuration is flexible and supports
mixed bare-metal / virtualized deployments.
First let's establish a connection to the cloud and verify it:
.. code-block:: python
destructor = os_faults.connect(cloud_config)
destructor.verify()
cloud_management = os_faults.connect(cloud_config)
cloud_management.verify()
The library can also read configuration from a file and the file can be in the
following three formats: os-faults.{json,yaml,yml}. The configuration file can
be specified in the `OS_FAULTS_CONFIG` environment variable or can be read from
one of the default locations:
The library can also read configuration from a file in YAML or JSON format.
The configuration file can be specified in the `OS_FAULTS_CONFIG` environment
variable. By default the library searches for file `os-faults.{json,yaml,yml}`
in one of locations:
* current directory
* ~/.config/os-faults
* /etc/openstack
Make some destructive actions:
Now let's make some destructive action:
.. code-block:: python
destructor.get_service(name='keystone').restart()
cloud_management.get_service(name='memcached').kill()
The library operates with 2 types of objects:
* `service` - is a software that runs in the cloud, e.g. `nova-api`
* `nodes` - nodes that host the cloud, e.g. a hardware server with a hostname
Human API
---------
Simplified API
--------------
Simplified API is used to inject faults in a human-friendly form.
Human API is simplified and self-descriptive. It includes multiple commands
that are written like normal English sentences.
**Service-oriented** command performs specified `action` against `service` on
all, on one random node or on the node specified by FQDN::
@ -106,7 +150,7 @@ all, on one random node or on the node specified by FQDN::
Examples:
* `Restart Keystone service` - restarts Keystone service on all nodes.
* `kill nova-api service on one node` - restarts Nova API on one
* `kill nova-api service on one node` - kills Nova API on one
randomly-picked node.
**Node-oriented** command performs specified `action` on node specified by FQDN
@ -116,7 +160,7 @@ or set of service's nodes::
Examples:
* `Reboot one node with mysql` - reboots one random node with MySQL.
* `Reset node-2.domain.tld node` - reset node `node-2.domain.tld`.
* `Reset node-2.domain.tld node` - resets node `node-2.domain.tld`.
**Network-oriented** command is a subset of node-oriented and performs network
management operation on selected nodes::
@ -141,8 +185,8 @@ Get a service and restart it:
.. code-block:: python
destructor = os_faults.connect(cloud_config)
service = destructor.get_service(name='glance-api')
cloud_management = os_faults.connect(cloud_config)
service = cloud_management.get_service(name='glance-api')
service.restart()
Available actions:
@ -160,14 +204,13 @@ Get all nodes in the cloud and reboot them:
.. code-block:: python
nodes = destructor.get_nodes()
nodes = cloud_management.get_nodes()
nodes.reboot()
Available actions:
* `reboot` - reboot all nodes gracefully
* `poweroff` - power off all nodes abruptly
* `reset` - reset (cold restart) all nodes
* `oom` - fill all node's RAM
* `disconnect` - disable network with the specified name on all nodes
* `connect` - enable network with the specified name on all nodes
@ -187,7 +230,7 @@ Get nodes where l3-agent runs and disable the management network on them:
.. code-block:: python
fqdns = neutron.l3_agent_list_hosting_router(router_id)
nodes = destructor.get_nodes(fqdns=fqdns)
nodes = cloud_management.get_nodes(fqdns=fqdns)
nodes.disconnect(network_name='management')
4. Operate with services
@ -197,6 +240,6 @@ Restart a service on a single node:
.. code-block:: python
service = destructor.get_service(name='keystone')
service = cloud_management.get_service(name='keystone')
nodes = service.get_nodes().pick()
service.restart(nodes)

8
doc/source/drivers.rst

@ -5,14 +5,16 @@ Drivers
Cloud management
----------------
.. cloud_driver_doc:: devstack
.. cloud_driver_doc:: devstack_systemd
.. cloud_driver_doc:: universal
.. cloud_driver_doc:: fuel
.. cloud_driver_doc:: tcpcloud
.. cloud_driver_doc:: devstack
Power management
----------------
@ -33,6 +35,8 @@ Service drivers
.. driver_doc:: process
.. driver_doc:: system_service
.. driver_doc:: linux_service
.. driver_doc:: screen

14
doc/source/quickstart/api.rst

@ -16,8 +16,8 @@ Simplified API is used to inject faults in a human-friendly form.
.. code-block:: python
import os_faults
destructor = os_faults.connect(config_filename='os-faults.yaml')
os_faults.human_api(destructor, 'restart keystone service')
cloud_management = os_faults.connect(config_filename='os-faults.yaml')
os_faults.human_api(cloud_management, 'restart keystone service')
**Service-oriented** command performs specified `action` against `service` on
@ -62,8 +62,8 @@ Get a service and restart it:
.. code-block:: python
destructor = os_faults.connect(cloud_config)
service = destructor.get_service(name='glance-api')
cloud_management = os_faults.connect(cloud_config)
service = cloud_management.get_service(name='glance-api')
service.restart()
Available actions:
@ -81,7 +81,7 @@ Get all nodes in the cloud and reboot them:
.. code-block:: python
nodes = destructor.get_nodes()
nodes = cloud_management.get_nodes()
nodes.reboot()
Available actions:
@ -108,7 +108,7 @@ Get nodes where l3-agent runs and disable the management network on them:
.. code-block:: python
fqdns = neutron.l3_agent_list_hosting_router(router_id)
nodes = destructor.get_nodes(fqdns=fqdns)
nodes = cloud_management.get_nodes(fqdns=fqdns)
nodes.disconnect(network_name='management')
4. Operate with services
@ -118,6 +118,6 @@ Restart a service on a single node:
.. code-block:: python
service = destructor.get_service(name='keystone')
service = cloud_management.get_service(name='keystone')
nodes = service.get_nodes().pick()
service.restart(nodes)

19
doc/source/quickstart/basics.rst

@ -6,18 +6,17 @@ Basics
Configuration file
------------------
The cloud deployment configuration schema is an extension to the cloud config
used by the `os-client-config <https://github.com/openstack/os-client-config>`_
library:
The cloud deployment configuration schema has simple YAML/JSON format:
.. code-block:: yaml
cloud_management:
driver: devstack
args:
address: 192.168.1.240
username: ubuntu
iface: enp0s3
address: devstack.local
username: stack
private_key_file: cloud_key
iface: enp0s8
power_managements:
- driver: libvirt
@ -47,8 +46,8 @@ Establish a connection to the cloud and verify it:
.. code-block:: python
import os_faults
destructor = os_faults.connect(config_filename='os-faults.yaml')
destructor.verify()
cloud_management = os_faults.connect(config_filename='os-faults.yaml')
cloud_management.verify()
or via CLI::
@ -60,9 +59,11 @@ Make some destructive actions:
.. code-block:: python
destructor.get_service(name='keystone').restart()
cloud_management.get_service(name='keystone').restart()
or via CLI::
$ os-inject-fault -c os-faults.yaml restart keystone service

2
doc/source/quickstart/config_spec.rst

@ -55,7 +55,7 @@ Example configuration:
cloud_management
----------------
This parameter specifies cloud managment driver and its argumanets.
This parameter specifies cloud management driver and its arguments.
``cloud_management`` is responsible for configuring connection to nodes
and contains arguments such as SSH username/password/key/proxy.

14
examples/uno.py → examples/example_api_01.py

@ -37,18 +37,18 @@ def main():
}
logging.info('# Create connection to the cloud')
destructor = os_faults.connect(cloud_config)
cloud_management = os_faults.connect(cloud_config)
logging.info('# Verify connection to the cloud')
destructor.verify()
cloud_management.verify()
# os_faults library operate with 2 types of objects:
# service - is software that runs in the cloud, e.g. keystone, mysql,
# rabbitmq, nova-api, glance-api
# nodes - nodes that host the cloud, e.g. hardware server with hostname
logging.info('# Get nodes where Keystone service runs')
service = destructor.get_service(name='keystone')
logging.info('# Get nodes where Keystone service is running')
service = cloud_management.get_service(name='keystone')
nodes = service.get_nodes()
logging.info('Nodes: %s', nodes)
@ -60,14 +60,14 @@ def main():
one.reset()
logging.info('# Get all nodes in the cloud')
nodes = destructor.get_nodes()
nodes = cloud_management.get_nodes()
logging.info('All cloud nodes: %s', nodes)
logging.info('# Reset all these nodes')
nodes.reset()
logging.info('# Get node by FQDN: node-2.domain.tld')
nodes = destructor.get_nodes(fqdns=['node-2.domain.tld'])
nodes = cloud_management.get_nodes(fqdns=['node-2.domain.tld'])
logging.info('Node node-2.domain.tld: %s', nodes)
logging.info('# Disable public network on node-2.domain.tld')
@ -77,7 +77,7 @@ def main():
nodes.connect(network_name='public')
logging.info('# Kill Glance API service on a single node')
service = destructor.get_service(name='glance-api')
service = cloud_management.get_service(name='glance-api')
nodes = service.get_nodes().pick()
service.kill(nodes)

8
examples/due.py → examples/example_api_02.py

@ -22,19 +22,19 @@ def main():
'driver': 'devstack',
'args': {
'address': 'devstack.local',
'username': 'developer',
'username': 'stack',
}
}
}
logging.info('# Create connection to the cloud')
destructor = os_faults.connect(cloud_config)
cloud_management = os_faults.connect(cloud_config)
logging.info('# Verify connection to the cloud')
destructor.verify()
cloud_management.verify()
logging.info('# Restart Keystone service on all nodes')
service = destructor.get_service(name='keystone')
service = cloud_management.get_service(name='keystone')
service.restart()

75
examples/example_api_03.py Normal file

@ -0,0 +1,75 @@
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import logging
import os_faults
def main():
cloud_config = {
'cloud_management': {
'driver': 'universal',
},
'node_discover': {
'driver': 'node_list',
'args': [
{
'ip': '192.168.5.127',
'auth': {
'username': 'root',
'private_key_file': 'openstack_key',
}
},
{
'ip': '192.168.5.128',
'auth': {
'username': 'root',
'private_key_file': 'openstack_key',
}
}
]
},
'services': {
'memcached': {
'driver': 'system_service',
'args': {
'service_name': 'memcached',
'grep': 'memcached',
}
}
},
'power_managements': [
{
'driver': 'libvirt',
'args': {
'connection_uri': 'qemu+unix:///system',
}
},
]
}
logging.info('# Create connection to the cloud')
cloud_management = os_faults.connect(cloud_config)
logging.info('# Verify connection to the cloud')
cloud_management.verify()
logging.info('# Kill Memcached service on all nodes')
service = cloud_management.get_service(name='memcached')
service.kill()
if __name__ == '__main__':
logging.basicConfig(format='%(asctime)s %(levelname)s %(message)s',
level=logging.INFO)
main()

2
examples/fuel_distribute_key.sh

@ -1,5 +1,7 @@
#!/bin/bash -x
# Distribute SSH keys on all nodes deployed by OpenStack Fuel
KEY_FILE_NAME="${HOME}/.ssh/os_faults"
HOST=${1:-fuel.local}
USERNAME=${2:-root}

6
examples/power_off_on_ipmi_node.py

@ -42,13 +42,13 @@ def main():
}
logging.info('Create connection to the cluster')
destructor = os_faults.connect(cloud_config)
cloud_management = os_faults.connect(cloud_config)
logging.info('Verify connection to the cluster')
destructor.verify()
cloud_management.verify()
logging.info('Get all cluster nodes')
nodes = destructor.get_nodes()
nodes = cloud_management.get_nodes()
logging.info('All cluster nodes: %s', nodes)
computes = nodes.filter(role='compute')

6
examples/power_off_on_vm_node.py

@ -36,13 +36,13 @@ def main():
}
logging.info('Create connection to the cluster')
destructor = os_faults.connect(cloud_config)
cloud_management = os_faults.connect(cloud_config)
logging.info('Verify connection to the cluster')
destructor.verify()
cloud_management.verify()
logging.info('Get all cluster nodes')
nodes = destructor.get_nodes()
nodes = cloud_management.get_nodes()
logging.info('All cluster nodes: %s', nodes)
logging.info('Pick and power off/on one of cluster nodes')

13
os_faults/__init__.py

@ -142,7 +142,7 @@ def _init_driver(params):
def connect(cloud_config=None, config_filename=None):
"""Connect to the cloud
"""Connects to the cloud
:param cloud_config: dict with cloud and power management params
:param config_filename: name of the file where to read config from
@ -224,13 +224,14 @@ def discover(cloud_config):
return cloud_config
def human_api(distractor, command):
"""Execute high-level text command with specified destructor
def human_api(cloud_management, command):
"""Executes a command written as English sentence
:param destructor: library instance as returned by :connect: function
:param cloud_management: library instance as returned by :connect:
function
:param command: text command
"""
human.execute(distractor, command)
human.execute(cloud_management, command)
def register_ansible_modules(paths):
@ -238,6 +239,6 @@ def register_ansible_modules(paths):
Allows to use custom ansible modules in NodeCollection.run_task method
:param path: list of paths to folders with ansible modules
:param paths: list of paths to folders with ansible modules
"""
executor.add_module_paths(paths)

8
os_faults/cmd/main.py

@ -48,8 +48,8 @@ def main(debug):
def verify(config):
"""Verify connection to the cloud"""
config = config or os_faults.get_default_config_file()
destructor = os_faults.connect(config_filename=config)
destructor.verify()
cloud_management = os_faults.connect(config_filename=config)
cloud_management.verify()
@main.command()
@ -71,9 +71,9 @@ def discover(config, output):
def nodes(config):
"""List cloud nodes"""
config = config or os_faults.get_default_config_file()
destructor = os_faults.connect(config_filename=config)
cloud_management = os_faults.connect(config_filename=config)
hosts = [{'ip': host.ip, 'mac': host.mac, 'fqdn': host.fqdn}
for host in destructor.get_nodes().hosts]
for host in cloud_management.get_nodes().hosts]
click.echo(yaml.safe_dump(hosts, default_flow_style=False), nl=False)

2
os_faults/drivers/cloud/devstack.py

@ -94,7 +94,7 @@ class ServiceInScreen(process.ServiceAsProcess):
class DevStackManagement(cloud_management.CloudManagement,
node_discover.NodeDiscover):
"""Devstack driver.
"""Devstack driver (**deprecated**).
This driver requires devstack installed in screen mode (USE_SCREEN=True).
Supports discovering of node MAC addresses.

4
os_faults/drivers/cloud/fuel.py

@ -44,9 +44,9 @@ class FuelNodeCollection(node_collection.NodeCollection):
class FuelManagement(cloud_management.CloudManagement,
node_discover.NodeDiscover):
"""Fuel driver.
"""OpenStack Fuel driver.
Cloud deployed by fuel. Supports discovering of slave nodes.
Cloud deployed by Fuel. Supports discovering of slave nodes.
**Example configuration:**

4
os_faults/drivers/cloud/universal.py

@ -36,7 +36,7 @@ class UniversalCloudManagement(cloud_management.CloudManagement,
parameters can be shared or overridden by corresponding parameters
from node discovery.
**Example single node configuration:**
**Example of single node configuration:**
.. code-block:: yaml
@ -52,7 +52,7 @@ class UniversalCloudManagement(cloud_management.CloudManagement,
iface: eth1
serial: 10
**Example multi-node configuration:**
**Example of multi-node configuration:**
Note that in this configuration a node discovery driver is required.

1
os_faults/drivers/nodes/node_list.py

@ -63,7 +63,6 @@ class NodeListDiscover(node_discover.NodeDiscover):
auth:
username: user1
password: secret1
sudo: False
jump:
host: 10.0.0.52
username: ubuntu

2
os_faults/drivers/services/linux.py

@ -15,7 +15,7 @@ from os_faults.drivers.services import process
class LinuxService(process.ServiceAsProcess):
"""Linux service
"""Linux service (**deprecated**, use `system_service` instead)
Service that is defined in init.d and can be controlled by `service`
CLI tool.

2
os_faults/drivers/services/systemd.py

@ -15,7 +15,7 @@ from os_faults.drivers.services import process
class SystemdService(process.ServiceAsProcess):
"""Systemd service.
"""Systemd service (**deprecated**, use `system_service` instead).
Service as Systemd unit and can be controlled by `systemctl` CLI tool.