Documentation update for get-goal-from-strategy

In this changeset, I updated the Watcher documentation to reflect
the changes that are introduced by this blueprint.

Partially Implements: blueprint get-goal-from-strategy

Change-Id: I40be39624097365220bf7d94cbe177bbf5bbe0ed

doc/source/architecture.rst

@@ -155,6 +155,7 @@ by the :ref:`Watcher API <archi_watcher_api_definition>` or the
 -  :ref:`Action plans <action_plan_definition>`
 -  :ref:`Actions <action_definition>`
 -  :ref:`Goals <goal_definition>`
+-  :ref:`Strategies <strategy_definition>`
 
 The Watcher domain being here "*optimization of some resources provided by an
 OpenStack system*".
@@ -196,8 +197,6 @@ Audit, the :ref:`Strategy <strategy_definition>` relies on two sets of data:
     which provides information about the past of the
     :ref:`Cluster <cluster_definition>`
 
-So far, only one :ref:`Strategy <strategy_definition>` can be associated to a
-given :ref:`Goal <goal_definition>` via the main Watcher configuration file.
 
 .. _data_model:
 
@@ -230,13 +229,15 @@ following parameters:
 
 -   A name
 -   A goal to achieve
+-   An optional strategy
 
 .. image:: ./images/sequence_create_audit_template.png
    :width: 100%
 
-The `Watcher API`_  just makes sure that the goal exists (i.e. it is declared
+The `Watcher API`_  makes sure that both the specified goal (mandatory) and
-in the Watcher configuration file) and stores a new audit template in the
+its associated strategy (optional) are registered inside the :ref:`Watcher
-:ref:`Watcher Database <watcher_database_definition>`.
+Database <watcher_database_definition>` before storing a new audit template in
+the :ref:`Watcher Database <watcher_database_definition>`.
 
 .. _sequence_diagrams_create_and_launch_audit:
 
@@ -260,12 +261,11 @@ the Audit in the
 The :ref:`Watcher Decision Engine <watcher_decision_engine_definition>` reads
 the Audit parameters from the
 :ref:`Watcher Database <watcher_database_definition>`. It instantiates the
-appropriate :ref:`Strategy <strategy_definition>` (using entry points)
+appropriate :ref:`strategy <strategy_definition>` (using entry points)
-associated to the :ref:`Goal <goal_definition>` of the
+given both the :ref:`goal <goal_definition>` and the strategy associated to the
-:ref:`Audit <audit_definition>` (it uses the information of the Watcher
+parent :ref:`audit template <audit_template_definition>` of the :ref:`Audit
-configuration file to find the mapping between the
+<audit_definition>`. If no strategy is associated to the audit template, the
-:ref:`Goal <goal_definition>` and the :ref:`Strategy <strategy_definition>`
+strategy is dynamically selected by the Decision Engine.
-python class).
 
 The :ref:`Watcher Decision Engine <watcher_decision_engine_definition>` also
 builds the :ref:`Cluster Data Model <cluster_data_model_definition>`. This

doc/source/deploy/configuration.rst

@@ -182,8 +182,6 @@ The configuration file is organized into the following sections:
 * ``[watcher_clients_auth]`` - Keystone auth configuration for clients
 * ``[watcher_applier]`` - Watcher Applier module configuration
 * ``[watcher_decision_engine]`` - Watcher Decision Engine module configuration
-* ``[watcher_goals]`` - Goals mapping configuration
-* ``[watcher_strategies]`` - Strategy configuration
 * ``[oslo_messaging_rabbit]`` - Oslo Messaging RabbitMQ driver configuration
 * ``[ceilometer_client]`` - Ceilometer client configuration
 * ``[cinder_client]`` - Cinder client configuration

doc/source/deploy/user-guide.rst

@@ -56,18 +56,39 @@ watcher binary without options.
 How do I run an audit of my cluster ?
 -------------------------------------
 
-First, you need to create an :ref:`audit template <audit_template_definition>`.
+First, you need to find the :ref:`goal <goal_definition>` you want to achieve:
-An :ref:`audit template <audit_template_definition>` defines an optimization
-:ref:`goal <goal_definition>` to achieve (i.e. the settings of your audit).
-This goal should be declared in the Watcher service configuration file
-**/etc/watcher/watcher.conf**.
 
 .. code:: bash
 
-  $ watcher audit-template-create my_first_audit DUMMY
+  $ watcher goal-list
 
-If you get "*You must provide a username via either --os-username or via
+.. note::
-env[OS_USERNAME]*" you may have to verify your credentials.
+
+    If you get "*You must provide a username via either --os-username or via
+    env[OS_USERNAME]*" you may have to verify your credentials.
+
+Then, you can create an :ref:`audit template <audit_template_definition>`.
+An :ref:`audit template <audit_template_definition>` defines an optimization
+:ref:`goal <goal_definition>` to achieve (i.e. the settings of your audit).
+
+.. code:: bash
+
+  $ watcher audit-template-create my_first_audit_template <your_goal_uuid>
+
+Although optional, you may want to actually set a specific strategy for your
+audit template. If so, you may can search of its UUID using the following
+command:
+
+.. code:: bash
+
+  $ watcher strategy-list --goal-uuid <your_goal_uuid>
+
+The command to create your audit template would then be:
+
+.. code:: bash
+
+  $ watcher audit-template-create my_first_audit_template <your_goal_uuid> \
+    --strategy-uuid <your_strategy_uuid>
 
 Then, you can create an audit. An audit is a request for optimizing your
 cluster depending on the specified :ref:`goal <goal_definition>`.

doc/source/dev/plugin/strategy-plugin.rst

@@ -15,7 +15,9 @@ plugin interface which gives anyone the ability to integrate an external
 strategy in order to make use of placement algorithms.
 
 This section gives some guidelines on how to implement and integrate custom
-strategies with Watcher.
+strategies with Watcher. If you wish to create a third-party package for your
+plugin, you can refer to our :ref:`documentation for third-party package
+creation <plugin-base_setup>`.
 
 
 Pre-requisites
@@ -26,64 +28,173 @@ configured so that it would provide you all the metrics you need to be able to
 use your strategy.
 
 
-Creating a new plugin
+Create a new plugin
-=====================
+===================
 
-First of all you have to:
+In order to create a new strategy, you have to:
 
-- Extend :py:class:`~.BaseStrategy`
+- Extend the :py:class:`~.UnclassifiedStrategy` class
-- Implement its :py:meth:`~.BaseStrategy.execute` method
+- Implement its :py:meth:`~.BaseStrategy.get_name` class method to return the
+  **unique** ID of the new strategy you want to create. This unique ID should
+  be the same as the name of :ref:`the entry point we will declare later on
+  <strategy_plugin_add_entrypoint>`.
+- Implement its :py:meth:`~.BaseStrategy.get_display_name` class method to
+  return the translated display name of the strategy you want to create.
+  Note: Do not use a variable to return the translated string so it can be
+  automatically collected by the translation tool.
+- Implement its :py:meth:`~.BaseStrategy.get_translatable_display_name`
+  class method to return the translation key (actually the english display
+  name) of your new strategy. The value return should be the same as the
+  string translated in :py:meth:`~.BaseStrategy.get_display_name`.
+- Implement its :py:meth:`~.BaseStrategy.execute` method to return the
+  solution you computed within your strategy.
 
-Here is an example showing how you can write a plugin called ``DummyStrategy``:
+Here is an example showing how you can write a plugin called ``NewStrategy``:
 
 .. code-block:: python
 
-    import uuid
+    import abc
 
-    class DummyStrategy(BaseStrategy):
+    import six
 
-        DEFAULT_NAME = "dummy"
+    from watcher._i18n import _
-        DEFAULT_DESCRIPTION = "Dummy Strategy"
+    from watcher.decision_engine.strategy.strategies import base
 
-        def __init__(self, name=DEFAULT_NAME, description=DEFAULT_DESCRIPTION):
-            super(DummyStrategy, self).__init__(name, description)
 
-        def execute(self, model):
+    class NewStrategy(base.UnclassifiedStrategy):
-            migration_type = 'live'
+
-            src_hypervisor = 'compute-host-1'
+        def __init__(self, osc=None):
-            dst_hypervisor = 'compute-host-2'
+            super(NewStrategy, self).__init__(osc)
-            instance_id = uuid.uuid4()
+
-            parameters = {'migration_type': migration_type,
+        def execute(self, original_model):
-                          'src_hypervisor': src_hypervisor,
+            self.solution.add_action(action_type="nop",
-                          'dst_hypervisor': dst_hypervisor}
-            self.solution.add_action(action_type="migration",
-                                     resource_id=instance_id,
                                      input_parameters=parameters)
             # Do some more stuff here ...
             return self.solution
 
+        @classmethod
+        def get_name(cls):
+            return "new_strategy"
+
+        @classmethod
+        def get_display_name(cls):
+            return _("New strategy")
+
+        @classmethod
+        def get_translatable_display_name(cls):
+            return "New strategy"
+
+
 As you can see in the above example, the :py:meth:`~.BaseStrategy.execute`
 method returns a :py:class:`~.BaseSolution` instance as required. This solution
 is what wraps the abstract set of actions the strategy recommends to you. This
 solution is then processed by a :ref:`planner <planner_definition>` to produce
-an action plan which shall contain the sequenced flow of actions to be
+an action plan which contains the sequenced flow of actions to be
 executed by the :ref:`Watcher Applier <watcher_applier_definition>`.
 
-Please note that your strategy class will be instantiated without any
+Please note that your strategy class will expect to find the same constructor
-parameter. Therefore, you should make sure not to make any of them required in
+signature as BaseStrategy to instantiate you strategy. Therefore, you should
-your ``__init__`` method.
+ensure that your ``__init__`` signature is identical to the
+:py:class:`~.BaseStrategy` one.
+
+
+Create a new goal
+=================
+
+As stated before, the ``NewStrategy`` class extends a class called
+:py:class:`~.UnclassifiedStrategy`. This class actually implements a set of
+abstract methods which are defined within the :py:class:`~.BaseStrategy` parent
+class.
+
+Once you are confident in your strategy plugin, the next step is now to
+classify your goal by assigning it a proper goal. To do so, you can either
+reuse existing goals defined in Watcher. As of now, four goal-oriented abstract
+classes are defined in Watcher:
+
+- :py:class:`~.UnclassifiedStrategy` which is the one I mentioned up until now.
+- :py:class:`~.DummyBaseStrategy` which is used by :py:class:`~.DummyStrategy`
+  for testing purposes.
+- :py:class:`~.ServerConsolidationBaseStrategy`
+- :py:class:`~.ThermalOptimizationBaseStrategy`
+
+If none of the above actually correspond to the goal your new strategy
+achieves, you can define a brand new one. To do so, you need to:
+
+- Extend the :py:class:`~.BaseStrategy` class to make your new goal-oriented
+  strategy abstract class :
+- Implement its :py:meth:`~.BaseStrategy.get_goal_name` class method to
+  return the **unique** ID of the goal you want to achieve.
+- Implement its :py:meth:`~.BaseStrategy.get_goal_display_name` class method
+  to return the translated display name of the goal you want to achieve.
+  Note: Do not use a variable to return the translated string so it can be
+  automatically collected by the translation tool.
+- Implement its :py:meth:`~.BaseStrategy.get_translatable_goal_display_name`
+  class method to return the goal translation key (actually the english
+  display name). The value return should be the same as the string translated
+  in :py:meth:`~.BaseStrategy.get_goal_display_name`.
+
+Here is an example showing how you can define a new ``NEW_GOAL`` goal and
+modify your ``NewStrategy`` plugin so it now achieves the latter:
+
+.. code-block:: python
+
+    import abc
+
+    import six
+
+    from watcher._i18n import _
+    from watcher.decision_engine.strategy.strategies import base
+
+    @six.add_metaclass(abc.ABCMeta)
+    class NewGoalBaseStrategy(base.BaseStrategy):
+
+        @classmethod
+        def get_goal_name(cls):
+            return "NEW_GOAL"
+
+        @classmethod
+        def get_goal_display_name(cls):
+            return _("New goal")
+
+        @classmethod
+        def get_translatable_goal_display_name(cls):
+            return "New goal"
+
+
+    class NewStrategy(NewGoalBaseStrategy):
+
+        def __init__(self, osc=None):
+            super(NewStrategy, self).__init__(osc)
+
+        def execute(self, original_model):
+            self.solution.add_action(action_type="nop",
+                                     input_parameters=parameters)
+            # Do some more stuff here ...
+            return self.solution
+
+        @classmethod
+        def get_name(cls):
+            return "new_strategy"
+
+        @classmethod
+        def get_display_name(cls):
+            return _("New strategy")
+
+        @classmethod
+        def get_translatable_display_name(cls):
+            return "New strategy"
 
 
 Abstract Plugin Class
 =====================
 
-Here below is the abstract :py:class:`~.BaseStrategy` class that every single
+Here below is the abstract :py:class:`~.BaseStrategy` class:
-strategy should implement:
 
 .. autoclass:: watcher.decision_engine.strategy.strategies.base.BaseStrategy
     :members:
     :noindex:
 
+.. _strategy_plugin_add_entrypoint:
 
 Add a new entry point
 =====================
@@ -93,7 +204,9 @@ strategy must be registered as a named entry point under the
 ``watcher_strategies`` entry point of your ``setup.py`` file. If you are using
 pbr_, this entry point should be placed in your ``setup.cfg`` file.
 
-The name you give to your entry point has to be unique.
+The name you give to your entry point has to be unique and should be the same
+as the value returned by the :py:meth:`~.BaseStrategy.get_id` class method of
+your strategy.
 
 Here below is how you would proceed to register ``DummyStrategy`` using pbr_:
 
@@ -101,7 +214,7 @@ Here below is how you would proceed to register ``DummyStrategy`` using pbr_:
 
     [entry_points]
     watcher_strategies =
-        dummy = thirdparty.dummy:DummyStrategy
+        dummy_strategy = thirdparty.dummy:DummyStrategy
 
 
 To get a better understanding on how to implement a more advanced strategy,
@@ -117,16 +230,10 @@ plugins when it is restarted. If a Python package containing a custom plugin is
 installed within the same environment as Watcher, Watcher will automatically
 make that plugin available for use.
 
-At this point, Watcher will use your new strategy if you reference it in the
+At this point, Watcher will scan and register inside the :ref:`Watcher Database
-``goals`` under the ``[watcher_goals]`` section of your ``watcher.conf``
+<watcher_database_definition>` all the strategies (alongside the goals they
-configuration file. For example, if you want to use a ``dummy`` strategy you
+should satisfy) you implemented upon restarting the :ref:`Watcher Decision
-just installed, you would have to associate it to a goal like this:
+Engine <watcher_decision_engine_definition>`.
-
-.. code-block:: ini
-
-    [watcher_goals]
-    goals = BALANCE_LOAD:basic,MINIMIZE_ENERGY_CONSUMPTION:dummy
-
 
 You should take care when installing strategy plugins. By their very nature,
 there are no guarantees that utilizing them as is will be supported, as
@@ -148,7 +255,6 @@ for various types of backends.  A list of the available backends is located
 here_. The Ceilosca project is a good example of how to create your own
 pluggable backend.
 
-
 Finally, if your strategy requires new metrics not covered by Ceilometer, you
 can add them through a Ceilometer `plugin`_.
 
@@ -191,7 +297,7 @@ Read usage metrics using the Watcher Cluster History Helper
 
 Here below is the abstract ``BaseClusterHistory`` class of the Helper.
 
-.. autoclass:: watcher.metrics_engine.cluster_history.api.BaseClusterHistory
+.. autoclass:: watcher.metrics_engine.cluster_history.base.BaseClusterHistory
     :members:
     :noindex:
 

doc/source/glossary.rst

@@ -99,14 +99,14 @@ The :ref:`Cluster <cluster_definition>` may be divided in one or several
 Cluster Data Model
 ==================
 
-.. watcher-term:: watcher.metrics_engine.cluster_model_collector.api
+.. watcher-term:: watcher.metrics_engine.cluster_model_collector.base
 
 .. _cluster_history_definition:
 
 Cluster History
 ===============
 
-.. watcher-term:: watcher.metrics_engine.cluster_history.api
+.. watcher-term:: watcher.metrics_engine.cluster_history.base
 
 .. _controller_node_definition:
 
@@ -223,8 +223,8 @@ measure of how much of the :ref:`Goal <goal_definition>` has been achieved in
 respect with constraints and :ref:`SLAs <sla_definition>` defined by the
 :ref:`Customer <customer_definition>`.
 
-The way efficacy is evaluated will depend on the
+The way efficacy is evaluated will depend on the :ref:`Goal <goal_definition>`
-:ref:`Goal <goal_definition>` to achieve.
+to achieve.
 
 Of course, the efficacy will be relevant only as long as the
 :ref:`Action Plan <action_plan_definition>` is relevant
@@ -323,7 +323,7 @@ Solution
 Strategy
 ========
 
-.. watcher-term::  watcher.decision_engine.strategy.strategies.base
+.. watcher-term::  watcher.api.controllers.v1.strategy
 
 .. _watcher_applier_definition:
 

doc/source/image_src/plantuml/sequence_create_audit_template.txt

@@ -2,15 +2,21 @@
 
 actor Administrator
 
-Administrator -> "Watcher CLI" : watcher audit-template-create <name> <goal>
+Administrator -> "Watcher CLI" : watcher audit-template-create <name> <goal> \
+[--strategy-uuid <strategy>]
 "Watcher CLI" -> "Watcher API" : POST audit_template(parameters)
 
-"Watcher API" -> "Watcher API" : make sure goal exist in configuration
+"Watcher API" -> "Watcher Database" : Request if goal exists in database
-"Watcher API" -> "Watcher Database" : create new audit_template in database
+"Watcher API" <-- "Watcher Database" : OK
 
-"Watcher API" <-- "Watcher Database" : new audit template uuid
+"Watcher API" -> "Watcher Database" : Request if strategy exists in database (if provided)
-"Watcher CLI" <-- "Watcher API" : return new audit template URL in HTTP Location Header
+"Watcher API" <-- "Watcher Database" : OK
-Administrator <-- "Watcher CLI" : new audit template uuid
+
+"Watcher API" -> "Watcher Database" : Create new audit_template in database
+"Watcher API" <-- "Watcher Database" : New audit template UUID
+
+"Watcher CLI" <-- "Watcher API" : Return new audit template URL in HTTP Location Header
+Administrator <-- "Watcher CLI" : New audit template UUID
 
 @enduml
 

doc/source/image_src/plantuml/sequence_launch_action_plan.txt

@@ -4,8 +4,8 @@ actor Administrator
 
 Administrator -> "Watcher CLI" : watcher action-plan-start <action_plan_uuid>
 
-"Watcher CLI" -> "Watcher API" : PATCH action_plan(state=TRIGGERED)
+"Watcher CLI" -> "Watcher API" : PATCH action_plan(state=PENDING)
-"Watcher API" -> "Watcher Database" : action_plan.state=TRIGGERED
+"Watcher API" -> "Watcher Database" : action_plan.state=PENDING
 
 "Watcher CLI" <-- "Watcher API" : HTTP 200
 

watcher/api/controllers/v1/strategy.py

@@ -14,6 +14,19 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+"""
+A :ref:`Strategy <strategy_definition>` is an algorithm implementation which is
+able to find a :ref:`Solution <solution_definition>` for a given
+:ref:`Goal <goal_definition>`.
+
+There may be several potential strategies which are able to achieve the same
+:ref:`Goal <goal_definition>`. This is why it is possible to configure which
+specific :ref:`Strategy <strategy_definition>` should be used for each goal.
+
+Some strategies may provide better optimization results but may take more time
+to find an optimal :ref:`Solution <solution_definition>`.
+"""
+
 from oslo_config import cfg
 
 import pecan

watcher/decision_engine/strategy/strategies/base.py

@@ -169,6 +169,29 @@ class DummyBaseStrategy(BaseStrategy):
         return "Dummy goal"
 
 
+@six.add_metaclass(abc.ABCMeta)
+class UnclassifiedStrategy(BaseStrategy):
+    """This base class is used to ease the development of new strategies
+
+    The goal defined within this strategy can be used to simplify the
+    documentation explaining how to implement a new strategy plugin by
+    ommitting the need for the strategy developer to define a goal straight
+    away.
+    """
+
+    @classmethod
+    def get_goal_name(cls):
+        return "UNCLASSIFIED"
+
+    @classmethod
+    def get_goal_display_name(cls):
+        return _("Unclassified")
+
+    @classmethod
+    def get_translatable_goal_display_name(cls):
+        return "Unclassified"
+
+
 @six.add_metaclass(abc.ABCMeta)
 class ServerConsolidationBaseStrategy(BaseStrategy):
 

watcher/decision_engine/strategy/strategies/vm_workload_consolidation.py

@@ -46,6 +46,7 @@ class VMWorkloadConsolidation(base.ServerConsolidationBaseStrategy):
 
     This strategy produces a solution resulting in more efficient
     utilization of cluster resources using following four phases:
+
     * Offload phase - handling over-utilized resources
     * Consolidation phase - handling under-utilized resources
     * Solution optimization - reducing number of migrations
@@ -58,7 +59,7 @@ class VMWorkloadConsolidation(base.ServerConsolidationBaseStrategy):
     If the cc equals 1 the full resource capacity may be used, cc
     values lower than 1 will lead to resource under utilization and
     values higher than 1 will lead to resource overbooking.
-    e.g. If targeted utilization is 80% of hypervisor capacity,
+    e.g. If targeted utilization is 80 percent of hypervisor capacity,
     the coefficient in the consolidation phase will be 0.8, but
     may any lower value in the offloading phase. The lower it gets
     the cluster will appear more released (distributed) for the
@@ -397,6 +398,7 @@ class VMWorkloadConsolidation(base.ServerConsolidationBaseStrategy):
         This is done by eliminating unnecessary or circular set of migrations
         which can be replaced by a more efficient solution.
         e.g.:
+
         * A->B, B->C => replace migrations A->B, B->C with
           a single migration A->C as both solution result in
           VM running on hypervisor C which can be achieved with
@@ -502,6 +504,7 @@ class VMWorkloadConsolidation(base.ServerConsolidationBaseStrategy):
         This strategy produces a solution resulting in more
         efficient utilization of cluster resources using following
         four phases:
+
         * Offload phase - handling over-utilized resources
         * Consolidation phase - handling under-utilized resources
         * Solution optimization - reducing number of migrations