further updates to plugins documentation

This should bring the PLUGINS.rst document up to date with the current
implementation in the code. There are still things to design and say
about external from tree support, as well as the 'settings' file,
however this covers the resources.sh.

Change-Id: Id7e2e7db4f5160d9019d76717db356518bfb5931

PLUGINS.rst

@@ -16,15 +16,16 @@ Proposed new basic flow:
   - run stack.sh to build the correct base environment
 - verify_base
   - for project in projects; do verify_project; done
-- resources-create
+- resources.sh create
+- resources.sh verify
 - shutdown
   - for project in projects; do shutdown; done
-- snapshot.sh pre_upgrade
-- resources-survive-shutdown
+- snapshot.sh pre_upgrade (NOT YET IMPLEMENTED)
+- resources.sh verify_noapi
 - upgrade ...
-- resources-survive-upgrade
+- resources.sh verify
 - verify_target
-- resources_cleanup
+- resources.sh destroy
 
 
 
@@ -38,13 +39,12 @@ Assuming the following tree in target projects::
          settings   - adds settings for the upgrade path
          upgrade.sh
          snapshot.sh - snapshots the state of the service, typically a
-            database dump
+            database dump (NOT YET IMPLEMENTED)
          from-juno/ - per release
          within-juno/
          from-kilo/
          within-kilo/
          resources.sh
-         verify.sh
 
 
 This same modular structure exists in the grenade tree with::
@@ -57,32 +57,26 @@ This same modular structure exists in the grenade tree with::
 resources.sh
 =================
 
-This is a script that's designed to be called in the following ways:
+resources.sh is a per-service resource create / verify / destroy
+interface. What a service does inside a script is up to them.
+
+You can assume your resource script will only be called if your
+service is running in an upgrade environment. The script should return
+zero on success for actions, and nonzero on failure.
+
+Calling Interface
+---------
+
+The following is the supported calling interface
 
 - resources.sh create
 
   creates a set of sample resources that should survive
-  upgrade. Script should exit with a non zero exit code if any
+  upgrade. Script should exit with a nonzero exit code if any
   resources could not be created.
 
   Example: create an instance in nova or a volume in cinder
 
-- resources.sh survived_shutdown
-
-  resource survival checks for after all services are shut down (the
-  in between phase for upgrades). Script should exit with a non zero
-  exit code if any resources were detected as offline.
-
-- resources.sh survived_upgrade
-
-  resource survival checks for after all services are started on the
-  new code revisions. Script should exit with a non zero exit code if
-  any resources no longer exist after the upgrade.
-
-- resources.sh cleanup
-
-  cleanup all resources
-
 - resources.sh verify
 
   verify that the resources were created. Services are running at this
@@ -105,8 +99,78 @@ This is a script that's designed to be called in the following ways:
   instance, or otherwise check its live-ness. With cinder, checking
   that the LVM volume exists and looks reasonable.
 
+- resources.sh destroy
+
+  Resource scripts should be responsible and cleanup all their
+  resources when asked to destroy.
+
+Calling Sequence
+----------------
+
+The calling sequence during a grenade run looks as follows:
+
+- # start old side
+- create (create will be called during the working old side)
+- verify
+- # shutdown all services
+- verify_noapi
+- # upgrade and start all services
+- verify
+- destroy
+
+The important thing to remember is verify/verify_noapi will be called
+multiple times, with multiple different versions of OpenStack. Those
+phases of the script must not be rerunnable multiple times.
+
+While create / destroy are only going to be called once in the current
+interface, bonus points for also making those idempotent for
+resiliancy in testing.
+
+Supporting Methods
+------------------
+
 In order to assist with the checks listed the following functions
 exist::
 
-  resource_data_add project key value
-  resource_data_get project key
+  resource_save project key value
+  resource_get project key
+
+This allow resource scripts to have memory, and keep track of things
+like the allocated IP addresses, IDs, and other non deterministic data
+that is returned from OpenStack API calls.
+
+Environment
+-----------
+
+Resource scripts get called in a specific environment already set:
+
+- TOP_DIR - will be set to the root of the devstack directory for the
+  BASE version of devstack incase this is needed to find files like a
+  working ``openrc``
+
+- GRENADE_DIR - the root directory of the grenade directory.
+
+The following snippet will give you access to both the grenade and
+TARGET devstack functions::
+
+  source $GRENADE_DIR/grenaderc
+  source $GRENADE_DIR/functions
+
+
+Best Practices
+--------------
+
+Do as many actions as non admin as possible. As early as you can in
+your resource script it's worth allocating a user/project for the
+script to run as. This ensures isolation against other scripts, and
+ensures that actions don't only work because admin gets to bypass
+safeties.
+
+Test side effects, not just API actions. The point of these resource
+survival scripts is to test that things created beyond the API / DB
+interaction still work later. Just testing that data can be stored /
+retrieved from the database isn't very interesting, and should be
+covered other places. The value in the resource scripts is these side
+effects. Actual VMs running, actual iscsi targets running, etc. And
+ensuring these things are not disrupted when the control plane is
+shifted out from under them.