Add Jinja evaluator

Allows to use Jinja instead of or along with YAQL for expression
evaluation.

 * Improved error reporting on API endpoints. Previously, Mistral API
   tend to mute important logs related to errors during YAML parsing
   or expression evaluation. The messages were shown in the http
   response, but would not appear in logs.

 * Renamed yaql_utils to evaluation_utils and added few more tests to
   ensure evaluation functions can be safely reused between Jinja and
   YAQL evaluators.

 * Updated action_v2 example to reflect similarities between YAQL and
   Jinja syntax.

Change-Id: Ie3cf8b4a6c068948d6dc051b12a02474689cf8a8
Implements: blueprint mistral-jinga-templates

doc/source/dsl/dsl_v2.rst

@@ -30,8 +30,8 @@ will be described in details below:
 Prerequisites
 -------------
 
-Mistral DSL takes advantage of
+Mistral DSL supports `YAQL <https://pypi.python.org/pypi/yaql/1.0.0>`__ and
-`YAQL <https://pypi.python.org/pypi/yaql/1.0.0>`__ expression language to
+`Jinja2 <http://jinja.pocoo.org/docs/dev/>`__ expression languages to
 reference workflow context variables and thereby implements passing data
 between workflow tasks. It's also referred to as Data Flow mechanism.
 YAQL is a simple but powerful query language that allows to extract
@@ -51,11 +51,12 @@ in the following sections of DSL:
 
 Mistral DSL is fully based on YAML and knowledge of YAML is a plus for
 better understanding of the material in this specification. It also
-takes advantage of YAQL query language to define expressions in workflow
+takes advantage of supported query languages to define expressions in workflow
 and action definitions.
 
 -  Yet Another Markup Language (YAML): http://yaml.org
 -  Yet Another Query Language (YAQL): https://pypi.python.org/pypi/yaql/1.0.0
+-  Jinja 2: http://jinja.pocoo.org/docs/dev/
 
 Workflows
 ---------
@@ -124,7 +125,7 @@ Common Workflow Attributes
 -  **description** - Arbitrary text containing workflow description. *Optional*.
 -  **input** - List defining required input parameter names and
    optionally their default values in a form "my_param: 123". *Optional*.
--  **output** - Any data structure arbitrarily containing YAQL
+-  **output** - Any data structure arbitrarily containing
    expressions that defines workflow output. May be nested. *Optional*.
 -  **task-defaults** - Default settings for some of task attributes
    defined at workflow level. *Optional*. Corresponding attribute
@@ -188,15 +189,15 @@ attributes:
    *Mutually exclusive with* **action**.
 -  **input** - Actual input parameter values of the task. *Optional*.
    Value of each parameter is a JSON-compliant type such as number,
-   string etc, dictionary or list. It can also be a YAQL expression to
+   string etc, dictionary or list. It can also be an expression to
    retrieve value from task context or any of the mentioned types
-   containing inline YAQL expressions (for example, string "<%
+   containing inline expressions (for example, string "<%
    $.movie_name %> is a cool movie!")
 -  **publish** - Dictionary of variables to publish to the workflow
    context. Any JSON-compatible data structure optionally containing
-   YAQL expression to select precisely what needs to be published.
+   expression to select precisely what needs to be published.
    Published variables will be accessible for downstream tasks via using
-   YAQL expressions. *Optional*.
+   expressions. *Optional*.
 -  **with-items** - If configured, it allows to run action or workflow
    associated with a task multiple times on a provided list of items.
    See `Processing collections using
@@ -278,10 +279,10 @@ Defines a pattern how task should be repeated in case of an error.
    repeated.
 -  **delay** - Defines a delay in seconds between subsequent task
    iterations.
--  **break-on** - Defines a YAQL expression that will break iteration
+-  **break-on** - Defines an expression that will break iteration
    loop if it evaluates to 'true'. If it fires then the task is
    considered error.
--  **continue-on** - Defines a YAQL expression that will continue iteration
+-  **continue-on** - Defines an expression that will continue iteration
    loop if it evaluates to 'true'. If it fires then the task is
    considered successful. If it evaluates to 'false' then policy will break the iteration.
 
@@ -293,7 +294,7 @@ Retry policy can also be configured on a single line as:
       action: my_action
       retry: count=10 delay=5 break-on=<% $.foo = 'bar' %>
 
-All parameter values for any policy can be defined as YAQL expressions.
+All parameter values for any policy can be defined as expressions.
 
 Simplified Input Syntax
 '''''''''''''''''''''''
@@ -407,7 +408,7 @@ Transitions with YAQL expressions
 '''''''''''''''''''''''''''''''''
 
 Task transitions can be determined by success/error/completeness of the
-previous tasks and also by additional YAQL guard expressions that can
+previous tasks and also by additional guard expressions that can
 access any data produced by upstream tasks. So in the example above task
 'create_vm' could also have a YAQL expression on transition to task
 'send_success_email' as follows:
@@ -420,8 +421,8 @@ access any data produced by upstream tasks. So in the example above task
        - send_success_email: <% $.vm_id != null %>
 
 And this would tell Mistral to run 'send_success_email' task only if
-'vm_id' variable published by task 'create_vm' is not empty. YAQL
+'vm_id' variable published by task 'create_vm' is not empty.
-expressions can also be applied to 'on-error' and 'on-complete'.
+Expressions can also be applied to 'on-error' and 'on-complete'.
 
 Fork
 ''''
@@ -475,7 +476,7 @@ run only if all upstream tasks (ones that lead to this task) are
 completed and corresponding conditions have triggered. Task A is
 considered an upstream task of Task B if Task A has Task B mentioned in
 any of its "on-success", "on-error" and "on-complete" clauses regardless
-of YAQL guard expressions.
+of guard expressions.
 
 Partial Join (join: 2)
 
@@ -945,14 +946,14 @@ Attributes
    used only for documenting purposes. Mistral now does not enforce
    actual input parameters to exactly correspond to this list. Based
    parameters will be calculated based on provided actual parameters
-   with using YAQL expressions so what's used in expressions implicitly
+   with using expressions so what's used in expressions implicitly
    define real input parameters. Dictionary of actual input parameters
-   is referenced in YAQL as '$.'. Redundant parameters will be simply
+   (expression context) is referenced as '$.' in YAQL and as '_.' in Jinja.
-   ignored.
+   Redundant parameters will be simply ignored.
 -  **output** - Any data structure defining how to calculate output of
    this action based on output of base action. It can optionally have
-   YAQL expressions to access properties of base action output
+   expressions to access properties of base action output through expression
-   referenced in YAQL as '$.'.
+   context.
 
 Workbooks
 ---------
@@ -1045,7 +1046,7 @@ Attributes
 Predefined Values/Functions in execution data context
 -----------------------------------------------------
 
-Using YAQL it is possible to use some predefined values in Mistral DSL.
+Using expressions it is possible to use some predefined values in Mistral DSL.
 
 -  **OpenStack context**
 -  **Task result**

mistral/exceptions.py

@@ -1,5 +1,6 @@
 # Copyright 2013 - Mirantis, Inc.
 # Copyright 2015 - StackStorm, Inc.
+# Copyright 2016 - Brocade Communications Systems, Inc.
 #
 #    Licensed under the Apache License, Version 2.0 (the "License");
 #    you may not use this file except in compliance with the License.
@@ -112,8 +113,15 @@ class DSLParsingException(MistralException):
     http_code = 400
 
 
-class YaqlGrammarException(DSLParsingException):
+class ExpressionGrammarException(DSLParsingException):
     http_code = 400
+
+
+class JinjaGrammarException(ExpressionGrammarException):
+    message = "Invalid grammar of Jinja expression"
+
+
+class YaqlGrammarException(ExpressionGrammarException):
     message = "Invalid grammar of YAQL expression"
 
 
@@ -124,8 +132,15 @@ class InvalidModelException(DSLParsingException):
 
 # Various common exceptions and errors.
 
-class YaqlEvaluationException(MistralException):
+class EvaluationException(MistralException):
     http_code = 400
+
+
+class JinjaEvaluationException(EvaluationException):
+    message = "Can not evaluate Jinja expression"
+
+
+class YaqlEvaluationException(EvaluationException):
     message = "Can not evaluate YAQL expression"
 
 

mistral/expressions/__init__.py

@@ -0,0 +1,103 @@
+# Copyright 2013 - Mirantis, Inc.
+# Copyright 2015 - StackStorm, Inc.
+# Copyright 2016 - Brocade Communications Systems, Inc.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License");
+#    you may not use this file except in compliance with the License.
+#    You may obtain a copy of the License at
+#
+#        http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS,
+#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#    See the License for the specific language governing permissions and
+#    limitations under the License.
+
+import copy
+
+from oslo_log import log as logging
+import six
+from stevedore import extension
+
+from mistral import exceptions as exc
+
+LOG = logging.getLogger(__name__)
+
+_mgr = extension.ExtensionManager(
+    namespace='mistral.expression.evaluators',
+    invoke_on_load=False
+)
+
+_evaluators = []
+patterns = {}
+
+for name in sorted(_mgr.names()):
+    evaluator = _mgr[name].plugin
+    _evaluators.append((name, evaluator))
+    patterns[name] = evaluator.find_expression_pattern.pattern
+
+
+def validate(expression):
+    LOG.debug("Validating expression [expression='%s']", expression)
+
+    if not isinstance(expression, six.string_types):
+        return
+
+    expression_found = None
+
+    for name, evaluator in _evaluators:
+        if evaluator.is_expression(expression):
+            if expression_found:
+                raise exc.ExpressionGrammarException(
+                    "The line already contains an expression of type '%s'. "
+                    "Mixing expression types in a single line is not allowed."
+                    % expression_found)
+
+            try:
+                evaluator.validate(expression)
+            except Exception:
+                raise
+            else:
+                expression_found = name
+
+
+def evaluate(expression, context):
+    for name, evaluator in _evaluators:
+        # Check if the passed value is expression so we don't need to do this
+        # every time on a caller side.
+        if (isinstance(expression, six.string_types) and
+                evaluator.is_expression(expression)):
+            return evaluator.evaluate(expression, context)
+
+    return expression
+
+
+def _evaluate_item(item, context):
+    if isinstance(item, six.string_types):
+        try:
+            return evaluate(item, context)
+        except AttributeError as e:
+            LOG.debug("Expression %s is not evaluated, [context=%s]: %s"
+                      % (item, context, e))
+            return item
+    else:
+        return evaluate_recursively(item, context)
+
+
+def evaluate_recursively(data, context):
+    data = copy.deepcopy(data)
+
+    if not context:
+        return data
+
+    if isinstance(data, dict):
+        for key in data:
+            data[key] = _evaluate_item(data[key], context)
+    elif isinstance(data, list):
+        for index, item in enumerate(data):
+            data[index] = _evaluate_item(item, context)
+    elif isinstance(data, six.string_types):
+        return _evaluate_item(data, context)
+
+    return data

mistral/expressions/base_expression.py

@@ -0,0 +1,55 @@
+# Copyright 2013 - Mirantis, Inc.
+# Copyright 2015 - StackStorm, Inc.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License");
+#    you may not use this file except in compliance with the License.
+#    You may obtain a copy of the License at
+#
+#        http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS,
+#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#    See the License for the specific language governing permissions and
+#    limitations under the License.
+
+import abc
+
+
+class Evaluator(object):
+    """Expression evaluator interface.
+
+    Having this interface gives the flexibility to change the actual expression
+    language used in Mistral DSL for conditions, output calculation etc.
+    """
+
+    @classmethod
+    @abc.abstractmethod
+    def validate(cls, expression):
+        """Parse and validates the expression.
+
+        :param expression: Expression string
+        :return: True if expression is valid
+        """
+        pass
+
+    @classmethod
+    @abc.abstractmethod
+    def evaluate(cls, expression, context):
+        """Evaluates the expression against the given data context.
+
+        :param expression: Expression string
+        :param context: Data context
+        :return: Expression result
+        """
+        pass
+
+    @classmethod
+    @abc.abstractmethod
+    def is_expression(cls, expression):
+        """Check expression string and decide whether it is expression or not.
+
+        :param expression: Expression string
+        :return: True if string is expression
+        """
+        pass

mistral/expressions/jinja_expression.py

@@ -0,0 +1,142 @@
+# Copyright 2016 - Brocade Communications Systems, Inc.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License");
+#    you may not use this file except in compliance with the License.
+#    You may obtain a copy of the License at
+#
+#        http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS,
+#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#    See the License for the specific language governing permissions and
+#    limitations under the License.
+
+import re
+
+import jinja2
+from jinja2 import parser as jinja_parse
+from oslo_log import log as logging
+import six
+
+from mistral import exceptions as exc
+from mistral.expressions.base_expression import Evaluator
+from mistral.utils import expression_utils
+
+
+LOG = logging.getLogger(__name__)
+
+JINJA_REGEXP = '({{(.*)?}})'
+JINJA_BLOCK_REGEXP = '({%(.*)?%})'
+
+_environment = jinja2.Environment(
+    undefined=jinja2.StrictUndefined,
+    trim_blocks=True,
+    lstrip_blocks=True
+)
+
+_filters = expression_utils.get_custom_functions()
+
+for name in _filters:
+    _environment.filters[name] = _filters[name]
+
+
+class JinjaEvaluator(Evaluator):
+    _env = _environment.overlay()
+
+    @classmethod
+    def validate(cls, expression):
+        LOG.debug(
+            "Validating Jinja expression [expression='%s']", expression)
+
+        if not isinstance(expression, six.string_types):
+            raise exc.JinjaEvaluationException("Unsupported type '%s'." %
+                                               type(expression))
+
+        try:
+            parser = jinja_parse.Parser(cls._env, expression, state='variable')
+            parser.parse_expression()
+        except jinja2.exceptions.TemplateError as e:
+            raise exc.JinjaGrammarException("Syntax error '%s'." %
+                                            str(e))
+
+    @classmethod
+    def evaluate(cls, expression, data_context):
+        LOG.debug(
+            "Evaluating Jinja expression [expression='%s', context=%s]"
+            % (expression, data_context)
+        )
+
+        opts = {
+            'undefined_to_none': False
+        }
+
+        ctx = expression_utils.get_jinja_context(data_context)
+
+        try:
+            result = cls._env.compile_expression(expression, **opts)(**ctx)
+
+            # For StrictUndefined values, UndefinedError only gets raised when
+            # the value is accessed, not when it gets created. The simplest way
+            # to access it is to try and cast it to string.
+            str(result)
+        except jinja2.exceptions.UndefinedError as e:
+            raise exc.JinjaEvaluationException("Undefined error '%s'." %
+                                               str(e))
+
+        LOG.debug("Jinja expression result: %s" % result)
+
+        return result
+
+    @classmethod
+    def is_expression(cls, s):
+        # The class should only be called from within InlineJinjaEvaluator. The
+        # return value prevents the class from being accidentally added as
+        # Extension
+        return False
+
+
+class InlineJinjaEvaluator(Evaluator):
+    # The regular expression for Jinja variables and blocks
+    find_expression_pattern = re.compile(JINJA_REGEXP)
+    find_block_pattern = re.compile(JINJA_BLOCK_REGEXP)
+
+    _env = _environment.overlay()
+
+    @classmethod
+    def validate(cls, expression):
+        LOG.debug(
+            "Validating Jinja expression [expression='%s']", expression)
+
+        if not isinstance(expression, six.string_types):
+            raise exc.JinjaEvaluationException("Unsupported type '%s'." %
+                                               type(expression))
+
+        try:
+            cls._env.parse(expression)
+        except jinja2.exceptions.TemplateError as e:
+            raise exc.JinjaGrammarException("Syntax error '%s'." %
+                                            str(e))
+
+    @classmethod
+    def evaluate(cls, expression, data_context):
+        LOG.debug(
+            "Evaluating Jinja expression [expression='%s', context=%s]"
+            % (expression, data_context)
+        )
+
+        patterns = cls.find_expression_pattern.findall(expression)
+        if patterns[0][0] == expression:
+            result = JinjaEvaluator.evaluate(patterns[0][1], data_context)
+        else:
+            ctx = expression_utils.get_jinja_context(data_context)
+            result = cls._env.from_string(expression).render(**ctx)
+
+            LOG.debug("Jinja expression result: %s" % result)
+
+        return result
+
+    @classmethod
+    def is_expression(cls, s):
+        return (cls.find_expression_pattern.search(s) or
+                cls.find_block_pattern.search(s))

mistral/expressions/yaql_expression.py

@@ -1,5 +1,6 @@
 # Copyright 2013 - Mirantis, Inc.
 # Copyright 2015 - StackStorm, Inc.
+# Copyright 2016 - Brocade Communications Systems, Inc.
 #
 #    Licensed under the Apache License, Version 2.0 (the "License");
 #    you may not use this file except in compliance with the License.
@@ -13,8 +14,6 @@
 #    See the License for the specific language governing permissions and
 #    limitations under the License.
 
-import abc
-import copy
 import inspect
 import re
 
@@ -24,50 +23,13 @@ from yaql.language import exceptions as yaql_exc
 from yaql.language import factory
 
 from mistral import exceptions as exc
-from mistral.utils import yaql_utils
+from mistral.expressions.base_expression import Evaluator
-
+from mistral.utils import expression_utils
 
 LOG = logging.getLogger(__name__)
 YAQL_ENGINE = factory.YaqlFactory().create()
 
-
+INLINE_YAQL_REGEXP = '<%.*?%>'
-class Evaluator(object):
-    """Expression evaluator interface.
-
-    Having this interface gives the flexibility to change the actual expression
-    language used in Mistral DSL for conditions, output calculation etc.
-    """
-
-    @classmethod
-    @abc.abstractmethod
-    def validate(cls, expression):
-        """Parse and validates the expression.
-
-        :param expression: Expression string
-        :return: True if expression is valid
-        """
-        pass
-
-    @classmethod
-    @abc.abstractmethod
-    def evaluate(cls, expression, context):
-        """Evaluates the expression against the given data context.
-
-        :param expression: Expression string
-        :param context: Data context
-        :return: Expression result
-        """
-        pass
-
-    @classmethod
-    @abc.abstractmethod
-    def is_expression(cls, expression):
-        """Check expression string and decide whether it is expression or not.
-
-        :param expression: Expression string
-        :return: True if string is expression
-        """
-        pass
 
 
 class YAQLEvaluator(Evaluator):
@@ -87,7 +49,7 @@ class YAQLEvaluator(Evaluator):
 
         try:
             result = YAQL_ENGINE(expression).evaluate(
-                context=yaql_utils.get_yaql_context(data_context)
+                context=expression_utils.get_yaql_context(data_context)
             )
         except (yaql_exc.YaqlException, KeyError, ValueError, TypeError) as e:
             raise exc.YaqlEvaluationException(
@@ -101,12 +63,9 @@ class YAQLEvaluator(Evaluator):
 
     @classmethod
     def is_expression(cls, s):
-        # TODO(rakhmerov): It should be generalized since it may not be YAQL.
+        # The class should not be used outside of InlineYAQLEvaluator since by
-        # Treat any string as a YAQL expression.
+        # convention, YAQL expression should always be wrapped in '<% %>'.
-        return isinstance(s, six.string_types)
+        return False
-
-
-INLINE_YAQL_REGEXP = '<%.*?%>'
 
 
 class InlineYAQLEvaluator(YAQLEvaluator):
@@ -155,56 +114,8 @@ class InlineYAQLEvaluator(YAQLEvaluator):
 
     @classmethod
     def is_expression(cls, s):
-        return s
+        return cls.find_expression_pattern.search(s)
 
     @classmethod
     def find_inline_expressions(cls, s):
         return cls.find_expression_pattern.findall(s)
-
-
-# TODO(rakhmerov): Make it configurable.
-_EVALUATOR = InlineYAQLEvaluator
-
-
-def validate(expression):
-    return _EVALUATOR.validate(expression)
-
-
-def evaluate(expression, context):
-    # Check if the passed value is expression so we don't need to do this
-    # every time on a caller side.
-    if (not isinstance(expression, six.string_types) or
-            not _EVALUATOR.is_expression(expression)):
-        return expression
-
-    return _EVALUATOR.evaluate(expression, context)
-
-
-def _evaluate_item(item, context):
-    if isinstance(item, six.string_types):
-        try:
-            return evaluate(item, context)
-        except AttributeError as e:
-            LOG.debug("Expression %s is not evaluated, [context=%s]: %s"
-                      % (item, context, e))
-            return item
-    else:
-        return evaluate_recursively(item, context)
-
-
-def evaluate_recursively(data, context):
-    data = copy.deepcopy(data)
-
-    if not context:
-        return data
-
-    if isinstance(data, dict):
-        for key in data:
-            data[key] = _evaluate_item(data[key], context)
-    elif isinstance(data, list):
-        for index, item in enumerate(data):
-            data[index] = _evaluate_item(item, context)
-    elif isinstance(data, six.string_types):
-        return _evaluate_item(data, context)
-
-    return data

mistral/tests/resources/action_jinja.yaml

@@ -0,0 +1,21 @@
+---
+version: "2.0"
+
+greeting:
+  description: "This action says 'Hello'"
+  tags: [hello]
+  base: std.echo
+  base-input:
+    output: 'Hello,  {{ _.name }}'
+  input:
+    - name
+  output:
+    string: '{{ _ }}'
+
+farewell:
+  base: std.echo
+  base-input:
+    output: 'Bye!'
+  output:
+    info: '{{ _ }}'
+

mistral/tests/resources/action_v2.yaml

@@ -10,12 +10,12 @@ greeting:
   input:
     - name
   output:
-    string: <% $.output %>
+    string: <% $ %>
 
 farewell:
   base: std.echo
   base-input:
     output: 'Bye!'
   output:
-    info: <% $.output %>
+    info: <% $ %>
 

mistral/tests/resources/wf_jinja.yaml

@@ -0,0 +1,34 @@
+---
+version: '2.0'
+
+wf:
+  type: direct
+
+  tasks:
+    hello:
+      action: std.echo output="Hello"
+      wait-before: 1
+      publish:
+        result: '{{ task("hello").result }}'
+
+wf1:
+  type: reverse
+  input:
+    - farewell
+
+  tasks:
+    addressee:
+      action: std.echo output="John"
+      publish:
+        name: '{{ task("addressee").result }}'
+
+    goodbye:
+      action: std.echo output="{{ _.farewell }}, {{ _.name }}"
+      requires: [addressee]
+
+wf2:
+  type: direct
+
+  tasks:
+    hello:
+      action: std.echo output="Hello"

mistral/tests/unit/expressions/test_jinja_expression.py

@@ -0,0 +1,397 @@
+# Copyright 2016 - Brocade Communications Systems, Inc.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License");
+#    you may not use this file except in compliance with the License.
+#    You may obtain a copy of the License at
+#
+#        http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS,
+#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#    See the License for the specific language governing permissions and
+#    limitations under the License.
+
+import mock
+
+from mistral import exceptions as exc
+from mistral.expressions import jinja_expression as expr
+from mistral.tests.unit import base
+from mistral import utils
+
+DATA = {
+    "server": {
+        "id": "03ea824a-aa24-4105-9131-66c48ae54acf",
+        "name": "cloud-fedora",
+        "status": "ACTIVE"
+    },
+    "status": "OK"
+}
+
+SERVERS = {
+    "servers": [
+        {'name': 'centos'},
+        {'name': 'ubuntu'},
+        {'name': 'fedora'}
+    ]
+}
+
+
+class JinjaEvaluatorTest(base.BaseTest):
+    def setUp(self):
+        super(JinjaEvaluatorTest, self).setUp()
+
+        self._evaluator = expr.JinjaEvaluator()
+
+    def test_expression_result(self):
+        res = self._evaluator.evaluate('_.server', DATA)
+        self.assertEqual({
+            'id': '03ea824a-aa24-4105-9131-66c48ae54acf',
+            'name': 'cloud-fedora',
+            'status': 'ACTIVE'
+        }, res)
+
+        res = self._evaluator.evaluate('_.server.id', DATA)
+        self.assertEqual('03ea824a-aa24-4105-9131-66c48ae54acf', res)
+
+        res = self._evaluator.evaluate("_.server.status == 'ACTIVE'", DATA)
+        self.assertTrue(res)
+
+    def test_wrong_expression(self):
+        res = self._evaluator.evaluate("_.status == 'Invalid value'", DATA)
+        self.assertFalse(res)
+
+        # One thing to note about Jinja is that by default it would not raise
+        # an exception on KeyError inside the expression, it will consider
+        # value to be None. Same with NameError, it won't return an original
+        # expression (which by itself seems confusing). Jinja allows us to
+        # change behavior in both cases by switching to StrictUndefined, but
+        # either one or the other will surely suffer.
+
+        self.assertRaises(
+            exc.JinjaEvaluationException,
+            self._evaluator.evaluate,
+            '_.wrong_key',
+            DATA
+        )
+
+        self.assertRaises(
+            exc.JinjaEvaluationException,
+            self._evaluator.evaluate,
+            'invalid_expression_string',
+            DATA
+        )
+
+    def test_select_result(self):
+        res = self._evaluator.evaluate(
+            '_.servers|selectattr("name", "equalto", "ubuntu")',
+            SERVERS
+        )
+        item = list(res)[0]
+        self.assertEqual({'name': 'ubuntu'}, item)
+
+    def test_function_string(self):
+        self.assertEqual('3', self._evaluator.evaluate('_|string', '3'))
+        self.assertEqual('3', self._evaluator.evaluate('_|string', 3))
+
+    def test_function_len(self):
+        self.assertEqual(3,
+                         self._evaluator.evaluate('_|length', 'hey'))
+        data = [{'some': 'thing'}]
+
+        self.assertEqual(
+            1,
+            self._evaluator.evaluate(
+                '_|selectattr("some", "equalto", "thing")|list|length',
+                data
+            )
+        )
+
+    def test_validate(self):
+        self._evaluator.validate('abc')
+        self._evaluator.validate('1')
+        self._evaluator.validate('1 + 2')
+        self._evaluator.validate('_.a1')
+        self._evaluator.validate('_.a1 * _.a2')
+
+    def test_validate_failed(self):
+        self.assertRaises(exc.JinjaGrammarException,
+                          self._evaluator.validate,
+                          '*')
+
+        self.assertRaises(exc.JinjaEvaluationException,
+                          self._evaluator.validate,
+                          [1, 2, 3])
+
+        self.assertRaises(exc.JinjaEvaluationException,
+                          self._evaluator.validate,
+                          {'a': 1})
+
+    def test_function_json_pp(self):
+        self.assertEqual('"3"', self._evaluator.evaluate('json_pp(_)', '3'))
+        self.assertEqual('3', self._evaluator.evaluate('json_pp(_)', 3))
+        self.assertEqual(
+            '[\n    1,\n    2\n]',
+            self._evaluator.evaluate('json_pp(_)', [1, 2])
+        )
+        self.assertEqual(
+            '{\n    "a": "b"\n}',
+            self._evaluator.evaluate('json_pp(_)', {'a': 'b'})
+        )
+        self.assertEqual(
+            '"Mistral\nis\nawesome"',
+            self._evaluator.evaluate(
+                'json_pp(_)', '\n'.join(['Mistral', 'is', 'awesome'])
+            )
+        )
+
+    def test_filter_json_pp(self):
+        self.assertEqual('"3"', self._evaluator.evaluate('_|json_pp', '3'))
+        self.assertEqual('3', self._evaluator.evaluate('_|json_pp', 3))
+        self.assertEqual(
+            '[\n    1,\n    2\n]',
+            self._evaluator.evaluate('_|json_pp', [1, 2])
+        )
+        self.assertEqual(
+            '{\n    "a": "b"\n}',
+            self._evaluator.evaluate('_|json_pp', {'a': 'b'})
+        )
+        self.assertEqual(
+            '"Mistral\nis\nawesome"',
+            self._evaluator.evaluate(
+                '_|json_pp', '\n'.join(['Mistral', 'is', 'awesome'])
+            )
+        )
+
+    def test_function_uuid(self):
+        uuid = self._evaluator.evaluate('uuid()', {})
+
+        self.assertTrue(utils.is_valid_uuid(uuid))
+
+    def test_filter_uuid(self):
+        uuid = self._evaluator.evaluate('_|uuid', '3')
+
+        self.assertTrue(utils.is_valid_uuid(uuid))
+
+    def test_function_env(self):
+        ctx = {'__env': 'some'}
+        self.assertEqual(ctx['__env'], self._evaluator.evaluate('env()', ctx))
+
+    def test_filter_env(self):
+        ctx = {'__env': 'some'}
+        self.assertEqual(ctx['__env'], self._evaluator.evaluate('_|env', ctx))
+
+    @mock.patch('mistral.db.v2.api.get_task_executions')
+    @mock.patch('mistral.workflow.data_flow.get_task_execution_result')
+    def test_filter_task_without_taskexecution(self, task_execution_result,
+                                               task_executions):
+        task = mock.MagicMock(return_value={})
+        task_executions.return_value = [task]
+        ctx = {
+            '__task_execution': None,
+            '__execution': {
+                'id': 'some'
+            }
+        }
+
+        result = self._evaluator.evaluate('_|task("some")', ctx)
+
+        self.assertEqual({
+            'id': task.id,
+            'name': task.name,
+            'published': task.published,
+            'result': task_execution_result(),
+            'spec': task.spec,
+            'state': task.state,
+            'state_info': task.state_info
+        }, result)
+
+    @mock.patch('mistral.db.v2.api.get_task_execution')
+    @mock.patch('mistral.workflow.data_flow.get_task_execution_result')
+    def test_filter_task_with_taskexecution(self, task_execution_result,
+                                            task_execution):
+        ctx = {
+            '__task_execution': {
+                'id': 'some',
+                'name': 'some'
+            }
+        }
+
+        result = self._evaluator.evaluate('_|task("some")', ctx)
+
+        self.assertEqual({
+            'id': task_execution().id,
+            'name': task_execution().name,
+            'published': task_execution().published,
+            'result': task_execution_result(),
+            'spec': task_execution().spec,
+            'state': task_execution().state,
+            'state_info': task_execution().state_info
+        }, result)
+
+    @mock.patch('mistral.db.v2.api.get_task_execution')
+    @mock.patch('mistral.workflow.data_flow.get_task_execution_result')
+    def test_function_task(self, task_execution_result, task_execution):
+        ctx = {
+            '__task_execution': {
+                'id': 'some',
+                'name': 'some'
+            }
+        }
+
+        result = self._evaluator.evaluate('task("some")', ctx)
+
+        self.assertEqual({
+            'id': task_execution().id,
+            'name': task_execution().name,
+            'published': task_execution().published,
+            'result': task_execution_result(),
+            'spec': task_execution().spec,
+            'state': task_execution().state,
+            'state_info': task_execution().state_info
+        }, result)
+
+    @mock.patch('mistral.db.v2.api.get_workflow_execution')
+    def test_filter_execution(self, workflow_execution):
+        wf_ex = mock.MagicMock(return_value={})
+        workflow_execution.return_value = wf_ex
+        ctx = {
+            '__execution': {
+                'id': 'some'
+            }
+        }
+
+        result = self._evaluator.evaluate('_|execution', ctx)
+
+        self.assertEqual({
+            'id': wf_ex.id,
+            'name': wf_ex.name,
+            'spec': wf_ex.spec,
+            'input': wf_ex.input,
+            'params': wf_ex.params
+        }, result)
+
+    @mock.patch('mistral.db.v2.api.get_workflow_execution')
+    def test_function_execution(self, workflow_execution):
+        wf_ex = mock.MagicMock(return_value={})
+        workflow_execution.return_value = wf_ex
+        ctx = {
+            '__execution': {
+                'id': 'some'
+            }
+        }
+
+        result = self._evaluator.evaluate('execution()', ctx)
+
+        self.assertEqual({
+            'id': wf_ex.id,
+            'name': wf_ex.name,
+            'spec': wf_ex.spec,
+            'input': wf_ex.input,
+            'params': wf_ex.params
+        }, result)
+
+
+class InlineJinjaEvaluatorTest(base.BaseTest):
+    def setUp(self):
+        super(InlineJinjaEvaluatorTest, self).setUp()
+
+        self._evaluator = expr.InlineJinjaEvaluator()
+
+    def test_multiple_placeholders(self):
+        expr_str = """
+            Statistics for tenant "{{ _.project_id }}"
+
+            Number of virtual machines: {{ _.vm_count }}
+            Number of active virtual machines: {{ _.active_vm_count }}
+            Number of networks: {{ _.net_count }}
+
+            -- Sincerely, Mistral Team.
+        """
+
+        result = self._evaluator.evaluate(
+            expr_str,
+            {
+                'project_id': '1-2-3-4',
+                'vm_count': 28,
+                'active_vm_count': 0,
+                'net_count': 1
+            }
+        )
+
+        expected_result = """
+            Statistics for tenant "1-2-3-4"
+
+            Number of virtual machines: 28
+            Number of active virtual machines: 0
+            Number of networks: 1
+
+            -- Sincerely, Mistral Team.
+        """
+
+        self.assertEqual(expected_result, result)
+
+    def test_block_placeholders(self):
+        expr_str = """
+            Statistics for tenant "{{ _.project_id }}"
+
+            Number of virtual machines: {{ _.vm_count }}
+            {% if _.active_vm_count %}
+            Number of active virtual machines: {{ _.active_vm_count }}
+            {% endif %}
+            Number of networks: {{ _.net_count }}
+
+            -- Sincerely, Mistral Team.
+        """
+
+        result = self._evaluator.evaluate(
+            expr_str,
+            {
+                'project_id': '1-2-3-4',
+                'vm_count': 28,
+                'active_vm_count': 0,
+                'net_count': 1
+            }
+        )
+
+        expected_result = """
+            Statistics for tenant "1-2-3-4"
+
+            Number of virtual machines: 28
+            Number of networks: 1
+
+            -- Sincerely, Mistral Team.
+        """
+
+        self.assertEqual(expected_result, result)
+
+    def test_single_value_casting(self):
+        self.assertEqual(3, self._evaluator.evaluate('{{ _ }}', 3))
+        self.assertEqual('33', self._evaluator.evaluate('{{ _ }}{{ _ }}', 3))
+
+    def test_function_string(self):
+        self.assertEqual('3', self._evaluator.evaluate('{{ _|string }}', '3'))
+        self.assertEqual('3', self._evaluator.evaluate('{{ _|string }}', 3))
+
+    def test_validate(self):
+        self._evaluator.validate('There is no expression.')
+        self._evaluator.validate('{{ abc }}')
+        self._evaluator.validate('{{ 1 }}')
+        self._evaluator.validate('{{ 1 + 2 }}')
+        self._evaluator.validate('{{ _.a1 }}')
+        self._evaluator.validate('{{ _.a1 * _.a2 }}')
+        self._evaluator.validate('{{ _.a1 }} is {{ _.a2 }}')
+        self._evaluator.validate('The value is {{ _.a1 }}.')
+
+    def test_validate_failed(self):
+        self.assertRaises(exc.JinjaGrammarException,
+                          self._evaluator.validate,
+                          'The value is {{ * }}.')
+
+        self.assertRaises(exc.JinjaEvaluationException,
+                          self._evaluator.validate,
+                          [1, 2, 3])
+
+        self.assertRaises(exc.JinjaEvaluationException,
+                          self._evaluator.validate,
+                          {'a': 1})

mistral/tests/unit/expressions/test_yaql_expression.py

@@ -0,0 +1,213 @@
+# Copyright 2013 - Mirantis, Inc.
+# Copyright 2015 - StackStorm, Inc.
+# Copyright 2016 - Brocade Communications Systems, Inc.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License");
+#    you may not use this file except in compliance with the License.
+#    You may obtain a copy of the License at
+#
+#        http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS,
+#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#    See the License for the specific language governing permissions and
+#    limitations under the License.
+
+from mistral import exceptions as exc
+from mistral.expressions import yaql_expression as expr
+from mistral.tests.unit import base
+from mistral import utils
+
+DATA = {
+    "server": {
+        "id": "03ea824a-aa24-4105-9131-66c48ae54acf",
+        "name": "cloud-fedora",
+        "status": "ACTIVE"
+    },
+    "status": "OK"
+}
+
+SERVERS = {
+    "servers": [
+        {'name': 'centos'},
+        {'name': 'ubuntu'},
+        {'name': 'fedora'}
+    ]
+}
+
+
+class YaqlEvaluatorTest(base.BaseTest):
+    def setUp(self):
+        super(YaqlEvaluatorTest, self).setUp()
+
+        self._evaluator = expr.YAQLEvaluator()
+
+    def test_expression_result(self):
+        res = self._evaluator.evaluate('$.server', DATA)
+        self.assertEqual({
+            'id': "03ea824a-aa24-4105-9131-66c48ae54acf",
+            'name': 'cloud-fedora',
+            'status': 'ACTIVE'
+        }, res)
+
+        res = self._evaluator.evaluate('$.server.id', DATA)
+        self.assertEqual('03ea824a-aa24-4105-9131-66c48ae54acf', res)
+
+        res = self._evaluator.evaluate("$.server.status = 'ACTIVE'", DATA)
+        self.assertTrue(res)
+
+    def test_wrong_expression(self):
+        res = self._evaluator.evaluate("$.status = 'Invalid value'", DATA)
+        self.assertFalse(res)
+
+        self.assertRaises(
+            exc.YaqlEvaluationException,
+            self._evaluator.evaluate,
+            '$.wrong_key',
+            DATA
+        )
+
+        expression_str = 'invalid_expression_string'
+        res = self._evaluator.evaluate(expression_str, DATA)
+        self.assertEqual(expression_str, res)
+
+    def test_select_result(self):
+        res = self._evaluator.evaluate(
+            '$.servers.where($.name = ubuntu)',
+            SERVERS
+        )
+        item = list(res)[0]
+        self.assertEqual({'name': 'ubuntu'}, item)
+
+    def test_function_string(self):
+        self.assertEqual('3', self._evaluator.evaluate('str($)', '3'))
+        self.assertEqual('3', self._evaluator.evaluate('str($)', 3))
+
+    def test_function_len(self):
+        self.assertEqual(3, self._evaluator.evaluate('len($)', 'hey'))
+        data = [{'some': 'thing'}]
+
+        self.assertEqual(
+            1,
+            self._evaluator.evaluate('$.where($.some = thing).len()', data)
+        )
+
+    def test_validate(self):
+        self._evaluator.validate('abc')
+        self._evaluator.validate('1')
+        self._evaluator.validate('1 + 2')
+        self._evaluator.validate('$.a1')
+        self._evaluator.validate('$.a1 * $.a2')
+
+    def test_validate_failed(self):
+        self.assertRaises(exc.YaqlGrammarException,
+                          self._evaluator.validate,
+                          '*')
+
+        self.assertRaises(exc.YaqlGrammarException,
+                          self._evaluator.validate,
+                          [1, 2, 3])
+
+        self.assertRaises(exc.YaqlGrammarException,
+                          self._evaluator.validate,
+                          {'a': 1})
+
+    def test_function_json_pp(self):
+        self.assertEqual('"3"', self._evaluator.evaluate('json_pp($)', '3'))
+        self.assertEqual('3', self._evaluator.evaluate('json_pp($)', 3))
+        self.assertEqual(
+            '[\n    1,\n    2\n]',
+            self._evaluator.evaluate('json_pp($)', [1, 2])
+        )
+        self.assertEqual(
+            '{\n    "a": "b"\n}',
+            self._evaluator.evaluate('json_pp($)', {'a': 'b'})
+        )
+        self.assertEqual(
+            '"Mistral\nis\nawesome"',
+            self._evaluator.evaluate(
+                'json_pp($)', '\n'.join(['Mistral', 'is', 'awesome'])
+            )
+        )
+
+    def test_function_uuid(self):
+        uuid = self._evaluator.evaluate('uuid()', {})
+
+        self.assertTrue(utils.is_valid_uuid(uuid))
+
+    def test_function_env(self):
+        ctx = {'__env': 'some'}
+
+        self.assertEqual(ctx['__env'], self._evaluator.evaluate('env()', ctx))
+
+
+class InlineYAQLEvaluatorTest(base.BaseTest):
+    def setUp(self):
+        super(InlineYAQLEvaluatorTest, self).setUp()
+
+        self._evaluator = expr.InlineYAQLEvaluator()
+
+    def test_multiple_placeholders(self):
+        expr_str = """
+            Statistics for tenant "<% $.project_id %>"
+
+            Number of virtual machines: <% $.vm_count %>
+            Number of active virtual machines: <% $.active_vm_count %>
+            Number of networks: <% $.net_count %>
+
+            -- Sincerely, Mistral Team.
+        """
+
+        result = self._evaluator.evaluate(
+            expr_str,
+            {
+                'project_id': '1-2-3-4',
+                'vm_count': 28,
+                'active_vm_count': 0,
+                'net_count': 1
+            }
+        )
+
+        expected_result = """
+            Statistics for tenant "1-2-3-4"
+
+            Number of virtual machines: 28
+            Number of active virtual machines: 0
+            Number of networks: 1
+
+            -- Sincerely, Mistral Team.
+        """
+
+        self.assertEqual(expected_result, result)
+
+    def test_single_value_casting(self):
+        self.assertEqual(3, self._evaluator.evaluate('<% $ %>', 3))
+        self.assertEqual('33', self._evaluator.evaluate('<% $ %><% $ %>', 3))
+
+    def test_function_string(self):
+        self.assertEqual('3', self._evaluator.evaluate('<% str($) %>', '3'))
+        self.assertEqual('3', self._evaluator.evaluate('<% str($) %>', 3))
+
+    def test_validate(self):
+        self._evaluator.validate('There is no expression.')
+        self._evaluator.validate('<% abc %>')
+        self._evaluator.validate('<% 1 %>')
+        self._evaluator.validate('<% 1 + 2 %>')
+        self._evaluator.validate('<% $.a1 %>')
+        self._evaluator.validate('<% $.a1 * $.a2 %>')
+        self._evaluator.validate('<% $.a1 %> is <% $.a2 %>')
+        self._evaluator.validate('The value is <% $.a1 %>.')
+
+    def test_validate_failed(self):
+        self.assertRaises(exc.YaqlGrammarException,
+                          self._evaluator.validate,
+                          'The value is <% * %>.')
+
+        self.assertRaises(exc.YaqlEvaluationException,
+                          self._evaluator.validate,
+                          [1, 2, 3])
+
+        self.assertRaises(exc.YaqlEvaluationException,
+                          self._evaluator.validate,
+                          {'a': 1})

mistral/tests/unit/test_expressions.py

@@ -1,5 +1,6 @@
 # Copyright 2013 - Mirantis, Inc.
 # Copyright 2015 - StackStorm, Inc.
+# Copyright 2016 - Brocade Communications Systems, Inc.
 #
 #    Licensed under the Apache License, Version 2.0 (the "License");
 #    you may not use this file except in compliance with the License.
@@ -35,168 +36,6 @@ SERVERS = {
 }
 
 
-class YaqlEvaluatorTest(base.BaseTest):
-    def setUp(self):
-        super(YaqlEvaluatorTest, self).setUp()
-
-        self._evaluator = expr.YAQLEvaluator()
-
-    def test_expression_result(self):
-        res = self._evaluator.evaluate('$.server', DATA)
-        self.assertEqual({
-            'id': "03ea824a-aa24-4105-9131-66c48ae54acf",
-            'name': 'cloud-fedora',
-            'status': 'ACTIVE'
-        }, res)
-
-        res = self._evaluator.evaluate('$.server.id', DATA)
-        self.assertEqual('03ea824a-aa24-4105-9131-66c48ae54acf', res)
-
-        res = self._evaluator.evaluate("$.server.status = 'ACTIVE'", DATA)
-        self.assertTrue(res)
-
-    def test_wrong_expression(self):
-        res = self._evaluator.evaluate("$.status = 'Invalid value'", DATA)
-        self.assertFalse(res)
-
-        self.assertRaises(
-            exc.YaqlEvaluationException,
-            self._evaluator.evaluate,
-            '$.wrong_key',
-            DATA
-        )
-
-        expression_str = 'invalid_expression_string'
-        res = self._evaluator.evaluate(expression_str, DATA)
-        self.assertEqual(expression_str, res)
-
-    def test_select_result(self):
-        res = self._evaluator.evaluate(
-            '$.servers.where($.name = ubuntu)',
-            SERVERS
-        )
-        item = list(res)[0]
-        self.assertEqual({'name': 'ubuntu'}, item)
-
-    def test_function_string(self):
-        self.assertEqual('3', self._evaluator.evaluate('str($)', '3'))
-        self.assertEqual('3', self._evaluator.evaluate('str($)', 3))
-
-    def test_function_len(self):
-        self.assertEqual(3, self._evaluator.evaluate('len($)', 'hey'))
-        data = [{'some': 'thing'}]
-
-        self.assertEqual(
-            1,
-            self._evaluator.evaluate('$.where($.some = thing).len()', data)
-        )
-
-    def test_validate(self):
-        self._evaluator.validate('abc')
-        self._evaluator.validate('1')
-        self._evaluator.validate('1 + 2')
-        self._evaluator.validate('$.a1')
-        self._evaluator.validate('$.a1 * $.a2')
-
-    def test_validate_failed(self):
-        self.assertRaises(exc.YaqlGrammarException,
-                          self._evaluator.validate,
-                          '*')
-
-        self.assertRaises(exc.YaqlGrammarException,
-                          self._evaluator.validate,
-                          [1, 2, 3])
-
-        self.assertRaises(exc.YaqlGrammarException,
-                          self._evaluator.validate,
-                          {'a': 1})
-
-    def test_json_pp(self):
-        self.assertEqual('"3"', self._evaluator.evaluate('json_pp($)', '3'))
-        self.assertEqual('3', self._evaluator.evaluate('json_pp($)', 3))
-        self.assertEqual(
-            '[\n    1,\n    2\n]',
-            self._evaluator.evaluate('json_pp($)', [1, 2])
-        )
-        self.assertEqual(
-            '{\n    "a": "b"\n}',
-            self._evaluator.evaluate('json_pp($)', {'a': 'b'})
-        )
-        self.assertEqual(
-            '"Mistral\nis\nawesome"',
-            self._evaluator.evaluate(
-                'json_pp($)', '\n'.join(['Mistral', 'is', 'awesome'])
-            )
-        )
-
-
-class InlineYAQLEvaluatorTest(base.BaseTest):
-    def setUp(self):
-        super(InlineYAQLEvaluatorTest, self).setUp()
-
-        self._evaluator = expr.InlineYAQLEvaluator()
-
-    def test_multiple_placeholders(self):
-        expr_str = """
-            Statistics for tenant "<% $.project_id %>"
-
-            Number of virtual machines: <% $.vm_count %>
-            Number of active virtual machines: <% $.active_vm_count %>
-            Number of networks: <% $.net_count %>
-
-            -- Sincerely, Mistral Team.
-        """
-
-        result = self._evaluator.evaluate(
-            expr_str,
-            {
-                'project_id': '1-2-3-4',
-                'vm_count': 28,
-                'active_vm_count': 0,
-                'net_count': 1
-            }
-        )
-
-        expected_result = """
-            Statistics for tenant "1-2-3-4"
-
-            Number of virtual machines: 28
-            Number of active virtual machines: 0
-            Number of networks: 1
-
-            -- Sincerely, Mistral Team.
-        """
-
-        self.assertEqual(expected_result, result)
-
-    def test_function_string(self):
-        self.assertEqual('3', self._evaluator.evaluate('<% str($) %>', '3'))
-        self.assertEqual('3', self._evaluator.evaluate('<% str($) %>', 3))
-
-    def test_validate(self):
-        self._evaluator.validate('There is no expression.')
-        self._evaluator.validate('<% abc %>')
-        self._evaluator.validate('<% 1 %>')
-        self._evaluator.validate('<% 1 + 2 %>')
-        self._evaluator.validate('<% $.a1 %>')
-        self._evaluator.validate('<% $.a1 * $.a2 %>')
-        self._evaluator.validate('<% $.a1 %> is <% $.a2 %>')
-        self._evaluator.validate('The value is <% $.a1 %>.')
-
-    def test_validate_failed(self):
-        self.assertRaises(exc.YaqlGrammarException,
-                          self._evaluator.validate,
-                          'The value is <% * %>.')
-
-        self.assertRaises(exc.YaqlEvaluationException,
-                          self._evaluator.validate,
-                          [1, 2, 3])
-
-        self.assertRaises(exc.YaqlEvaluationException,
-                          self._evaluator.validate,
-                          {'a': 1})
-
-
 class ExpressionsTest(base.BaseTest):
     def test_evaluate_complex_expressions(self):
         data = {
@@ -327,3 +166,26 @@ class ExpressionsTest(base.BaseTest):
         expected = 'mysql://admin:secrete@vm1234.example.com/test'
 
         self.assertEqual(expected, applied['conn'])
+
+    def test_validate_jinja_with_yaql_context(self):
+        self.assertRaises(exc.JinjaGrammarException,
+                          expr.validate,
+                          '{{ $ }}')
+
+    def test_validate_mixing_jinja_and_yaql(self):
+        self.assertRaises(exc.ExpressionGrammarException,
+                          expr.validate,
+                          '<% $.a %>{{ _.a }}')
+
+        self.assertRaises(exc.ExpressionGrammarException,
+                          expr.validate,
+                          '{{ _.a }}<% $.a %>')
+
+    def test_evaluate_mixing_jinja_and_yaql(self):
+        actual = expr.evaluate('<% $.a %>{{ _.a }}', {'a': 'b'})
+
+        self.assertEqual('<% $.a %>b', actual)
+
+        actual = expr.evaluate('{{ _.a }}<% $.a %>', {'a': 'b'})
+
+        self.assertEqual('b<% $.a %>', actual)

mistral/tests/unit/workbook/v2/test_actions.py

@@ -1,4 +1,5 @@
 # Copyright 2015 - StackStorm, Inc.
+# Copyright 2016 - Brocade Communications Systems, Inc.
 #
 #    Licensed under the Apache License, Version 2.0 (the "License");
 #    you may not use this file except in compliance with the License.
@@ -37,7 +38,10 @@ class ActionSpecValidation(base.WorkbookSpecValidationTestCase):
             ({'actions': {'a1': {'base': 'std.echo output="foo"'}}}, False),
             ({'actions': {'a1': {'base': 'std.echo output="<% $.x %>"'}}},
              False),
-            ({'actions': {'a1': {'base': 'std.echo output="<% * %>"'}}}, True)
+            ({'actions': {'a1': {'base': 'std.echo output="<% * %>"'}}}, True),
+            ({'actions': {'a1': {'base': 'std.echo output="{{ _.x }}"'}}},
+             False),
+            ({'actions': {'a1': {'base': 'std.echo output="{{ * }}"'}}}, True)
         ]
 
         for actions, expect_error in tests:
@@ -49,7 +53,9 @@ class ActionSpecValidation(base.WorkbookSpecValidationTestCase):
             ({'base-input': {}}, True),
             ({'base-input': None}, True),
             ({'base-input': {'k1': 'v1', 'k2': '<% $.v2 %>'}}, False),
-            ({'base-input': {'k1': 'v1', 'k2': '<% * %>'}}, True)
+            ({'base-input': {'k1': 'v1', 'k2': '<% * %>'}}, True),
+            ({'base-input': {'k1': 'v1', 'k2': '{{ _.v2 }}'}}, False),
+            ({'base-input': {'k1': 'v1', 'k2': '{{ * }}'}}, True)
         ]
 
         actions = {
@@ -100,6 +106,8 @@ class ActionSpecValidation(base.WorkbookSpecValidationTestCase):
             ({'output': 'foobar'}, False),
             ({'output': '<% $.x %>'}, False),
             ({'output': '<% * %>'}, True),
+            ({'output': '{{ _.x }}'}, False),
+            ({'output': '{{ * }}'}, True),
             ({'output': ['v1']}, False),
             ({'output': {'k1': 'v1'}}, False)
         ]

mistral/tests/unit/workbook/v2/test_tasks.py

@@ -1,5 +1,6 @@
 # Copyright 2015 - Huawei Technologies Co. Ltd
 # Copyright 2015 - StackStorm, Inc.
+# Copyright 2016 - Brocade Communications Systems, Inc.
 #
 #    Licensed under the Apache License, Version 2.0 (the "License");
 #    you may not use this file except in compliance with the License.
@@ -55,12 +56,18 @@ class TaskSpecValidation(v2_base.WorkflowSpecValidationTestCase):
             ({'action': 'std.http url=<% $.url %>'}, False),
             ({'action': 'std.http url=<% $.url %> timeout=<% $.t %>'}, False),
             ({'action': 'std.http url=<% * %>'}, True),
+            ({'action': 'std.http url={{ _.url }}'}, False),
+            ({'action': 'std.http url={{ _.url }} timeout={{ _.t }}'}, False),
+            ({'action': 'std.http url={{ $ }}'}, True),
             ({'workflow': 'test.wf'}, False),
             ({'workflow': 'test.wf k1="v1"'}, False),
             ({'workflow': 'test.wf k1="v1" k2="v2"'}, False),
             ({'workflow': 'test.wf k1=<% $.v1 %>'}, False),
             ({'workflow': 'test.wf k1=<% $.v1 %> k2=<% $.v2 %>'}, False),
             ({'workflow': 'test.wf k1=<% * %>'}, True),
+            ({'workflow': 'test.wf k1={{ _.v1 }}'}, False),
+            ({'workflow': 'test.wf k1={{ _.v1 }} k2={{ _.v2 }}'}, False),
+            ({'workflow': 'test.wf k1={{ $ }}'}, True),
             ({'action': 'std.noop', 'workflow': 'test.wf'}, True),
             ({'action': 123}, True),
             ({'workflow': 123}, True),
@@ -87,7 +94,10 @@ class TaskSpecValidation(v2_base.WorkflowSpecValidationTestCase):
             ({'input': {'k1': 'v1'}}, False),
             ({'input': {'k1': '<% $.v1 %>'}}, False),
             ({'input': {'k1': '<% 1 + 2 %>'}}, False),
-            ({'input': {'k1': '<% * %>'}}, True)
+            ({'input': {'k1': '<% * %>'}}, True),
+            ({'input': {'k1': '{{ _.v1 }}'}}, False),
+            ({'input': {'k1': '{{ 1 + 2 }}'}}, False),
+            ({'input': {'k1': '{{ * }}'}}, True)
         ]
 
         for task_input, expect_error in tests:
@@ -116,7 +126,15 @@ class TaskSpecValidation(v2_base.WorkflowSpecValidationTestCase):
             ({'with-items': ['x in <% $.y %>', 'i in [1, 2, 3]']}, False),
             ({'with-items': ['x in <% $.y %>', 'i in <% $.j %>']}, False),
             ({'with-items': ['x in <% * %>']}, True),
-            ({'with-items': ['x in <% $.y %>', 'i in <% * %>']}, True)
+            ({'with-items': ['x in <% $.y %>', 'i in <% * %>']}, True),
+            ({'with-items': '{{ _.y }}'}, True),
+            ({'with-items': 'x in {{ _.y }}'}, False),
+            ({'with-items': ['x in [1, 2, 3]']}, False),
+            ({'with-items': ['x in {{ _.y }}']}, False),
+            ({'with-items': ['x in {{ _.y }}', 'i in [1, 2, 3]']}, False),
+            ({'with-items': ['x in {{ _.y }}', 'i in {{ _.j }}']}, False),
+            ({'with-items': ['x in {{ * }}']}, True),
+            ({'with-items': ['x in {{ _.y }}', 'i in {{ * }}']}, True)
         ]
 
         for with_item, expect_error in tests:
@@ -136,7 +154,10 @@ class TaskSpecValidation(v2_base.WorkflowSpecValidationTestCase):
             ({'publish': {'k1': 'v1'}}, False),
             ({'publish': {'k1': '<% $.v1 %>'}}, False),
             ({'publish': {'k1': '<% 1 + 2 %>'}}, False),
-            ({'publish': {'k1': '<% * %>'}}, True)
+            ({'publish': {'k1': '<% * %>'}}, True),
+            ({'publish': {'k1': '{{ _.v1 }}'}}, False),
+            ({'publish': {'k1': '{{ 1 + 2 }}'}}, False),
+            ({'publish': {'k1': '{{ * }}'}}, True)
         ]
 
         for output, expect_error in tests:
@@ -164,39 +185,61 @@ class TaskSpecValidation(v2_base.WorkflowSpecValidationTestCase):
             ({'retry': {'count': '<% * %>', 'delay': 1}}, True),
             ({'retry': {'count': 3, 'delay': '<% 1 %>'}}, False),
             ({'retry': {'count': 3, 'delay': '<% * %>'}}, True),
+            ({'retry': {
+                'continue-on': '{{ 1 }}', 'delay': 2,
+                'break-on': '{{ 1 }}', 'count': 2
+            }}, False),
+            ({'retry': {
+                'count': 3, 'delay': 1, 'continue-on': '{{ 1 }}'
+            }}, False),
+            ({'retry': {'count': '{{ 3 }}', 'delay': 1}}, False),
+            ({'retry': {'count': '{{ * }}', 'delay': 1}}, True),
+            ({'retry': {'count': 3, 'delay': '{{ 1 }}'}}, False),
+            ({'retry': {'count': 3, 'delay': '{{ * }}'}}, True),
             ({'retry': {'count': -3, 'delay': 1}}, True),
             ({'retry': {'count': 3, 'delay': -1}}, True),
             ({'retry': {'count': '3', 'delay': 1}}, True),
             ({'retry': {'count': 3, 'delay': '1'}}, True),
             ({'retry': 'count=3 delay=1 break-on=<% false %>'}, False),
+            ({'retry': 'count=3 delay=1 break-on={{ false }}'}, False),
             ({'retry': 'count=3 delay=1'}, False),
             ({'retry': 'coun=3 delay=1'}, True),
             ({'retry': None}, True),
             ({'wait-before': 1}, False),
             ({'wait-before': '<% 1 %>'}, False),
             ({'wait-before': '<% * %>'}, True),
+            ({'wait-before': '{{ 1 }}'}, False),
+            ({'wait-before': '{{ * }}'}, True),
             ({'wait-before': -1}, True),
             ({'wait-before': 1.0}, True),
             ({'wait-before': '1'}, True),
             ({'wait-after': 1}, False),
             ({'wait-after': '<% 1 %>'}, False),
             ({'wait-after': '<% * %>'}, True),
+            ({'wait-after': '{{ 1 }}'}, False),
+            ({'wait-after': '{{ * }}'}, True),
             ({'wait-after': -1}, True),
             ({'wait-after': 1.0}, True),
             ({'wait-after': '1'}, True),
             ({'timeout': 300}, False),
             ({'timeout': '<% 300 %>'}, False),
             ({'timeout': '<% * %>'}, True),
+            ({'timeout': '{{ 300 }}'}, False),
+            ({'timeout': '{{ * }}'}, True),
             ({'timeout': -300}, True),
             ({'timeout': 300.0}, True),
             ({'timeout': '300'}, True),
             ({'pause-before': False}, False),
             ({'pause-before': '<% False %>'}, False),
             ({'pause-before': '<% * %>'}, True),
+            ({'pause-before': '{{ False }}'}, False),
+            ({'pause-before': '{{ * }}'}, True),
             ({'pause-before': 'False'}, True),
             ({'concurrency': 10}, False),
             ({'concurrency': '<% 10 %>'}, False),
             ({'concurrency': '<% * %>'}, True),
+            ({'concurrency': '{{ 10 }}'}, False),
+            ({'concurrency': '{{ * }}'}, True),
             ({'concurrency': -10}, True),
             ({'concurrency': 10.0}, True),
             ({'concurrency': '10'}, True)
@@ -218,6 +261,10 @@ class TaskSpecValidation(v2_base.WorkflowSpecValidationTestCase):
             ({'on-success': [{'email': '<% 1 %>'}, 'echo']}, False),
             ({'on-success': [{'email': '<% $.v1 in $.v2 %>'}]}, False),
             ({'on-success': [{'email': '<% * %>'}]}, True),
+            ({'on-success': [{'email': '{{ 1 }}'}]}, False),
+            ({'on-success': [{'email': '{{ 1 }}'}, 'echo']}, False),
+            ({'on-success': [{'email': '{{ _.v1 in _.v2 }}'}]}, False),
+            ({'on-success': [{'email': '{{ * }}'}]}, True),
             ({'on-success': 'email'}, False),
             ({'on-success': None}, True),
             ({'on-success': ['']}, True),
@@ -229,6 +276,10 @@ class TaskSpecValidation(v2_base.WorkflowSpecValidationTestCase):
             ({'on-error': [{'email': '<% 1 %>'}, 'echo']}, False),
             ({'on-error': [{'email': '<% $.v1 in $.v2 %>'}]}, False),
             ({'on-error': [{'email': '<% * %>'}]}, True),
+            ({'on-error': [{'email': '{{ 1 }}'}]}, False),
+            ({'on-error': [{'email': '{{ 1 }}'}, 'echo']}, False),
+            ({'on-error': [{'email': '{{ _.v1 in _.v2 }}'}]}, False),
+            ({'on-error': [{'email': '{{ * }}'}]}, True),
             ({'on-error': 'email'}, False),
             ({'on-error': None}, True),
             ({'on-error': ['']}, True),
@@ -240,6 +291,10 @@ class TaskSpecValidation(v2_base.WorkflowSpecValidationTestCase):
             ({'on-complete': [{'email': '<% 1 %>'}, 'echo']}, False),
             ({'on-complete': [{'email': '<% $.v1 in $.v2 %>'}]}, False),
             ({'on-complete': [{'email': '<% * %>'}]}, True),
+            ({'on-complete': [{'email': '{{ 1 }}'}]}, False),
+            ({'on-complete': [{'email': '{{ 1 }}'}, 'echo']}, False),
+            ({'on-complete': [{'email': '{{ _.v1 in _.v2 }}'}]}, False),
+            ({'on-complete': [{'email': '{{ * }}'}]}, True),
             ({'on-complete': 'email'}, False),
             ({'on-complete': None}, True),
             ({'on-complete': ['']}, True),
@@ -322,7 +377,10 @@ class TaskSpecValidation(v2_base.WorkflowSpecValidationTestCase):
             ({'keep-result': False}, False),
             ({'keep-result': "<% 'a' in $.val %>"}, False),
             ({'keep-result': '<% 1 + 2 %>'}, False),
-            ({'keep-result': '<% * %>'}, True)
+            ({'keep-result': '<% * %>'}, True),
+            ({'keep-result': "{{ 'a' in _.val }}"}, False),
+            ({'keep-result': '{{ 1 + 2 }}'}, False),
+            ({'keep-result': '{{ * }}'}, True)
         ]
 
         for keep_result, expect_error in tests:

mistral/tests/unit/workbook/v2/test_workflows.py

@@ -1,4 +1,5 @@
 # Copyright 2015 - StackStorm, Inc.
+# Copyright 2016 - Brocade Communications Systems, Inc.
 #
 #    Licensed under the Apache License, Version 2.0 (the "License");
 #    you may not use this file except in compliance with the License.
@@ -234,6 +235,9 @@ class WorkflowSpecValidation(base.WorkflowSpecValidationTestCase):
             ({'vars': {'v1': '<% $.input_var1 %>'}}, False),
             ({'vars': {'v1': '<% 1 + 2 %>'}}, False),
             ({'vars': {'v1': '<% * %>'}}, True),
+            ({'vars': {'v1': '{{ _.input_var1 }}'}}, False),
+            ({'vars': {'v1': '{{ 1 + 2 }}'}}, False),
+            ({'vars': {'v1': '{{ * }}'}}, True),
             ({'vars': []}, True),
             ({'vars': 'whatever'}, True),
             ({'vars': None}, True),
@@ -280,6 +284,10 @@ class WorkflowSpecValidation(base.WorkflowSpecValidationTestCase):
             ({'on-success': [{'email': '<% 1 %>'}, 'echo']}, False),
             ({'on-success': [{'email': '<% $.v1 in $.v2 %>'}]}, False),
             ({'on-success': [{'email': '<% * %>'}]}, True),
+            ({'on-success': [{'email': '{{ 1 }}'}]}, False),
+            ({'on-success': [{'email': '{{ 1 }}'}, 'echo']}, False),
+            ({'on-success': [{'email': '{{ _.v1 in _.v2 }}'}]}, False),
+            ({'on-success': [{'email': '{{ * }}'}]}, True),
             ({'on-success': 'email'}, False),
             ({'on-success': None}, True),
             ({'on-success': ['']}, True),
@@ -291,6 +299,10 @@ class WorkflowSpecValidation(base.WorkflowSpecValidationTestCase):
             ({'on-error': [{'email': '<% 1 %>'}, 'echo']}, False),
             ({'on-error': [{'email': '<% $.v1 in $.v2 %>'}]}, False),
             ({'on-error': [{'email': '<% * %>'}]}, True),
+            ({'on-error': [{'email': '{{ 1 }}'}]}, False),
+            ({'on-error': [{'email': '{{ 1 }}'}, 'echo']}, False),
+            ({'on-error': [{'email': '{{ _.v1 in _.v2 }}'}]}, False),
+            ({'on-error': [{'email': '{{ * }}'}]}, True),
             ({'on-error': 'email'}, False),
             ({'on-error': None}, True),
             ({'on-error': ['']}, True),
@@ -302,6 +314,10 @@ class WorkflowSpecValidation(base.WorkflowSpecValidationTestCase):
             ({'on-complete': [{'email': '<% 1 %>'}, 'echo']}, False),
             ({'on-complete': [{'email': '<% $.v1 in $.v2 %>'}]}, False),
             ({'on-complete': [{'email': '<% * %>'}]}, True),
+            ({'on-complete': [{'email': '{{ 1 }}'}]}, False),
+            ({'on-complete': [{'email': '{{ 1 }}'}, 'echo']}, False),
+            ({'on-complete': [{'email': '{{ _.v1 in _.v2 }}'}]}, False),
+            ({'on-complete': [{'email': '{{ * }}'}]}, True),
             ({'on-complete': 'email'}, False),
             ({'on-complete': None}, True),
             ({'on-complete': ['']}, True),
@@ -321,6 +337,10 @@ class WorkflowSpecValidation(base.WorkflowSpecValidationTestCase):
             ({'retry': {'count': '<% * %>', 'delay': 1}}, True),
             ({'retry': {'count': 3, 'delay': '<% 1 %>'}}, False),
             ({'retry': {'count': 3, 'delay': '<% * %>'}}, True),
+            ({'retry': {'count': '{{ 3 }}', 'delay': 1}}, False),
+            ({'retry': {'count': '{{ * }}', 'delay': 1}}, True),
+            ({'retry': {'count': 3, 'delay': '{{ 1 }}'}}, False),
+            ({'retry': {'count': 3, 'delay': '{{ * }}'}}, True),
             ({'retry': {'count': -3, 'delay': 1}}, True),
             ({'retry': {'count': 3, 'delay': -1}}, True),
             ({'retry': {'count': '3', 'delay': 1}}, True),
@@ -329,28 +349,38 @@ class WorkflowSpecValidation(base.WorkflowSpecValidationTestCase):
             ({'wait-before': 1}, False),
             ({'wait-before': '<% 1 %>'}, False),
             ({'wait-before': '<% * %>'}, True),
+            ({'wait-before': '{{ 1 }}'}, False),
+            ({'wait-before': '{{ * }}'}, True),
             ({'wait-before': -1}, True),
             ({'wait-before': 1.0}, True),
             ({'wait-before': '1'}, True),
             ({'wait-after': 1}, False),
             ({'wait-after': '<% 1 %>'}, False),
             ({'wait-after': '<% * %>'}, True),
+            ({'wait-after': '{{ 1 }}'}, False),
+            ({'wait-after': '{{ * }}'}, True),
             ({'wait-after': -1}, True),
             ({'wait-after': 1.0}, True),
             ({'wait-after': '1'}, True),
             ({'timeout': 300}, False),
             ({'timeout': '<% 300 %>'}, False),
             ({'timeout': '<% * %>'}, True),
+            ({'timeout': '{{ 300 }}'}, False),
+            ({'timeout': '{{ * }}'}, True),
             ({'timeout': -300}, True),
             ({'timeout': 300.0}, True),
             ({'timeout': '300'}, True),
             ({'pause-before': False}, False),
             ({'pause-before': '<% False %>'}, False),
             ({'pause-before': '<% * %>'}, True),
+            ({'pause-before': '{{ False }}'}, False),
+            ({'pause-before': '{{ * }}'}, True),
             ({'pause-before': 'False'}, True),
             ({'concurrency': 10}, False),
             ({'concurrency': '<% 10 %>'}, False),
             ({'concurrency': '<% * %>'}, True),
+            ({'concurrency': '{{ 10 }}'}, False),
+            ({'concurrency': '{{ * }}'}, True),
             ({'concurrency': -10}, True),
             ({'concurrency': 10.0}, True),
             ({'concurrency': '10'}, True)

mistral/utils/__init__.py

@@ -2,6 +2,7 @@
 #
 # Copyright 2013 - Mirantis, Inc.
 # Copyright 2015 - Huawei Technologies Co. Ltd
+# Copyright 2016 - Brocade Communications Systems, Inc.
 #
 #    Licensed under the Apache License, Version 2.0 (the "License");
 #    you may not use this file except in compliance with the License.
@@ -46,6 +47,15 @@ def generate_unicode_uuid():
     return six.text_type(str(uuid.uuid4()))
 
 
+def is_valid_uuid(uuid_string):
+    try:
+        val = uuid.UUID(uuid_string, version=4)
+    except ValueError:
+        return False
+
+    return val.hex == uuid_string.replace('-', '')
+
+
 def _get_greenlet_local_storage():
     greenlet_id = corolocal.get_ident()
 

mistral/utils/expression_utils.py

@@ -1,4 +1,5 @@
 # Copyright 2015 - Mirantis, Inc.
+# Copyright 2016 - Brocade Communications Systems, Inc.
 #
 #    Licensed under the Apache License, Version 2.0 (the "License");
 #    you may not use this file except in compliance with the License.
@@ -12,6 +13,7 @@
 #    See the License for the specific language governing permissions and
 #    limitations under the License.
 
+from functools import partial
 
 from oslo_serialization import jsonutils
 from stevedore import extension
@@ -21,18 +23,17 @@ from mistral.db.v2 import api as db_api
 from mistral import utils
 
 
-ROOT_CONTEXT = None
+ROOT_YAQL_CONTEXT = None
 
 
 def get_yaql_context(data_context):
-    global ROOT_CONTEXT
+    global ROOT_YAQL_CONTEXT
 
-    if not ROOT_CONTEXT:
+    if not ROOT_YAQL_CONTEXT:
-        ROOT_CONTEXT = yaql.create_context()
+        ROOT_YAQL_CONTEXT = yaql.create_context()
 
-        _register_functions(ROOT_CONTEXT)
+        _register_yaql_functions(ROOT_YAQL_CONTEXT)
-
+    new_ctx = ROOT_YAQL_CONTEXT.create_child_context()
-    new_ctx = ROOT_CONTEXT.create_child_context()
     new_ctx['$'] = data_context
 
     if isinstance(data_context, dict):
@@ -43,24 +44,50 @@ def get_yaql_context(data_context):
     return new_ctx
 
 
-def _register_custom_functions(yaql_ctx):
+def get_jinja_context(data_context):
-    """Register custom YAQL functions
+    new_ctx = {
+        '_': data_context
+    }
 
-    Custom YAQL functions must be added as entry points in the
+    _register_jinja_functions(new_ctx)
-    'mistral.yaql_functions' namespace
+
-    :param yaql_ctx: YAQL context object
+    if isinstance(data_context, dict):
+        new_ctx['__env'] = data_context.get('__env')
+        new_ctx['__execution'] = data_context.get('__execution')
+        new_ctx['__task_execution'] = data_context.get('__task_execution')
+
+    return new_ctx
+
+
+def get_custom_functions():
+    """Get custom functions
+
+    Retreives the list of custom evaluation functions
     """
+    functions = dict()
+
     mgr = extension.ExtensionManager(
-        namespace='mistral.yaql_functions',
+        namespace='mistral.expression.functions',
         invoke_on_load=False
     )
     for name in mgr.names():
-        yaql_function = mgr[name].plugin
+        functions[name] = mgr[name].plugin
-        yaql_ctx.register_function(yaql_function, name=name)
+
+    return functions
 
 
-def _register_functions(yaql_ctx):
+def _register_yaql_functions(yaql_ctx):
-    _register_custom_functions(yaql_ctx)
+    functions = get_custom_functions()
+
+    for name in functions:
+        yaql_ctx.register_function(functions[name], name=name)
+
+
+def _register_jinja_functions(jinja_ctx):
+    functions = get_custom_functions()
+
+    for name in functions:
+        jinja_ctx[name] = partial(functions[name], jinja_ctx['_'])
 
 
 # Additional YAQL functions needed by Mistral.
@@ -83,9 +110,9 @@ def execution_(context):
     }
 
 
-def json_pp_(data):
+def json_pp_(context, data=None):
     return jsonutils.dumps(
-        data,
+        data or context,
         indent=4
     ).replace("\\n", "\n").replace(" \n", "\n")
 
@@ -128,5 +155,5 @@ def task_(context, task_name):
     }
 
 
-def uuid_(context):
+def uuid_(context=None):
     return utils.generate_unicode_uuid()

mistral/utils/rest_utils.py

@@ -1,6 +1,7 @@
 # -*- coding: utf-8 -*-
 #
 # Copyright 2014 - Mirantis, Inc.
+# Copyright 2016 - Brocade Communications Systems, Inc.
 #
 #    Licensed under the Apache License, Version 2.0 (the "License");
 #    you may not use this file except in compliance with the License.
@@ -17,6 +18,7 @@
 import functools
 import json
 
+from oslo_log import log as logging
 import pecan
 import six
 
@@ -25,6 +27,8 @@ from wsme import exc as wsme_exc
 
 from mistral import exceptions as exc
 
+LOG = logging.getLogger(__name__)
+
 
 def wrap_wsme_controller_exception(func):
     """Decorator for controllers method.
@@ -39,6 +43,7 @@ def wrap_wsme_controller_exception(func):
         except (exc.MistralException, exc.MistralError) as e:
             pecan.response.translatable_error = e
 
+            LOG.error('Error during API call: %s' % str(e))
             raise wsme_exc.ClientSideError(
                 msg=six.text_type(e),
                 status_code=e.http_code
@@ -58,6 +63,7 @@ def wrap_pecan_controller_exception(func):
         try:
             return func(*args, **kwargs)
         except (exc.MistralException, exc.MistralError) as e:
+            LOG.error('Error during API call: %s' % str(e))
             return webob.Response(
                 status=e.http_code,
                 content_type='application/json',

mistral/workbook/base.py

@@ -27,7 +27,7 @@ from mistral.workbook import types
 
 CMD_PTRN = re.compile("^[\w\.]+[^=\(\s\"]*")
 
-INLINE_YAQL = expr.INLINE_YAQL_REGEXP
+EXPRESSION = '|'.join([expr.patterns[name] for name in expr.patterns])
 _ALL_IN_BRACKETS = "\[.*\]\s*"
 _ALL_IN_QUOTES = "\"[^\"]*\"\s*"
 _ALL_IN_APOSTROPHES = "'[^']*'\s*"
@@ -37,7 +37,7 @@ _FALSE = "false"
 _NULL = "null"
 
 ALL = (
-    _ALL_IN_QUOTES, _ALL_IN_APOSTROPHES, INLINE_YAQL,
+    _ALL_IN_QUOTES, _ALL_IN_APOSTROPHES, EXPRESSION,
     _ALL_IN_BRACKETS, _TRUE, _FALSE, _NULL, _DIGITS
 )
 
@@ -194,7 +194,7 @@ class BaseSpec(object):
         """
         pass
 
-    def validate_yaql_expr(self, dsl_part):
+    def validate_expr(self, dsl_part):
         if isinstance(dsl_part, six.string_types):
             expr.validate(dsl_part)
         elif isinstance(dsl_part, list):
@@ -278,9 +278,10 @@ class BaseSpec(object):
 
         params = {}
 
-        for k, v in re.findall(PARAMS_PTRN, cmd_str):
+        for match in re.findall(PARAMS_PTRN, cmd_str):
+            k = match[0]
             # Remove embracing quotes.
-            v = v.strip()
+            v = match[1].strip()
             if v[0] == '"' or v[0] == "'":
                 v = v[1:-1]
             else:

mistral/workbook/types.py

@@ -12,6 +12,9 @@
 #    See the License for the specific language governing permissions and
 #    limitations under the License.
 
+from mistral import expressions
+
+
 NONEMPTY_STRING = {
     "type": "string",
     "minLength": 1
@@ -34,16 +37,18 @@ POSITIVE_NUMBER = {
     "minimum": 0.0
 }
 
-YAQL = {
+EXPRESSION = {
-    "type": "string",
+    "oneOf": [{
-    "pattern": "^<%.*?%>\\s*$"
+        "type": "string",
+        "pattern": "^%s\\s*$" % expressions.patterns[name]
+    } for name in expressions.patterns]
 }
 
-YAQL_CONDITION = {
+EXPRESSION_CONDITION = {
     "type": "object",
     "minProperties": 1,
     "patternProperties": {
-        "^\w+$": YAQL
+        "^\w+$": EXPRESSION
     }
 }
 
@@ -54,8 +59,7 @@ ANY = {
         {"type": "integer"},
         {"type": "number"},
         {"type": "object"},
-        {"type": "string"},
+        {"type": "string"}
-        YAQL
     ]
 }
 
@@ -67,8 +71,7 @@ ANY_NULLABLE = {
         {"type": "integer"},
         {"type": "number"},
         {"type": "object"},
-        {"type": "string"},
+        {"type": "string"}
-        YAQL
     ]
 }
 
@@ -89,31 +92,31 @@ ONE_KEY_DICT = {
     }
 }
 
-STRING_OR_YAQL_CONDITION = {
+STRING_OR_EXPRESSION_CONDITION = {
     "oneOf": [
         NONEMPTY_STRING,
-        YAQL_CONDITION
+        EXPRESSION_CONDITION
     ]
 }
 
-YAQL_OR_POSITIVE_INTEGER = {
+EXPRESSION_OR_POSITIVE_INTEGER = {
     "oneOf": [
-        YAQL,
+        EXPRESSION,
         POSITIVE_INTEGER
     ]
 }
 
-YAQL_OR_BOOLEAN = {
+EXPRESSION_OR_BOOLEAN = {
     "oneOf": [
-        YAQL,
+        EXPRESSION,
         {"type": "boolean"}
     ]
 }
 
 
-UNIQUE_STRING_OR_YAQL_CONDITION_LIST = {
+UNIQUE_STRING_OR_EXPRESSION_CONDITION_LIST = {
     "type": "array",
-    "items": STRING_OR_YAQL_CONDITION,
+    "items": STRING_OR_EXPRESSION_CONDITION,
     "uniqueItems": True,
     "minItems": 1
 }

mistral/workbook/v2/actions.py

@@ -54,12 +54,12 @@ class ActionSpec(base.BaseSpec):
 
         # Validate YAQL expressions.
         inline_params = self._parse_cmd_and_input(self._data.get('base'))[1]
-        self.validate_yaql_expr(inline_params)
+        self.validate_expr(inline_params)
 
-        self.validate_yaql_expr(self._data.get('base-input', {}))
+        self.validate_expr(self._data.get('base-input', {}))
 
         if isinstance(self._data.get('output'), six.string_types):
-            self.validate_yaql_expr(self._data.get('output'))
+            self.validate_expr(self._data.get('output'))
 
     def get_name(self):
         return self._name

mistral/workbook/v2/policies.py

@@ -19,11 +19,11 @@ from mistral.workbook.v2 import retry_policy
 
 
 RETRY_SCHEMA = retry_policy.RetrySpec.get_schema(includes=None)
-WAIT_BEFORE_SCHEMA = types.YAQL_OR_POSITIVE_INTEGER
+WAIT_BEFORE_SCHEMA = types.EXPRESSION_OR_POSITIVE_INTEGER
-WAIT_AFTER_SCHEMA = types.YAQL_OR_POSITIVE_INTEGER
+WAIT_AFTER_SCHEMA = types.EXPRESSION_OR_POSITIVE_INTEGER
-TIMEOUT_SCHEMA = types.YAQL_OR_POSITIVE_INTEGER
+TIMEOUT_SCHEMA = types.EXPRESSION_OR_POSITIVE_INTEGER
-PAUSE_BEFORE_SCHEMA = types.YAQL_OR_BOOLEAN
+PAUSE_BEFORE_SCHEMA = types.EXPRESSION_OR_BOOLEAN
-CONCURRENCY_SCHEMA = types.YAQL_OR_POSITIVE_INTEGER
+CONCURRENCY_SCHEMA = types.EXPRESSION_OR_POSITIVE_INTEGER
 
 
 class PoliciesSpec(base.BaseSpec):
@@ -59,11 +59,11 @@ class PoliciesSpec(base.BaseSpec):
         super(PoliciesSpec, self).validate_schema()
 
         # Validate YAQL expressions.
-        self.validate_yaql_expr(self._data.get('wait-before', 0))
+        self.validate_expr(self._data.get('wait-before', 0))
-        self.validate_yaql_expr(self._data.get('wait-after', 0))
+        self.validate_expr(self._data.get('wait-after', 0))
-        self.validate_yaql_expr(self._data.get('timeout', 0))
+        self.validate_expr(self._data.get('timeout', 0))
-        self.validate_yaql_expr(self._data.get('pause-before', False))
+        self.validate_expr(self._data.get('pause-before', False))
-        self.validate_yaql_expr(self._data.get('concurrency', 0))
+        self.validate_expr(self._data.get('concurrency', 0))
 
     def get_retry(self):
         return self._retry

mistral/workbook/v2/retry_policy.py

@@ -26,15 +26,15 @@ class RetrySpec(base.BaseSpec):
         "properties": {
             "count": {
                 "oneOf": [
-                    types.YAQL,
+                    types.EXPRESSION,
                     types.POSITIVE_INTEGER
                 ]
             },
-            "break-on": types.YAQL,
+            "break-on": types.EXPRESSION,
-            "continue-on": types.YAQL,
+            "continue-on": types.EXPRESSION,
             "delay": {
                 "oneOf": [
-                    types.YAQL,
+                    types.EXPRESSION,
                     types.POSITIVE_INTEGER
                 ]
             },
@@ -74,10 +74,10 @@ class RetrySpec(base.BaseSpec):
         super(RetrySpec, self).validate_schema()
 
         # Validate YAQL expressions.
-        self.validate_yaql_expr(self._data.get('count'))
+        self.validate_expr(self._data.get('count'))
-        self.validate_yaql_expr(self._data.get('delay'))
+        self.validate_expr(self._data.get('delay'))
-        self.validate_yaql_expr(self._data.get('break-on'))
+        self.validate_expr(self._data.get('break-on'))
-        self.validate_yaql_expr(self._data.get('continue-on'))
+        self.validate_expr(self._data.get('continue-on'))
 
     def get_count(self):
         return self._count

mistral/workbook/v2/task_defaults.py

@@ -32,7 +32,7 @@ class TaskDefaultsSpec(base.BaseSpec):
     _on_clause_type = {
         "oneOf": [
             types.NONEMPTY_STRING,
-            types.UNIQUE_STRING_OR_YAQL_CONDITION_LIST
+            types.UNIQUE_STRING_OR_EXPRESSION_CONDITION_LIST
         ]
     }
 
@@ -93,7 +93,7 @@ class TaskDefaultsSpec(base.BaseSpec):
     def _validate_transitions(self, on_clause):
         val = self._data.get(on_clause, [])
 
-        [self.validate_yaql_expr(t)
+        [self.validate_expr(t)
          for t in ([val] if isinstance(val, six.string_types) else val)]
 
     def get_policies(self):

mistral/workbook/v2/tasks.py

@@ -19,15 +19,15 @@ import re
 import six
 
 from mistral import exceptions as exc
-from mistral import expressions as expr
+from mistral import expressions
 from mistral import utils
 from mistral.workbook import types
 from mistral.workbook.v2 import base
 from mistral.workbook.v2 import policies
 
-
+_expr_ptrns = [expressions.patterns[name] for name in expressions.patterns]
 WITH_ITEMS_PTRN = re.compile(
-    "\s*([\w\d_\-]+)\s*in\s*(\[.+\]|%s)" % expr.INLINE_YAQL_REGEXP
+    "\s*([\w\d_\-]+)\s*in\s*(\[.+\]|%s)" % '|'.join(_expr_ptrns)
 )
 RESERVED_TASK_NAMES = [
     'noop',
@@ -62,8 +62,8 @@ class TaskSpec(base.BaseSpec):
             "pause-before": policies.PAUSE_BEFORE_SCHEMA,
             "concurrency": policies.CONCURRENCY_SCHEMA,
             "target": types.NONEMPTY_STRING,
-            "keep-result": types.YAQL_OR_BOOLEAN,
+            "keep-result": types.EXPRESSION_OR_BOOLEAN,
-            "safe-rerun": types.YAQL_OR_BOOLEAN
+            "safe-rerun": types.EXPRESSION_OR_BOOLEAN
         },
         "additionalProperties": False,
         "anyOf": [
@@ -122,12 +122,12 @@ class TaskSpec(base.BaseSpec):
         # Validate YAQL expressions.
         if action or workflow:
             inline_params = self._parse_cmd_and_input(action or workflow)[1]
-            self.validate_yaql_expr(inline_params)
+            self.validate_expr(inline_params)
 
-        self.validate_yaql_expr(self._data.get('input', {}))
+        self.validate_expr(self._data.get('input', {}))
-        self.validate_yaql_expr(self._data.get('publish', {}))
+        self.validate_expr(self._data.get('publish', {}))
-        self.validate_yaql_expr(self._data.get('keep-result', {}))
+        self.validate_expr(self._data.get('keep-result', {}))
-        self.validate_yaql_expr(self._data.get('safe-rerun', {}))
+        self.validate_expr(self._data.get('safe-rerun', {}))
 
     def _transform_with_items(self):
         raw = self._data.get('with-items', [])
@@ -149,11 +149,13 @@ class TaskSpec(base.BaseSpec):
                        "%s" % self._data)
                 raise exc.InvalidModelException(msg)
 
-            var_name, array = match.groups()
+            match_groups = match.groups()
+            var_name = match_groups[0]
+            array = match_groups[1]
 
             # Validate YAQL expression that may follow after "in" for the
             # with-items syntax "var in {[some, list] | <% $.array %> }".
-            self.validate_yaql_expr(array)
+            self.validate_expr(array)
 
             if array.startswith('['):
                 try:
@@ -223,7 +225,7 @@ class DirectWorkflowTaskSpec(TaskSpec):
     _on_clause_type = {
         "oneOf": [
             types.NONEMPTY_STRING,
-            types.UNIQUE_STRING_OR_YAQL_CONDITION_LIST
+            types.UNIQUE_STRING_OR_EXPRESSION_CONDITION_LIST
         ]
     }
 
@@ -271,7 +273,7 @@ class DirectWorkflowTaskSpec(TaskSpec):
     def _validate_transitions(self, on_clause):
         val = self._data.get(on_clause, [])
 
-        [self.validate_yaql_expr(t)
+        [self.validate_expr(t)
          for t in ([val] if isinstance(val, six.string_types) else val)]
 
     @staticmethod

mistral/workbook/v2/workflows.py

@@ -77,9 +77,9 @@ class WorkflowSpec(base.BaseSpec):
                 "Workflow doesn't have any tasks [data=%s]" % self._data
             )
 
-        # Validate YAQL expressions.
+        # Validate expressions.
-        self.validate_yaql_expr(self._data.get('output', {}))
+        self.validate_expr(self._data.get('output', {}))
-        self.validate_yaql_expr(self._data.get('vars', {}))
+        self.validate_expr(self._data.get('vars', {}))
 
     def validate_semantics(self):
         super(WorkflowSpec, self).validate_semantics()

requirements.txt

@@ -7,6 +7,7 @@ Babel>=2.3.4 # BSD
 croniter>=0.3.4 # MIT License
 cachetools>=1.1.0 # MIT License
 eventlet!=0.18.3,>=0.18.2 # MIT
+Jinja2>=2.8 # BSD License (3 clause)
 jsonschema!=2.5.0,<3.0.0,>=2.0.0 # MIT
 keystonemiddleware!=4.5.0,>=4.2.0 # Apache-2.0
 mock>=2.0 # BSD

setup.cfg

@@ -69,12 +69,16 @@ mistral.actions =
     std.javascript = mistral.actions.std_actions:JavaScriptAction
     std.sleep = mistral.actions.std_actions:SleepAction
 
-mistral.yaql_functions =
+mistral.expression.functions =
-    json_pp = mistral.utils.yaql_utils:json_pp_
+    json_pp = mistral.utils.expression_utils:json_pp_
-    task = mistral.utils.yaql_utils:task_
+    task = mistral.utils.expression_utils:task_
-    execution = mistral.utils.yaql_utils:execution_
+    execution = mistral.utils.expression_utils:execution_
-    env = mistral.utils.yaql_utils:env_
+    env = mistral.utils.expression_utils:env_
-    uuid = mistral.utils.yaql_utils:uuid_
+    uuid = mistral.utils.expression_utils:uuid_
+
+mistral.expression.evaluators =
+    yaql = mistral.expressions.yaql_expression:InlineYAQLEvaluator
+    jinja = mistral.expressions.jinja_expression:InlineJinjaEvaluator
 
 mistral.auth =
     keystone = mistral.auth.keystone:KeystoneAuthHandler