diff --git a/doc/source/dsl/dsl_v2.rst b/doc/source/dsl/dsl_v2.rst index c09c6e804..b2e125d78 100644 --- a/doc/source/dsl/dsl_v2.rst +++ b/doc/source/dsl/dsl_v2.rst @@ -30,8 +30,8 @@ will be described in details below: Prerequisites ------------- -Mistral DSL takes advantage of -`YAQL `__ expression language to +Mistral DSL supports `YAQL `__ and +`Jinja2 `__ expression languages to reference workflow context variables and thereby implements passing data between workflow tasks. It's also referred to as Data Flow mechanism. YAQL is a simple but powerful query language that allows to extract @@ -51,11 +51,12 @@ in the following sections of DSL: Mistral DSL is fully based on YAML and knowledge of YAML is a plus for better understanding of the material in this specification. It also -takes advantage of YAQL query language to define expressions in workflow +takes advantage of supported query languages to define expressions in workflow and action definitions. - Yet Another Markup Language (YAML): http://yaml.org - Yet Another Query Language (YAQL): https://pypi.python.org/pypi/yaql/1.0.0 +- Jinja 2: http://jinja.pocoo.org/docs/dev/ Workflows --------- @@ -124,7 +125,7 @@ Common Workflow Attributes - **description** - Arbitrary text containing workflow description. *Optional*. - **input** - List defining required input parameter names and optionally their default values in a form "my_param: 123". *Optional*. -- **output** - Any data structure arbitrarily containing YAQL +- **output** - Any data structure arbitrarily containing expressions that defines workflow output. May be nested. *Optional*. - **task-defaults** - Default settings for some of task attributes defined at workflow level. *Optional*. Corresponding attribute @@ -188,15 +189,15 @@ attributes: *Mutually exclusive with* **action**. - **input** - Actual input parameter values of the task. *Optional*. Value of each parameter is a JSON-compliant type such as number, - string etc, dictionary or list. It can also be a YAQL expression to + string etc, dictionary or list. It can also be an expression to retrieve value from task context or any of the mentioned types - containing inline YAQL expressions (for example, string "<% + containing inline expressions (for example, string "<% $.movie_name %> is a cool movie!") - **publish** - Dictionary of variables to publish to the workflow context. Any JSON-compatible data structure optionally containing - YAQL expression to select precisely what needs to be published. + expression to select precisely what needs to be published. Published variables will be accessible for downstream tasks via using - YAQL expressions. *Optional*. + expressions. *Optional*. - **with-items** - If configured, it allows to run action or workflow associated with a task multiple times on a provided list of items. See `Processing collections using @@ -278,10 +279,10 @@ Defines a pattern how task should be repeated in case of an error. repeated. - **delay** - Defines a delay in seconds between subsequent task iterations. -- **break-on** - Defines a YAQL expression that will break iteration +- **break-on** - Defines an expression that will break iteration loop if it evaluates to 'true'. If it fires then the task is considered error. -- **continue-on** - Defines a YAQL expression that will continue iteration +- **continue-on** - Defines an expression that will continue iteration loop if it evaluates to 'true'. If it fires then the task is considered successful. If it evaluates to 'false' then policy will break the iteration. @@ -293,7 +294,7 @@ Retry policy can also be configured on a single line as:   action: my_action   retry: count=10 delay=5 break-on=<% $.foo = 'bar' %> -All parameter values for any policy can be defined as YAQL expressions. +All parameter values for any policy can be defined as expressions. Simplified Input Syntax ''''''''''''''''''''''' @@ -407,7 +408,7 @@ Transitions with YAQL expressions ''''''''''''''''''''''''''''''''' Task transitions can be determined by success/error/completeness of the -previous tasks and also by additional YAQL guard expressions that can +previous tasks and also by additional guard expressions that can access any data produced by upstream tasks. So in the example above task 'create_vm' could also have a YAQL expression on transition to task 'send_success_email' as follows: @@ -420,8 +421,8 @@ access any data produced by upstream tasks. So in the example above task    - send_success_email: <% $.vm_id != null %> And this would tell Mistral to run 'send_success_email' task only if -'vm_id' variable published by task 'create_vm' is not empty. YAQL -expressions can also be applied to 'on-error' and 'on-complete'. +'vm_id' variable published by task 'create_vm' is not empty. +Expressions can also be applied to 'on-error' and 'on-complete'. Fork '''' @@ -475,7 +476,7 @@ run only if all upstream tasks (ones that lead to this task) are completed and corresponding conditions have triggered. Task A is considered an upstream task of Task B if Task A has Task B mentioned in any of its "on-success", "on-error" and "on-complete" clauses regardless -of YAQL guard expressions. +of guard expressions. Partial Join (join: 2) @@ -945,14 +946,14 @@ Attributes used only for documenting purposes. Mistral now does not enforce actual input parameters to exactly correspond to this list. Based parameters will be calculated based on provided actual parameters - with using YAQL expressions so what's used in expressions implicitly + with using expressions so what's used in expressions implicitly define real input parameters. Dictionary of actual input parameters - is referenced in YAQL as '$.'. Redundant parameters will be simply - ignored. + (expression context) is referenced as '$.' in YAQL and as '_.' in Jinja. + Redundant parameters will be simply ignored. - **output** - Any data structure defining how to calculate output of this action based on output of base action. It can optionally have - YAQL expressions to access properties of base action output - referenced in YAQL as '$.'. + expressions to access properties of base action output through expression + context. Workbooks --------- @@ -1045,7 +1046,7 @@ Attributes Predefined Values/Functions in execution data context ----------------------------------------------------- -Using YAQL it is possible to use some predefined values in Mistral DSL. +Using expressions it is possible to use some predefined values in Mistral DSL. - **OpenStack context** - **Task result** diff --git a/mistral/exceptions.py b/mistral/exceptions.py index 92b65f2ae..81003b9ff 100644 --- a/mistral/exceptions.py +++ b/mistral/exceptions.py @@ -1,5 +1,6 @@ # Copyright 2013 - Mirantis, Inc. # Copyright 2015 - StackStorm, Inc. +# Copyright 2016 - Brocade Communications Systems, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -112,8 +113,15 @@ class DSLParsingException(MistralException): http_code = 400 -class YaqlGrammarException(DSLParsingException): +class ExpressionGrammarException(DSLParsingException): http_code = 400 + + +class JinjaGrammarException(ExpressionGrammarException): + message = "Invalid grammar of Jinja expression" + + +class YaqlGrammarException(ExpressionGrammarException): message = "Invalid grammar of YAQL expression" @@ -124,8 +132,15 @@ class InvalidModelException(DSLParsingException): # Various common exceptions and errors. -class YaqlEvaluationException(MistralException): +class EvaluationException(MistralException): http_code = 400 + + +class JinjaEvaluationException(EvaluationException): + message = "Can not evaluate Jinja expression" + + +class YaqlEvaluationException(EvaluationException): message = "Can not evaluate YAQL expression" diff --git a/mistral/expressions/__init__.py b/mistral/expressions/__init__.py new file mode 100644 index 000000000..11612155c --- /dev/null +++ b/mistral/expressions/__init__.py @@ -0,0 +1,103 @@ +# Copyright 2013 - Mirantis, Inc. +# Copyright 2015 - StackStorm, Inc. +# Copyright 2016 - Brocade Communications Systems, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import copy + +from oslo_log import log as logging +import six +from stevedore import extension + +from mistral import exceptions as exc + +LOG = logging.getLogger(__name__) + +_mgr = extension.ExtensionManager( + namespace='mistral.expression.evaluators', + invoke_on_load=False +) + +_evaluators = [] +patterns = {} + +for name in sorted(_mgr.names()): + evaluator = _mgr[name].plugin + _evaluators.append((name, evaluator)) + patterns[name] = evaluator.find_expression_pattern.pattern + + +def validate(expression): + LOG.debug("Validating expression [expression='%s']", expression) + + if not isinstance(expression, six.string_types): + return + + expression_found = None + + for name, evaluator in _evaluators: + if evaluator.is_expression(expression): + if expression_found: + raise exc.ExpressionGrammarException( + "The line already contains an expression of type '%s'. " + "Mixing expression types in a single line is not allowed." + % expression_found) + + try: + evaluator.validate(expression) + except Exception: + raise + else: + expression_found = name + + +def evaluate(expression, context): + for name, evaluator in _evaluators: + # Check if the passed value is expression so we don't need to do this + # every time on a caller side. + if (isinstance(expression, six.string_types) and + evaluator.is_expression(expression)): + return evaluator.evaluate(expression, context) + + return expression + + +def _evaluate_item(item, context): + if isinstance(item, six.string_types): + try: + return evaluate(item, context) + except AttributeError as e: + LOG.debug("Expression %s is not evaluated, [context=%s]: %s" + % (item, context, e)) + return item + else: + return evaluate_recursively(item, context) + + +def evaluate_recursively(data, context): + data = copy.deepcopy(data) + + if not context: + return data + + if isinstance(data, dict): + for key in data: + data[key] = _evaluate_item(data[key], context) + elif isinstance(data, list): + for index, item in enumerate(data): + data[index] = _evaluate_item(item, context) + elif isinstance(data, six.string_types): + return _evaluate_item(data, context) + + return data diff --git a/mistral/expressions/base_expression.py b/mistral/expressions/base_expression.py new file mode 100644 index 000000000..b3b2e8e8f --- /dev/null +++ b/mistral/expressions/base_expression.py @@ -0,0 +1,55 @@ +# Copyright 2013 - Mirantis, Inc. +# Copyright 2015 - StackStorm, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import abc + + +class Evaluator(object): + """Expression evaluator interface. + + Having this interface gives the flexibility to change the actual expression + language used in Mistral DSL for conditions, output calculation etc. + """ + + @classmethod + @abc.abstractmethod + def validate(cls, expression): + """Parse and validates the expression. + + :param expression: Expression string + :return: True if expression is valid + """ + pass + + @classmethod + @abc.abstractmethod + def evaluate(cls, expression, context): + """Evaluates the expression against the given data context. + + :param expression: Expression string + :param context: Data context + :return: Expression result + """ + pass + + @classmethod + @abc.abstractmethod + def is_expression(cls, expression): + """Check expression string and decide whether it is expression or not. + + :param expression: Expression string + :return: True if string is expression + """ + pass diff --git a/mistral/expressions/jinja_expression.py b/mistral/expressions/jinja_expression.py new file mode 100644 index 000000000..8a966239f --- /dev/null +++ b/mistral/expressions/jinja_expression.py @@ -0,0 +1,142 @@ +# Copyright 2016 - Brocade Communications Systems, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import re + +import jinja2 +from jinja2 import parser as jinja_parse +from oslo_log import log as logging +import six + +from mistral import exceptions as exc +from mistral.expressions.base_expression import Evaluator +from mistral.utils import expression_utils + + +LOG = logging.getLogger(__name__) + +JINJA_REGEXP = '({{(.*)?}})' +JINJA_BLOCK_REGEXP = '({%(.*)?%})' + +_environment = jinja2.Environment( + undefined=jinja2.StrictUndefined, + trim_blocks=True, + lstrip_blocks=True +) + +_filters = expression_utils.get_custom_functions() + +for name in _filters: + _environment.filters[name] = _filters[name] + + +class JinjaEvaluator(Evaluator): + _env = _environment.overlay() + + @classmethod + def validate(cls, expression): + LOG.debug( + "Validating Jinja expression [expression='%s']", expression) + + if not isinstance(expression, six.string_types): + raise exc.JinjaEvaluationException("Unsupported type '%s'." % + type(expression)) + + try: + parser = jinja_parse.Parser(cls._env, expression, state='variable') + parser.parse_expression() + except jinja2.exceptions.TemplateError as e: + raise exc.JinjaGrammarException("Syntax error '%s'." % + str(e)) + + @classmethod + def evaluate(cls, expression, data_context): + LOG.debug( + "Evaluating Jinja expression [expression='%s', context=%s]" + % (expression, data_context) + ) + + opts = { + 'undefined_to_none': False + } + + ctx = expression_utils.get_jinja_context(data_context) + + try: + result = cls._env.compile_expression(expression, **opts)(**ctx) + + # For StrictUndefined values, UndefinedError only gets raised when + # the value is accessed, not when it gets created. The simplest way + # to access it is to try and cast it to string. + str(result) + except jinja2.exceptions.UndefinedError as e: + raise exc.JinjaEvaluationException("Undefined error '%s'." % + str(e)) + + LOG.debug("Jinja expression result: %s" % result) + + return result + + @classmethod + def is_expression(cls, s): + # The class should only be called from within InlineJinjaEvaluator. The + # return value prevents the class from being accidentally added as + # Extension + return False + + +class InlineJinjaEvaluator(Evaluator): + # The regular expression for Jinja variables and blocks + find_expression_pattern = re.compile(JINJA_REGEXP) + find_block_pattern = re.compile(JINJA_BLOCK_REGEXP) + + _env = _environment.overlay() + + @classmethod + def validate(cls, expression): + LOG.debug( + "Validating Jinja expression [expression='%s']", expression) + + if not isinstance(expression, six.string_types): + raise exc.JinjaEvaluationException("Unsupported type '%s'." % + type(expression)) + + try: + cls._env.parse(expression) + except jinja2.exceptions.TemplateError as e: + raise exc.JinjaGrammarException("Syntax error '%s'." % + str(e)) + + @classmethod + def evaluate(cls, expression, data_context): + LOG.debug( + "Evaluating Jinja expression [expression='%s', context=%s]" + % (expression, data_context) + ) + + patterns = cls.find_expression_pattern.findall(expression) + if patterns[0][0] == expression: + result = JinjaEvaluator.evaluate(patterns[0][1], data_context) + else: + ctx = expression_utils.get_jinja_context(data_context) + result = cls._env.from_string(expression).render(**ctx) + + LOG.debug("Jinja expression result: %s" % result) + + return result + + @classmethod + def is_expression(cls, s): + return (cls.find_expression_pattern.search(s) or + cls.find_block_pattern.search(s)) diff --git a/mistral/expressions.py b/mistral/expressions/yaql_expression.py similarity index 58% rename from mistral/expressions.py rename to mistral/expressions/yaql_expression.py index 48d1f81f1..09d9f9ff9 100644 --- a/mistral/expressions.py +++ b/mistral/expressions/yaql_expression.py @@ -1,5 +1,6 @@ # Copyright 2013 - Mirantis, Inc. # Copyright 2015 - StackStorm, Inc. +# Copyright 2016 - Brocade Communications Systems, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -13,8 +14,6 @@ # See the License for the specific language governing permissions and # limitations under the License. -import abc -import copy import inspect import re @@ -24,50 +23,13 @@ from yaql.language import exceptions as yaql_exc from yaql.language import factory from mistral import exceptions as exc -from mistral.utils import yaql_utils - +from mistral.expressions.base_expression import Evaluator +from mistral.utils import expression_utils LOG = logging.getLogger(__name__) YAQL_ENGINE = factory.YaqlFactory().create() - -class Evaluator(object): - """Expression evaluator interface. - - Having this interface gives the flexibility to change the actual expression - language used in Mistral DSL for conditions, output calculation etc. - """ - - @classmethod - @abc.abstractmethod - def validate(cls, expression): - """Parse and validates the expression. - - :param expression: Expression string - :return: True if expression is valid - """ - pass - - @classmethod - @abc.abstractmethod - def evaluate(cls, expression, context): - """Evaluates the expression against the given data context. - - :param expression: Expression string - :param context: Data context - :return: Expression result - """ - pass - - @classmethod - @abc.abstractmethod - def is_expression(cls, expression): - """Check expression string and decide whether it is expression or not. - - :param expression: Expression string - :return: True if string is expression - """ - pass +INLINE_YAQL_REGEXP = '<%.*?%>' class YAQLEvaluator(Evaluator): @@ -87,7 +49,7 @@ class YAQLEvaluator(Evaluator): try: result = YAQL_ENGINE(expression).evaluate( - context=yaql_utils.get_yaql_context(data_context) + context=expression_utils.get_yaql_context(data_context) ) except (yaql_exc.YaqlException, KeyError, ValueError, TypeError) as e: raise exc.YaqlEvaluationException( @@ -101,12 +63,9 @@ class YAQLEvaluator(Evaluator): @classmethod def is_expression(cls, s): - # TODO(rakhmerov): It should be generalized since it may not be YAQL. - # Treat any string as a YAQL expression. - return isinstance(s, six.string_types) - - -INLINE_YAQL_REGEXP = '<%.*?%>' + # The class should not be used outside of InlineYAQLEvaluator since by + # convention, YAQL expression should always be wrapped in '<% %>'. + return False class InlineYAQLEvaluator(YAQLEvaluator): @@ -155,56 +114,8 @@ class InlineYAQLEvaluator(YAQLEvaluator): @classmethod def is_expression(cls, s): - return s + return cls.find_expression_pattern.search(s) @classmethod def find_inline_expressions(cls, s): return cls.find_expression_pattern.findall(s) - - -# TODO(rakhmerov): Make it configurable. -_EVALUATOR = InlineYAQLEvaluator - - -def validate(expression): - return _EVALUATOR.validate(expression) - - -def evaluate(expression, context): - # Check if the passed value is expression so we don't need to do this - # every time on a caller side. - if (not isinstance(expression, six.string_types) or - not _EVALUATOR.is_expression(expression)): - return expression - - return _EVALUATOR.evaluate(expression, context) - - -def _evaluate_item(item, context): - if isinstance(item, six.string_types): - try: - return evaluate(item, context) - except AttributeError as e: - LOG.debug("Expression %s is not evaluated, [context=%s]: %s" - % (item, context, e)) - return item - else: - return evaluate_recursively(item, context) - - -def evaluate_recursively(data, context): - data = copy.deepcopy(data) - - if not context: - return data - - if isinstance(data, dict): - for key in data: - data[key] = _evaluate_item(data[key], context) - elif isinstance(data, list): - for index, item in enumerate(data): - data[index] = _evaluate_item(item, context) - elif isinstance(data, six.string_types): - return _evaluate_item(data, context) - - return data diff --git a/mistral/tests/resources/action_jinja.yaml b/mistral/tests/resources/action_jinja.yaml new file mode 100644 index 000000000..54cee82d8 --- /dev/null +++ b/mistral/tests/resources/action_jinja.yaml @@ -0,0 +1,21 @@ +--- +version: "2.0" + +greeting: + description: "This action says 'Hello'" + tags: [hello] + base: std.echo + base-input: + output: 'Hello, {{ _.name }}' + input: + - name + output: + string: '{{ _ }}' + +farewell: + base: std.echo + base-input: + output: 'Bye!' + output: + info: '{{ _ }}' + diff --git a/mistral/tests/resources/action_v2.yaml b/mistral/tests/resources/action_v2.yaml index f0caede5b..bf2b879db 100644 --- a/mistral/tests/resources/action_v2.yaml +++ b/mistral/tests/resources/action_v2.yaml @@ -10,12 +10,12 @@ greeting: input: - name output: - string: <% $.output %> + string: <% $ %> farewell: base: std.echo base-input: output: 'Bye!' output: - info: <% $.output %> + info: <% $ %> diff --git a/mistral/tests/resources/wf_jinja.yaml b/mistral/tests/resources/wf_jinja.yaml new file mode 100644 index 000000000..7a5539749 --- /dev/null +++ b/mistral/tests/resources/wf_jinja.yaml @@ -0,0 +1,34 @@ +--- +version: '2.0' + +wf: + type: direct + + tasks: + hello: + action: std.echo output="Hello" + wait-before: 1 + publish: + result: '{{ task("hello").result }}' + +wf1: + type: reverse + input: + - farewell + + tasks: + addressee: + action: std.echo output="John" + publish: + name: '{{ task("addressee").result }}' + + goodbye: + action: std.echo output="{{ _.farewell }}, {{ _.name }}" + requires: [addressee] + +wf2: + type: direct + + tasks: + hello: + action: std.echo output="Hello" diff --git a/mistral/tests/unit/expressions/test_jinja_expression.py b/mistral/tests/unit/expressions/test_jinja_expression.py new file mode 100644 index 000000000..47956f00f --- /dev/null +++ b/mistral/tests/unit/expressions/test_jinja_expression.py @@ -0,0 +1,397 @@ +# Copyright 2016 - Brocade Communications Systems, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import mock + +from mistral import exceptions as exc +from mistral.expressions import jinja_expression as expr +from mistral.tests.unit import base +from mistral import utils + +DATA = { + "server": { + "id": "03ea824a-aa24-4105-9131-66c48ae54acf", + "name": "cloud-fedora", + "status": "ACTIVE" + }, + "status": "OK" +} + +SERVERS = { + "servers": [ + {'name': 'centos'}, + {'name': 'ubuntu'}, + {'name': 'fedora'} + ] +} + + +class JinjaEvaluatorTest(base.BaseTest): + def setUp(self): + super(JinjaEvaluatorTest, self).setUp() + + self._evaluator = expr.JinjaEvaluator() + + def test_expression_result(self): + res = self._evaluator.evaluate('_.server', DATA) + self.assertEqual({ + 'id': '03ea824a-aa24-4105-9131-66c48ae54acf', + 'name': 'cloud-fedora', + 'status': 'ACTIVE' + }, res) + + res = self._evaluator.evaluate('_.server.id', DATA) + self.assertEqual('03ea824a-aa24-4105-9131-66c48ae54acf', res) + + res = self._evaluator.evaluate("_.server.status == 'ACTIVE'", DATA) + self.assertTrue(res) + + def test_wrong_expression(self): + res = self._evaluator.evaluate("_.status == 'Invalid value'", DATA) + self.assertFalse(res) + + # One thing to note about Jinja is that by default it would not raise + # an exception on KeyError inside the expression, it will consider + # value to be None. Same with NameError, it won't return an original + # expression (which by itself seems confusing). Jinja allows us to + # change behavior in both cases by switching to StrictUndefined, but + # either one or the other will surely suffer. + + self.assertRaises( + exc.JinjaEvaluationException, + self._evaluator.evaluate, + '_.wrong_key', + DATA + ) + + self.assertRaises( + exc.JinjaEvaluationException, + self._evaluator.evaluate, + 'invalid_expression_string', + DATA + ) + + def test_select_result(self): + res = self._evaluator.evaluate( + '_.servers|selectattr("name", "equalto", "ubuntu")', + SERVERS + ) + item = list(res)[0] + self.assertEqual({'name': 'ubuntu'}, item) + + def test_function_string(self): + self.assertEqual('3', self._evaluator.evaluate('_|string', '3')) + self.assertEqual('3', self._evaluator.evaluate('_|string', 3)) + + def test_function_len(self): + self.assertEqual(3, + self._evaluator.evaluate('_|length', 'hey')) + data = [{'some': 'thing'}] + + self.assertEqual( + 1, + self._evaluator.evaluate( + '_|selectattr("some", "equalto", "thing")|list|length', + data + ) + ) + + def test_validate(self): + self._evaluator.validate('abc') + self._evaluator.validate('1') + self._evaluator.validate('1 + 2') + self._evaluator.validate('_.a1') + self._evaluator.validate('_.a1 * _.a2') + + def test_validate_failed(self): + self.assertRaises(exc.JinjaGrammarException, + self._evaluator.validate, + '*') + + self.assertRaises(exc.JinjaEvaluationException, + self._evaluator.validate, + [1, 2, 3]) + + self.assertRaises(exc.JinjaEvaluationException, + self._evaluator.validate, + {'a': 1}) + + def test_function_json_pp(self): + self.assertEqual('"3"', self._evaluator.evaluate('json_pp(_)', '3')) + self.assertEqual('3', self._evaluator.evaluate('json_pp(_)', 3)) + self.assertEqual( + '[\n 1,\n 2\n]', + self._evaluator.evaluate('json_pp(_)', [1, 2]) + ) + self.assertEqual( + '{\n "a": "b"\n}', + self._evaluator.evaluate('json_pp(_)', {'a': 'b'}) + ) + self.assertEqual( + '"Mistral\nis\nawesome"', + self._evaluator.evaluate( + 'json_pp(_)', '\n'.join(['Mistral', 'is', 'awesome']) + ) + ) + + def test_filter_json_pp(self): + self.assertEqual('"3"', self._evaluator.evaluate('_|json_pp', '3')) + self.assertEqual('3', self._evaluator.evaluate('_|json_pp', 3)) + self.assertEqual( + '[\n 1,\n 2\n]', + self._evaluator.evaluate('_|json_pp', [1, 2]) + ) + self.assertEqual( + '{\n "a": "b"\n}', + self._evaluator.evaluate('_|json_pp', {'a': 'b'}) + ) + self.assertEqual( + '"Mistral\nis\nawesome"', + self._evaluator.evaluate( + '_|json_pp', '\n'.join(['Mistral', 'is', 'awesome']) + ) + ) + + def test_function_uuid(self): + uuid = self._evaluator.evaluate('uuid()', {}) + + self.assertTrue(utils.is_valid_uuid(uuid)) + + def test_filter_uuid(self): + uuid = self._evaluator.evaluate('_|uuid', '3') + + self.assertTrue(utils.is_valid_uuid(uuid)) + + def test_function_env(self): + ctx = {'__env': 'some'} + self.assertEqual(ctx['__env'], self._evaluator.evaluate('env()', ctx)) + + def test_filter_env(self): + ctx = {'__env': 'some'} + self.assertEqual(ctx['__env'], self._evaluator.evaluate('_|env', ctx)) + + @mock.patch('mistral.db.v2.api.get_task_executions') + @mock.patch('mistral.workflow.data_flow.get_task_execution_result') + def test_filter_task_without_taskexecution(self, task_execution_result, + task_executions): + task = mock.MagicMock(return_value={}) + task_executions.return_value = [task] + ctx = { + '__task_execution': None, + '__execution': { + 'id': 'some' + } + } + + result = self._evaluator.evaluate('_|task("some")', ctx) + + self.assertEqual({ + 'id': task.id, + 'name': task.name, + 'published': task.published, + 'result': task_execution_result(), + 'spec': task.spec, + 'state': task.state, + 'state_info': task.state_info + }, result) + + @mock.patch('mistral.db.v2.api.get_task_execution') + @mock.patch('mistral.workflow.data_flow.get_task_execution_result') + def test_filter_task_with_taskexecution(self, task_execution_result, + task_execution): + ctx = { + '__task_execution': { + 'id': 'some', + 'name': 'some' + } + } + + result = self._evaluator.evaluate('_|task("some")', ctx) + + self.assertEqual({ + 'id': task_execution().id, + 'name': task_execution().name, + 'published': task_execution().published, + 'result': task_execution_result(), + 'spec': task_execution().spec, + 'state': task_execution().state, + 'state_info': task_execution().state_info + }, result) + + @mock.patch('mistral.db.v2.api.get_task_execution') + @mock.patch('mistral.workflow.data_flow.get_task_execution_result') + def test_function_task(self, task_execution_result, task_execution): + ctx = { + '__task_execution': { + 'id': 'some', + 'name': 'some' + } + } + + result = self._evaluator.evaluate('task("some")', ctx) + + self.assertEqual({ + 'id': task_execution().id, + 'name': task_execution().name, + 'published': task_execution().published, + 'result': task_execution_result(), + 'spec': task_execution().spec, + 'state': task_execution().state, + 'state_info': task_execution().state_info + }, result) + + @mock.patch('mistral.db.v2.api.get_workflow_execution') + def test_filter_execution(self, workflow_execution): + wf_ex = mock.MagicMock(return_value={}) + workflow_execution.return_value = wf_ex + ctx = { + '__execution': { + 'id': 'some' + } + } + + result = self._evaluator.evaluate('_|execution', ctx) + + self.assertEqual({ + 'id': wf_ex.id, + 'name': wf_ex.name, + 'spec': wf_ex.spec, + 'input': wf_ex.input, + 'params': wf_ex.params + }, result) + + @mock.patch('mistral.db.v2.api.get_workflow_execution') + def test_function_execution(self, workflow_execution): + wf_ex = mock.MagicMock(return_value={}) + workflow_execution.return_value = wf_ex + ctx = { + '__execution': { + 'id': 'some' + } + } + + result = self._evaluator.evaluate('execution()', ctx) + + self.assertEqual({ + 'id': wf_ex.id, + 'name': wf_ex.name, + 'spec': wf_ex.spec, + 'input': wf_ex.input, + 'params': wf_ex.params + }, result) + + +class InlineJinjaEvaluatorTest(base.BaseTest): + def setUp(self): + super(InlineJinjaEvaluatorTest, self).setUp() + + self._evaluator = expr.InlineJinjaEvaluator() + + def test_multiple_placeholders(self): + expr_str = """ + Statistics for tenant "{{ _.project_id }}" + + Number of virtual machines: {{ _.vm_count }} + Number of active virtual machines: {{ _.active_vm_count }} + Number of networks: {{ _.net_count }} + + -- Sincerely, Mistral Team. + """ + + result = self._evaluator.evaluate( + expr_str, + { + 'project_id': '1-2-3-4', + 'vm_count': 28, + 'active_vm_count': 0, + 'net_count': 1 + } + ) + + expected_result = """ + Statistics for tenant "1-2-3-4" + + Number of virtual machines: 28 + Number of active virtual machines: 0 + Number of networks: 1 + + -- Sincerely, Mistral Team. + """ + + self.assertEqual(expected_result, result) + + def test_block_placeholders(self): + expr_str = """ + Statistics for tenant "{{ _.project_id }}" + + Number of virtual machines: {{ _.vm_count }} + {% if _.active_vm_count %} + Number of active virtual machines: {{ _.active_vm_count }} + {% endif %} + Number of networks: {{ _.net_count }} + + -- Sincerely, Mistral Team. + """ + + result = self._evaluator.evaluate( + expr_str, + { + 'project_id': '1-2-3-4', + 'vm_count': 28, + 'active_vm_count': 0, + 'net_count': 1 + } + ) + + expected_result = """ + Statistics for tenant "1-2-3-4" + + Number of virtual machines: 28 + Number of networks: 1 + + -- Sincerely, Mistral Team. + """ + + self.assertEqual(expected_result, result) + + def test_single_value_casting(self): + self.assertEqual(3, self._evaluator.evaluate('{{ _ }}', 3)) + self.assertEqual('33', self._evaluator.evaluate('{{ _ }}{{ _ }}', 3)) + + def test_function_string(self): + self.assertEqual('3', self._evaluator.evaluate('{{ _|string }}', '3')) + self.assertEqual('3', self._evaluator.evaluate('{{ _|string }}', 3)) + + def test_validate(self): + self._evaluator.validate('There is no expression.') + self._evaluator.validate('{{ abc }}') + self._evaluator.validate('{{ 1 }}') + self._evaluator.validate('{{ 1 + 2 }}') + self._evaluator.validate('{{ _.a1 }}') + self._evaluator.validate('{{ _.a1 * _.a2 }}') + self._evaluator.validate('{{ _.a1 }} is {{ _.a2 }}') + self._evaluator.validate('The value is {{ _.a1 }}.') + + def test_validate_failed(self): + self.assertRaises(exc.JinjaGrammarException, + self._evaluator.validate, + 'The value is {{ * }}.') + + self.assertRaises(exc.JinjaEvaluationException, + self._evaluator.validate, + [1, 2, 3]) + + self.assertRaises(exc.JinjaEvaluationException, + self._evaluator.validate, + {'a': 1}) diff --git a/mistral/tests/unit/expressions/test_yaql_expression.py b/mistral/tests/unit/expressions/test_yaql_expression.py new file mode 100644 index 000000000..c4cf17cda --- /dev/null +++ b/mistral/tests/unit/expressions/test_yaql_expression.py @@ -0,0 +1,213 @@ +# Copyright 2013 - Mirantis, Inc. +# Copyright 2015 - StackStorm, Inc. +# Copyright 2016 - Brocade Communications Systems, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from mistral import exceptions as exc +from mistral.expressions import yaql_expression as expr +from mistral.tests.unit import base +from mistral import utils + +DATA = { + "server": { + "id": "03ea824a-aa24-4105-9131-66c48ae54acf", + "name": "cloud-fedora", + "status": "ACTIVE" + }, + "status": "OK" +} + +SERVERS = { + "servers": [ + {'name': 'centos'}, + {'name': 'ubuntu'}, + {'name': 'fedora'} + ] +} + + +class YaqlEvaluatorTest(base.BaseTest): + def setUp(self): + super(YaqlEvaluatorTest, self).setUp() + + self._evaluator = expr.YAQLEvaluator() + + def test_expression_result(self): + res = self._evaluator.evaluate('$.server', DATA) + self.assertEqual({ + 'id': "03ea824a-aa24-4105-9131-66c48ae54acf", + 'name': 'cloud-fedora', + 'status': 'ACTIVE' + }, res) + + res = self._evaluator.evaluate('$.server.id', DATA) + self.assertEqual('03ea824a-aa24-4105-9131-66c48ae54acf', res) + + res = self._evaluator.evaluate("$.server.status = 'ACTIVE'", DATA) + self.assertTrue(res) + + def test_wrong_expression(self): + res = self._evaluator.evaluate("$.status = 'Invalid value'", DATA) + self.assertFalse(res) + + self.assertRaises( + exc.YaqlEvaluationException, + self._evaluator.evaluate, + '$.wrong_key', + DATA + ) + + expression_str = 'invalid_expression_string' + res = self._evaluator.evaluate(expression_str, DATA) + self.assertEqual(expression_str, res) + + def test_select_result(self): + res = self._evaluator.evaluate( + '$.servers.where($.name = ubuntu)', + SERVERS + ) + item = list(res)[0] + self.assertEqual({'name': 'ubuntu'}, item) + + def test_function_string(self): + self.assertEqual('3', self._evaluator.evaluate('str($)', '3')) + self.assertEqual('3', self._evaluator.evaluate('str($)', 3)) + + def test_function_len(self): + self.assertEqual(3, self._evaluator.evaluate('len($)', 'hey')) + data = [{'some': 'thing'}] + + self.assertEqual( + 1, + self._evaluator.evaluate('$.where($.some = thing).len()', data) + ) + + def test_validate(self): + self._evaluator.validate('abc') + self._evaluator.validate('1') + self._evaluator.validate('1 + 2') + self._evaluator.validate('$.a1') + self._evaluator.validate('$.a1 * $.a2') + + def test_validate_failed(self): + self.assertRaises(exc.YaqlGrammarException, + self._evaluator.validate, + '*') + + self.assertRaises(exc.YaqlGrammarException, + self._evaluator.validate, + [1, 2, 3]) + + self.assertRaises(exc.YaqlGrammarException, + self._evaluator.validate, + {'a': 1}) + + def test_function_json_pp(self): + self.assertEqual('"3"', self._evaluator.evaluate('json_pp($)', '3')) + self.assertEqual('3', self._evaluator.evaluate('json_pp($)', 3)) + self.assertEqual( + '[\n 1,\n 2\n]', + self._evaluator.evaluate('json_pp($)', [1, 2]) + ) + self.assertEqual( + '{\n "a": "b"\n}', + self._evaluator.evaluate('json_pp($)', {'a': 'b'}) + ) + self.assertEqual( + '"Mistral\nis\nawesome"', + self._evaluator.evaluate( + 'json_pp($)', '\n'.join(['Mistral', 'is', 'awesome']) + ) + ) + + def test_function_uuid(self): + uuid = self._evaluator.evaluate('uuid()', {}) + + self.assertTrue(utils.is_valid_uuid(uuid)) + + def test_function_env(self): + ctx = {'__env': 'some'} + + self.assertEqual(ctx['__env'], self._evaluator.evaluate('env()', ctx)) + + +class InlineYAQLEvaluatorTest(base.BaseTest): + def setUp(self): + super(InlineYAQLEvaluatorTest, self).setUp() + + self._evaluator = expr.InlineYAQLEvaluator() + + def test_multiple_placeholders(self): + expr_str = """ + Statistics for tenant "<% $.project_id %>" + + Number of virtual machines: <% $.vm_count %> + Number of active virtual machines: <% $.active_vm_count %> + Number of networks: <% $.net_count %> + + -- Sincerely, Mistral Team. + """ + + result = self._evaluator.evaluate( + expr_str, + { + 'project_id': '1-2-3-4', + 'vm_count': 28, + 'active_vm_count': 0, + 'net_count': 1 + } + ) + + expected_result = """ + Statistics for tenant "1-2-3-4" + + Number of virtual machines: 28 + Number of active virtual machines: 0 + Number of networks: 1 + + -- Sincerely, Mistral Team. + """ + + self.assertEqual(expected_result, result) + + def test_single_value_casting(self): + self.assertEqual(3, self._evaluator.evaluate('<% $ %>', 3)) + self.assertEqual('33', self._evaluator.evaluate('<% $ %><% $ %>', 3)) + + def test_function_string(self): + self.assertEqual('3', self._evaluator.evaluate('<% str($) %>', '3')) + self.assertEqual('3', self._evaluator.evaluate('<% str($) %>', 3)) + + def test_validate(self): + self._evaluator.validate('There is no expression.') + self._evaluator.validate('<% abc %>') + self._evaluator.validate('<% 1 %>') + self._evaluator.validate('<% 1 + 2 %>') + self._evaluator.validate('<% $.a1 %>') + self._evaluator.validate('<% $.a1 * $.a2 %>') + self._evaluator.validate('<% $.a1 %> is <% $.a2 %>') + self._evaluator.validate('The value is <% $.a1 %>.') + + def test_validate_failed(self): + self.assertRaises(exc.YaqlGrammarException, + self._evaluator.validate, + 'The value is <% * %>.') + + self.assertRaises(exc.YaqlEvaluationException, + self._evaluator.validate, + [1, 2, 3]) + + self.assertRaises(exc.YaqlEvaluationException, + self._evaluator.validate, + {'a': 1}) diff --git a/mistral/tests/unit/test_expressions.py b/mistral/tests/unit/test_expressions.py index 761e37c6a..62ac6da89 100644 --- a/mistral/tests/unit/test_expressions.py +++ b/mistral/tests/unit/test_expressions.py @@ -1,5 +1,6 @@ # Copyright 2013 - Mirantis, Inc. # Copyright 2015 - StackStorm, Inc. +# Copyright 2016 - Brocade Communications Systems, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -35,168 +36,6 @@ SERVERS = { } -class YaqlEvaluatorTest(base.BaseTest): - def setUp(self): - super(YaqlEvaluatorTest, self).setUp() - - self._evaluator = expr.YAQLEvaluator() - - def test_expression_result(self): - res = self._evaluator.evaluate('$.server', DATA) - self.assertEqual({ - 'id': "03ea824a-aa24-4105-9131-66c48ae54acf", - 'name': 'cloud-fedora', - 'status': 'ACTIVE' - }, res) - - res = self._evaluator.evaluate('$.server.id', DATA) - self.assertEqual('03ea824a-aa24-4105-9131-66c48ae54acf', res) - - res = self._evaluator.evaluate("$.server.status = 'ACTIVE'", DATA) - self.assertTrue(res) - - def test_wrong_expression(self): - res = self._evaluator.evaluate("$.status = 'Invalid value'", DATA) - self.assertFalse(res) - - self.assertRaises( - exc.YaqlEvaluationException, - self._evaluator.evaluate, - '$.wrong_key', - DATA - ) - - expression_str = 'invalid_expression_string' - res = self._evaluator.evaluate(expression_str, DATA) - self.assertEqual(expression_str, res) - - def test_select_result(self): - res = self._evaluator.evaluate( - '$.servers.where($.name = ubuntu)', - SERVERS - ) - item = list(res)[0] - self.assertEqual({'name': 'ubuntu'}, item) - - def test_function_string(self): - self.assertEqual('3', self._evaluator.evaluate('str($)', '3')) - self.assertEqual('3', self._evaluator.evaluate('str($)', 3)) - - def test_function_len(self): - self.assertEqual(3, self._evaluator.evaluate('len($)', 'hey')) - data = [{'some': 'thing'}] - - self.assertEqual( - 1, - self._evaluator.evaluate('$.where($.some = thing).len()', data) - ) - - def test_validate(self): - self._evaluator.validate('abc') - self._evaluator.validate('1') - self._evaluator.validate('1 + 2') - self._evaluator.validate('$.a1') - self._evaluator.validate('$.a1 * $.a2') - - def test_validate_failed(self): - self.assertRaises(exc.YaqlGrammarException, - self._evaluator.validate, - '*') - - self.assertRaises(exc.YaqlGrammarException, - self._evaluator.validate, - [1, 2, 3]) - - self.assertRaises(exc.YaqlGrammarException, - self._evaluator.validate, - {'a': 1}) - - def test_json_pp(self): - self.assertEqual('"3"', self._evaluator.evaluate('json_pp($)', '3')) - self.assertEqual('3', self._evaluator.evaluate('json_pp($)', 3)) - self.assertEqual( - '[\n 1,\n 2\n]', - self._evaluator.evaluate('json_pp($)', [1, 2]) - ) - self.assertEqual( - '{\n "a": "b"\n}', - self._evaluator.evaluate('json_pp($)', {'a': 'b'}) - ) - self.assertEqual( - '"Mistral\nis\nawesome"', - self._evaluator.evaluate( - 'json_pp($)', '\n'.join(['Mistral', 'is', 'awesome']) - ) - ) - - -class InlineYAQLEvaluatorTest(base.BaseTest): - def setUp(self): - super(InlineYAQLEvaluatorTest, self).setUp() - - self._evaluator = expr.InlineYAQLEvaluator() - - def test_multiple_placeholders(self): - expr_str = """ - Statistics for tenant "<% $.project_id %>" - - Number of virtual machines: <% $.vm_count %> - Number of active virtual machines: <% $.active_vm_count %> - Number of networks: <% $.net_count %> - - -- Sincerely, Mistral Team. - """ - - result = self._evaluator.evaluate( - expr_str, - { - 'project_id': '1-2-3-4', - 'vm_count': 28, - 'active_vm_count': 0, - 'net_count': 1 - } - ) - - expected_result = """ - Statistics for tenant "1-2-3-4" - - Number of virtual machines: 28 - Number of active virtual machines: 0 - Number of networks: 1 - - -- Sincerely, Mistral Team. - """ - - self.assertEqual(expected_result, result) - - def test_function_string(self): - self.assertEqual('3', self._evaluator.evaluate('<% str($) %>', '3')) - self.assertEqual('3', self._evaluator.evaluate('<% str($) %>', 3)) - - def test_validate(self): - self._evaluator.validate('There is no expression.') - self._evaluator.validate('<% abc %>') - self._evaluator.validate('<% 1 %>') - self._evaluator.validate('<% 1 + 2 %>') - self._evaluator.validate('<% $.a1 %>') - self._evaluator.validate('<% $.a1 * $.a2 %>') - self._evaluator.validate('<% $.a1 %> is <% $.a2 %>') - self._evaluator.validate('The value is <% $.a1 %>.') - - def test_validate_failed(self): - self.assertRaises(exc.YaqlGrammarException, - self._evaluator.validate, - 'The value is <% * %>.') - - self.assertRaises(exc.YaqlEvaluationException, - self._evaluator.validate, - [1, 2, 3]) - - self.assertRaises(exc.YaqlEvaluationException, - self._evaluator.validate, - {'a': 1}) - - class ExpressionsTest(base.BaseTest): def test_evaluate_complex_expressions(self): data = { @@ -327,3 +166,26 @@ class ExpressionsTest(base.BaseTest): expected = 'mysql://admin:secrete@vm1234.example.com/test' self.assertEqual(expected, applied['conn']) + + def test_validate_jinja_with_yaql_context(self): + self.assertRaises(exc.JinjaGrammarException, + expr.validate, + '{{ $ }}') + + def test_validate_mixing_jinja_and_yaql(self): + self.assertRaises(exc.ExpressionGrammarException, + expr.validate, + '<% $.a %>{{ _.a }}') + + self.assertRaises(exc.ExpressionGrammarException, + expr.validate, + '{{ _.a }}<% $.a %>') + + def test_evaluate_mixing_jinja_and_yaql(self): + actual = expr.evaluate('<% $.a %>{{ _.a }}', {'a': 'b'}) + + self.assertEqual('<% $.a %>b', actual) + + actual = expr.evaluate('{{ _.a }}<% $.a %>', {'a': 'b'}) + + self.assertEqual('b<% $.a %>', actual) diff --git a/mistral/tests/unit/workbook/v2/test_actions.py b/mistral/tests/unit/workbook/v2/test_actions.py index 4e0c56123..d599a86d7 100644 --- a/mistral/tests/unit/workbook/v2/test_actions.py +++ b/mistral/tests/unit/workbook/v2/test_actions.py @@ -1,4 +1,5 @@ # Copyright 2015 - StackStorm, Inc. +# Copyright 2016 - Brocade Communications Systems, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -37,7 +38,10 @@ class ActionSpecValidation(base.WorkbookSpecValidationTestCase): ({'actions': {'a1': {'base': 'std.echo output="foo"'}}}, False), ({'actions': {'a1': {'base': 'std.echo output="<% $.x %>"'}}}, False), - ({'actions': {'a1': {'base': 'std.echo output="<% * %>"'}}}, True) + ({'actions': {'a1': {'base': 'std.echo output="<% * %>"'}}}, True), + ({'actions': {'a1': {'base': 'std.echo output="{{ _.x }}"'}}}, + False), + ({'actions': {'a1': {'base': 'std.echo output="{{ * }}"'}}}, True) ] for actions, expect_error in tests: @@ -49,7 +53,9 @@ class ActionSpecValidation(base.WorkbookSpecValidationTestCase): ({'base-input': {}}, True), ({'base-input': None}, True), ({'base-input': {'k1': 'v1', 'k2': '<% $.v2 %>'}}, False), - ({'base-input': {'k1': 'v1', 'k2': '<% * %>'}}, True) + ({'base-input': {'k1': 'v1', 'k2': '<% * %>'}}, True), + ({'base-input': {'k1': 'v1', 'k2': '{{ _.v2 }}'}}, False), + ({'base-input': {'k1': 'v1', 'k2': '{{ * }}'}}, True) ] actions = { @@ -100,6 +106,8 @@ class ActionSpecValidation(base.WorkbookSpecValidationTestCase): ({'output': 'foobar'}, False), ({'output': '<% $.x %>'}, False), ({'output': '<% * %>'}, True), + ({'output': '{{ _.x }}'}, False), + ({'output': '{{ * }}'}, True), ({'output': ['v1']}, False), ({'output': {'k1': 'v1'}}, False) ] diff --git a/mistral/tests/unit/workbook/v2/test_tasks.py b/mistral/tests/unit/workbook/v2/test_tasks.py index 964970a8c..d97aa4e5f 100644 --- a/mistral/tests/unit/workbook/v2/test_tasks.py +++ b/mistral/tests/unit/workbook/v2/test_tasks.py @@ -1,5 +1,6 @@ # Copyright 2015 - Huawei Technologies Co. Ltd # Copyright 2015 - StackStorm, Inc. +# Copyright 2016 - Brocade Communications Systems, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -55,12 +56,18 @@ class TaskSpecValidation(v2_base.WorkflowSpecValidationTestCase): ({'action': 'std.http url=<% $.url %>'}, False), ({'action': 'std.http url=<% $.url %> timeout=<% $.t %>'}, False), ({'action': 'std.http url=<% * %>'}, True), + ({'action': 'std.http url={{ _.url }}'}, False), + ({'action': 'std.http url={{ _.url }} timeout={{ _.t }}'}, False), + ({'action': 'std.http url={{ $ }}'}, True), ({'workflow': 'test.wf'}, False), ({'workflow': 'test.wf k1="v1"'}, False), ({'workflow': 'test.wf k1="v1" k2="v2"'}, False), ({'workflow': 'test.wf k1=<% $.v1 %>'}, False), ({'workflow': 'test.wf k1=<% $.v1 %> k2=<% $.v2 %>'}, False), ({'workflow': 'test.wf k1=<% * %>'}, True), + ({'workflow': 'test.wf k1={{ _.v1 }}'}, False), + ({'workflow': 'test.wf k1={{ _.v1 }} k2={{ _.v2 }}'}, False), + ({'workflow': 'test.wf k1={{ $ }}'}, True), ({'action': 'std.noop', 'workflow': 'test.wf'}, True), ({'action': 123}, True), ({'workflow': 123}, True), @@ -87,7 +94,10 @@ class TaskSpecValidation(v2_base.WorkflowSpecValidationTestCase): ({'input': {'k1': 'v1'}}, False), ({'input': {'k1': '<% $.v1 %>'}}, False), ({'input': {'k1': '<% 1 + 2 %>'}}, False), - ({'input': {'k1': '<% * %>'}}, True) + ({'input': {'k1': '<% * %>'}}, True), + ({'input': {'k1': '{{ _.v1 }}'}}, False), + ({'input': {'k1': '{{ 1 + 2 }}'}}, False), + ({'input': {'k1': '{{ * }}'}}, True) ] for task_input, expect_error in tests: @@ -116,7 +126,15 @@ class TaskSpecValidation(v2_base.WorkflowSpecValidationTestCase): ({'with-items': ['x in <% $.y %>', 'i in [1, 2, 3]']}, False), ({'with-items': ['x in <% $.y %>', 'i in <% $.j %>']}, False), ({'with-items': ['x in <% * %>']}, True), - ({'with-items': ['x in <% $.y %>', 'i in <% * %>']}, True) + ({'with-items': ['x in <% $.y %>', 'i in <% * %>']}, True), + ({'with-items': '{{ _.y }}'}, True), + ({'with-items': 'x in {{ _.y }}'}, False), + ({'with-items': ['x in [1, 2, 3]']}, False), + ({'with-items': ['x in {{ _.y }}']}, False), + ({'with-items': ['x in {{ _.y }}', 'i in [1, 2, 3]']}, False), + ({'with-items': ['x in {{ _.y }}', 'i in {{ _.j }}']}, False), + ({'with-items': ['x in {{ * }}']}, True), + ({'with-items': ['x in {{ _.y }}', 'i in {{ * }}']}, True) ] for with_item, expect_error in tests: @@ -136,7 +154,10 @@ class TaskSpecValidation(v2_base.WorkflowSpecValidationTestCase): ({'publish': {'k1': 'v1'}}, False), ({'publish': {'k1': '<% $.v1 %>'}}, False), ({'publish': {'k1': '<% 1 + 2 %>'}}, False), - ({'publish': {'k1': '<% * %>'}}, True) + ({'publish': {'k1': '<% * %>'}}, True), + ({'publish': {'k1': '{{ _.v1 }}'}}, False), + ({'publish': {'k1': '{{ 1 + 2 }}'}}, False), + ({'publish': {'k1': '{{ * }}'}}, True) ] for output, expect_error in tests: @@ -164,39 +185,61 @@ class TaskSpecValidation(v2_base.WorkflowSpecValidationTestCase): ({'retry': {'count': '<% * %>', 'delay': 1}}, True), ({'retry': {'count': 3, 'delay': '<% 1 %>'}}, False), ({'retry': {'count': 3, 'delay': '<% * %>'}}, True), + ({'retry': { + 'continue-on': '{{ 1 }}', 'delay': 2, + 'break-on': '{{ 1 }}', 'count': 2 + }}, False), + ({'retry': { + 'count': 3, 'delay': 1, 'continue-on': '{{ 1 }}' + }}, False), + ({'retry': {'count': '{{ 3 }}', 'delay': 1}}, False), + ({'retry': {'count': '{{ * }}', 'delay': 1}}, True), + ({'retry': {'count': 3, 'delay': '{{ 1 }}'}}, False), + ({'retry': {'count': 3, 'delay': '{{ * }}'}}, True), ({'retry': {'count': -3, 'delay': 1}}, True), ({'retry': {'count': 3, 'delay': -1}}, True), ({'retry': {'count': '3', 'delay': 1}}, True), ({'retry': {'count': 3, 'delay': '1'}}, True), ({'retry': 'count=3 delay=1 break-on=<% false %>'}, False), + ({'retry': 'count=3 delay=1 break-on={{ false }}'}, False), ({'retry': 'count=3 delay=1'}, False), ({'retry': 'coun=3 delay=1'}, True), ({'retry': None}, True), ({'wait-before': 1}, False), ({'wait-before': '<% 1 %>'}, False), ({'wait-before': '<% * %>'}, True), + ({'wait-before': '{{ 1 }}'}, False), + ({'wait-before': '{{ * }}'}, True), ({'wait-before': -1}, True), ({'wait-before': 1.0}, True), ({'wait-before': '1'}, True), ({'wait-after': 1}, False), ({'wait-after': '<% 1 %>'}, False), ({'wait-after': '<% * %>'}, True), + ({'wait-after': '{{ 1 }}'}, False), + ({'wait-after': '{{ * }}'}, True), ({'wait-after': -1}, True), ({'wait-after': 1.0}, True), ({'wait-after': '1'}, True), ({'timeout': 300}, False), ({'timeout': '<% 300 %>'}, False), ({'timeout': '<% * %>'}, True), + ({'timeout': '{{ 300 }}'}, False), + ({'timeout': '{{ * }}'}, True), ({'timeout': -300}, True), ({'timeout': 300.0}, True), ({'timeout': '300'}, True), ({'pause-before': False}, False), ({'pause-before': '<% False %>'}, False), ({'pause-before': '<% * %>'}, True), + ({'pause-before': '{{ False }}'}, False), + ({'pause-before': '{{ * }}'}, True), ({'pause-before': 'False'}, True), ({'concurrency': 10}, False), ({'concurrency': '<% 10 %>'}, False), ({'concurrency': '<% * %>'}, True), + ({'concurrency': '{{ 10 }}'}, False), + ({'concurrency': '{{ * }}'}, True), ({'concurrency': -10}, True), ({'concurrency': 10.0}, True), ({'concurrency': '10'}, True) @@ -218,6 +261,10 @@ class TaskSpecValidation(v2_base.WorkflowSpecValidationTestCase): ({'on-success': [{'email': '<% 1 %>'}, 'echo']}, False), ({'on-success': [{'email': '<% $.v1 in $.v2 %>'}]}, False), ({'on-success': [{'email': '<% * %>'}]}, True), + ({'on-success': [{'email': '{{ 1 }}'}]}, False), + ({'on-success': [{'email': '{{ 1 }}'}, 'echo']}, False), + ({'on-success': [{'email': '{{ _.v1 in _.v2 }}'}]}, False), + ({'on-success': [{'email': '{{ * }}'}]}, True), ({'on-success': 'email'}, False), ({'on-success': None}, True), ({'on-success': ['']}, True), @@ -229,6 +276,10 @@ class TaskSpecValidation(v2_base.WorkflowSpecValidationTestCase): ({'on-error': [{'email': '<% 1 %>'}, 'echo']}, False), ({'on-error': [{'email': '<% $.v1 in $.v2 %>'}]}, False), ({'on-error': [{'email': '<% * %>'}]}, True), + ({'on-error': [{'email': '{{ 1 }}'}]}, False), + ({'on-error': [{'email': '{{ 1 }}'}, 'echo']}, False), + ({'on-error': [{'email': '{{ _.v1 in _.v2 }}'}]}, False), + ({'on-error': [{'email': '{{ * }}'}]}, True), ({'on-error': 'email'}, False), ({'on-error': None}, True), ({'on-error': ['']}, True), @@ -240,6 +291,10 @@ class TaskSpecValidation(v2_base.WorkflowSpecValidationTestCase): ({'on-complete': [{'email': '<% 1 %>'}, 'echo']}, False), ({'on-complete': [{'email': '<% $.v1 in $.v2 %>'}]}, False), ({'on-complete': [{'email': '<% * %>'}]}, True), + ({'on-complete': [{'email': '{{ 1 }}'}]}, False), + ({'on-complete': [{'email': '{{ 1 }}'}, 'echo']}, False), + ({'on-complete': [{'email': '{{ _.v1 in _.v2 }}'}]}, False), + ({'on-complete': [{'email': '{{ * }}'}]}, True), ({'on-complete': 'email'}, False), ({'on-complete': None}, True), ({'on-complete': ['']}, True), @@ -322,7 +377,10 @@ class TaskSpecValidation(v2_base.WorkflowSpecValidationTestCase): ({'keep-result': False}, False), ({'keep-result': "<% 'a' in $.val %>"}, False), ({'keep-result': '<% 1 + 2 %>'}, False), - ({'keep-result': '<% * %>'}, True) + ({'keep-result': '<% * %>'}, True), + ({'keep-result': "{{ 'a' in _.val }}"}, False), + ({'keep-result': '{{ 1 + 2 }}'}, False), + ({'keep-result': '{{ * }}'}, True) ] for keep_result, expect_error in tests: diff --git a/mistral/tests/unit/workbook/v2/test_workflows.py b/mistral/tests/unit/workbook/v2/test_workflows.py index 70cc6c490..849f1a66d 100644 --- a/mistral/tests/unit/workbook/v2/test_workflows.py +++ b/mistral/tests/unit/workbook/v2/test_workflows.py @@ -1,4 +1,5 @@ # Copyright 2015 - StackStorm, Inc. +# Copyright 2016 - Brocade Communications Systems, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -234,6 +235,9 @@ class WorkflowSpecValidation(base.WorkflowSpecValidationTestCase): ({'vars': {'v1': '<% $.input_var1 %>'}}, False), ({'vars': {'v1': '<% 1 + 2 %>'}}, False), ({'vars': {'v1': '<% * %>'}}, True), + ({'vars': {'v1': '{{ _.input_var1 }}'}}, False), + ({'vars': {'v1': '{{ 1 + 2 }}'}}, False), + ({'vars': {'v1': '{{ * }}'}}, True), ({'vars': []}, True), ({'vars': 'whatever'}, True), ({'vars': None}, True), @@ -280,6 +284,10 @@ class WorkflowSpecValidation(base.WorkflowSpecValidationTestCase): ({'on-success': [{'email': '<% 1 %>'}, 'echo']}, False), ({'on-success': [{'email': '<% $.v1 in $.v2 %>'}]}, False), ({'on-success': [{'email': '<% * %>'}]}, True), + ({'on-success': [{'email': '{{ 1 }}'}]}, False), + ({'on-success': [{'email': '{{ 1 }}'}, 'echo']}, False), + ({'on-success': [{'email': '{{ _.v1 in _.v2 }}'}]}, False), + ({'on-success': [{'email': '{{ * }}'}]}, True), ({'on-success': 'email'}, False), ({'on-success': None}, True), ({'on-success': ['']}, True), @@ -291,6 +299,10 @@ class WorkflowSpecValidation(base.WorkflowSpecValidationTestCase): ({'on-error': [{'email': '<% 1 %>'}, 'echo']}, False), ({'on-error': [{'email': '<% $.v1 in $.v2 %>'}]}, False), ({'on-error': [{'email': '<% * %>'}]}, True), + ({'on-error': [{'email': '{{ 1 }}'}]}, False), + ({'on-error': [{'email': '{{ 1 }}'}, 'echo']}, False), + ({'on-error': [{'email': '{{ _.v1 in _.v2 }}'}]}, False), + ({'on-error': [{'email': '{{ * }}'}]}, True), ({'on-error': 'email'}, False), ({'on-error': None}, True), ({'on-error': ['']}, True), @@ -302,6 +314,10 @@ class WorkflowSpecValidation(base.WorkflowSpecValidationTestCase): ({'on-complete': [{'email': '<% 1 %>'}, 'echo']}, False), ({'on-complete': [{'email': '<% $.v1 in $.v2 %>'}]}, False), ({'on-complete': [{'email': '<% * %>'}]}, True), + ({'on-complete': [{'email': '{{ 1 }}'}]}, False), + ({'on-complete': [{'email': '{{ 1 }}'}, 'echo']}, False), + ({'on-complete': [{'email': '{{ _.v1 in _.v2 }}'}]}, False), + ({'on-complete': [{'email': '{{ * }}'}]}, True), ({'on-complete': 'email'}, False), ({'on-complete': None}, True), ({'on-complete': ['']}, True), @@ -321,6 +337,10 @@ class WorkflowSpecValidation(base.WorkflowSpecValidationTestCase): ({'retry': {'count': '<% * %>', 'delay': 1}}, True), ({'retry': {'count': 3, 'delay': '<% 1 %>'}}, False), ({'retry': {'count': 3, 'delay': '<% * %>'}}, True), + ({'retry': {'count': '{{ 3 }}', 'delay': 1}}, False), + ({'retry': {'count': '{{ * }}', 'delay': 1}}, True), + ({'retry': {'count': 3, 'delay': '{{ 1 }}'}}, False), + ({'retry': {'count': 3, 'delay': '{{ * }}'}}, True), ({'retry': {'count': -3, 'delay': 1}}, True), ({'retry': {'count': 3, 'delay': -1}}, True), ({'retry': {'count': '3', 'delay': 1}}, True), @@ -329,28 +349,38 @@ class WorkflowSpecValidation(base.WorkflowSpecValidationTestCase): ({'wait-before': 1}, False), ({'wait-before': '<% 1 %>'}, False), ({'wait-before': '<% * %>'}, True), + ({'wait-before': '{{ 1 }}'}, False), + ({'wait-before': '{{ * }}'}, True), ({'wait-before': -1}, True), ({'wait-before': 1.0}, True), ({'wait-before': '1'}, True), ({'wait-after': 1}, False), ({'wait-after': '<% 1 %>'}, False), ({'wait-after': '<% * %>'}, True), + ({'wait-after': '{{ 1 }}'}, False), + ({'wait-after': '{{ * }}'}, True), ({'wait-after': -1}, True), ({'wait-after': 1.0}, True), ({'wait-after': '1'}, True), ({'timeout': 300}, False), ({'timeout': '<% 300 %>'}, False), ({'timeout': '<% * %>'}, True), + ({'timeout': '{{ 300 }}'}, False), + ({'timeout': '{{ * }}'}, True), ({'timeout': -300}, True), ({'timeout': 300.0}, True), ({'timeout': '300'}, True), ({'pause-before': False}, False), ({'pause-before': '<% False %>'}, False), ({'pause-before': '<% * %>'}, True), + ({'pause-before': '{{ False }}'}, False), + ({'pause-before': '{{ * }}'}, True), ({'pause-before': 'False'}, True), ({'concurrency': 10}, False), ({'concurrency': '<% 10 %>'}, False), ({'concurrency': '<% * %>'}, True), + ({'concurrency': '{{ 10 }}'}, False), + ({'concurrency': '{{ * }}'}, True), ({'concurrency': -10}, True), ({'concurrency': 10.0}, True), ({'concurrency': '10'}, True) diff --git a/mistral/utils/__init__.py b/mistral/utils/__init__.py index ee9883905..aef3c557f 100644 --- a/mistral/utils/__init__.py +++ b/mistral/utils/__init__.py @@ -2,6 +2,7 @@ # # Copyright 2013 - Mirantis, Inc. # Copyright 2015 - Huawei Technologies Co. Ltd +# Copyright 2016 - Brocade Communications Systems, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -46,6 +47,15 @@ def generate_unicode_uuid(): return six.text_type(str(uuid.uuid4())) +def is_valid_uuid(uuid_string): + try: + val = uuid.UUID(uuid_string, version=4) + except ValueError: + return False + + return val.hex == uuid_string.replace('-', '') + + def _get_greenlet_local_storage(): greenlet_id = corolocal.get_ident() diff --git a/mistral/utils/yaql_utils.py b/mistral/utils/expression_utils.py similarity index 68% rename from mistral/utils/yaql_utils.py rename to mistral/utils/expression_utils.py index 9f705c387..d215d6e1f 100644 --- a/mistral/utils/yaql_utils.py +++ b/mistral/utils/expression_utils.py @@ -1,4 +1,5 @@ # Copyright 2015 - Mirantis, Inc. +# Copyright 2016 - Brocade Communications Systems, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -12,6 +13,7 @@ # See the License for the specific language governing permissions and # limitations under the License. +from functools import partial from oslo_serialization import jsonutils from stevedore import extension @@ -21,18 +23,17 @@ from mistral.db.v2 import api as db_api from mistral import utils -ROOT_CONTEXT = None +ROOT_YAQL_CONTEXT = None def get_yaql_context(data_context): - global ROOT_CONTEXT + global ROOT_YAQL_CONTEXT - if not ROOT_CONTEXT: - ROOT_CONTEXT = yaql.create_context() + if not ROOT_YAQL_CONTEXT: + ROOT_YAQL_CONTEXT = yaql.create_context() - _register_functions(ROOT_CONTEXT) - - new_ctx = ROOT_CONTEXT.create_child_context() + _register_yaql_functions(ROOT_YAQL_CONTEXT) + new_ctx = ROOT_YAQL_CONTEXT.create_child_context() new_ctx['$'] = data_context if isinstance(data_context, dict): @@ -43,24 +44,50 @@ def get_yaql_context(data_context): return new_ctx -def _register_custom_functions(yaql_ctx): - """Register custom YAQL functions +def get_jinja_context(data_context): + new_ctx = { + '_': data_context + } - Custom YAQL functions must be added as entry points in the - 'mistral.yaql_functions' namespace - :param yaql_ctx: YAQL context object + _register_jinja_functions(new_ctx) + + if isinstance(data_context, dict): + new_ctx['__env'] = data_context.get('__env') + new_ctx['__execution'] = data_context.get('__execution') + new_ctx['__task_execution'] = data_context.get('__task_execution') + + return new_ctx + + +def get_custom_functions(): + """Get custom functions + + Retreives the list of custom evaluation functions """ + functions = dict() + mgr = extension.ExtensionManager( - namespace='mistral.yaql_functions', + namespace='mistral.expression.functions', invoke_on_load=False ) for name in mgr.names(): - yaql_function = mgr[name].plugin - yaql_ctx.register_function(yaql_function, name=name) + functions[name] = mgr[name].plugin + + return functions -def _register_functions(yaql_ctx): - _register_custom_functions(yaql_ctx) +def _register_yaql_functions(yaql_ctx): + functions = get_custom_functions() + + for name in functions: + yaql_ctx.register_function(functions[name], name=name) + + +def _register_jinja_functions(jinja_ctx): + functions = get_custom_functions() + + for name in functions: + jinja_ctx[name] = partial(functions[name], jinja_ctx['_']) # Additional YAQL functions needed by Mistral. @@ -83,9 +110,9 @@ def execution_(context): } -def json_pp_(data): +def json_pp_(context, data=None): return jsonutils.dumps( - data, + data or context, indent=4 ).replace("\\n", "\n").replace(" \n", "\n") @@ -128,5 +155,5 @@ def task_(context, task_name): } -def uuid_(context): +def uuid_(context=None): return utils.generate_unicode_uuid() diff --git a/mistral/utils/rest_utils.py b/mistral/utils/rest_utils.py index e6a943c04..065be2c59 100644 --- a/mistral/utils/rest_utils.py +++ b/mistral/utils/rest_utils.py @@ -1,6 +1,7 @@ # -*- coding: utf-8 -*- # # Copyright 2014 - Mirantis, Inc. +# Copyright 2016 - Brocade Communications Systems, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -17,6 +18,7 @@ import functools import json +from oslo_log import log as logging import pecan import six @@ -25,6 +27,8 @@ from wsme import exc as wsme_exc from mistral import exceptions as exc +LOG = logging.getLogger(__name__) + def wrap_wsme_controller_exception(func): """Decorator for controllers method. @@ -39,6 +43,7 @@ def wrap_wsme_controller_exception(func): except (exc.MistralException, exc.MistralError) as e: pecan.response.translatable_error = e + LOG.error('Error during API call: %s' % str(e)) raise wsme_exc.ClientSideError( msg=six.text_type(e), status_code=e.http_code @@ -58,6 +63,7 @@ def wrap_pecan_controller_exception(func): try: return func(*args, **kwargs) except (exc.MistralException, exc.MistralError) as e: + LOG.error('Error during API call: %s' % str(e)) return webob.Response( status=e.http_code, content_type='application/json', diff --git a/mistral/workbook/base.py b/mistral/workbook/base.py index 62ed563d8..90c19e991 100644 --- a/mistral/workbook/base.py +++ b/mistral/workbook/base.py @@ -27,7 +27,7 @@ from mistral.workbook import types CMD_PTRN = re.compile("^[\w\.]+[^=\(\s\"]*") -INLINE_YAQL = expr.INLINE_YAQL_REGEXP +EXPRESSION = '|'.join([expr.patterns[name] for name in expr.patterns]) _ALL_IN_BRACKETS = "\[.*\]\s*" _ALL_IN_QUOTES = "\"[^\"]*\"\s*" _ALL_IN_APOSTROPHES = "'[^']*'\s*" @@ -37,7 +37,7 @@ _FALSE = "false" _NULL = "null" ALL = ( - _ALL_IN_QUOTES, _ALL_IN_APOSTROPHES, INLINE_YAQL, + _ALL_IN_QUOTES, _ALL_IN_APOSTROPHES, EXPRESSION, _ALL_IN_BRACKETS, _TRUE, _FALSE, _NULL, _DIGITS ) @@ -194,7 +194,7 @@ class BaseSpec(object): """ pass - def validate_yaql_expr(self, dsl_part): + def validate_expr(self, dsl_part): if isinstance(dsl_part, six.string_types): expr.validate(dsl_part) elif isinstance(dsl_part, list): @@ -278,9 +278,10 @@ class BaseSpec(object): params = {} - for k, v in re.findall(PARAMS_PTRN, cmd_str): + for match in re.findall(PARAMS_PTRN, cmd_str): + k = match[0] # Remove embracing quotes. - v = v.strip() + v = match[1].strip() if v[0] == '"' or v[0] == "'": v = v[1:-1] else: diff --git a/mistral/workbook/types.py b/mistral/workbook/types.py index fb9733433..0158604a7 100644 --- a/mistral/workbook/types.py +++ b/mistral/workbook/types.py @@ -12,6 +12,9 @@ # See the License for the specific language governing permissions and # limitations under the License. +from mistral import expressions + + NONEMPTY_STRING = { "type": "string", "minLength": 1 @@ -34,16 +37,18 @@ POSITIVE_NUMBER = { "minimum": 0.0 } -YAQL = { - "type": "string", - "pattern": "^<%.*?%>\\s*$" +EXPRESSION = { + "oneOf": [{ + "type": "string", + "pattern": "^%s\\s*$" % expressions.patterns[name] + } for name in expressions.patterns] } -YAQL_CONDITION = { +EXPRESSION_CONDITION = { "type": "object", "minProperties": 1, "patternProperties": { - "^\w+$": YAQL + "^\w+$": EXPRESSION } } @@ -54,8 +59,7 @@ ANY = { {"type": "integer"}, {"type": "number"}, {"type": "object"}, - {"type": "string"}, - YAQL + {"type": "string"} ] } @@ -67,8 +71,7 @@ ANY_NULLABLE = { {"type": "integer"}, {"type": "number"}, {"type": "object"}, - {"type": "string"}, - YAQL + {"type": "string"} ] } @@ -89,31 +92,31 @@ ONE_KEY_DICT = { } } -STRING_OR_YAQL_CONDITION = { +STRING_OR_EXPRESSION_CONDITION = { "oneOf": [ NONEMPTY_STRING, - YAQL_CONDITION + EXPRESSION_CONDITION ] } -YAQL_OR_POSITIVE_INTEGER = { +EXPRESSION_OR_POSITIVE_INTEGER = { "oneOf": [ - YAQL, + EXPRESSION, POSITIVE_INTEGER ] } -YAQL_OR_BOOLEAN = { +EXPRESSION_OR_BOOLEAN = { "oneOf": [ - YAQL, + EXPRESSION, {"type": "boolean"} ] } -UNIQUE_STRING_OR_YAQL_CONDITION_LIST = { +UNIQUE_STRING_OR_EXPRESSION_CONDITION_LIST = { "type": "array", - "items": STRING_OR_YAQL_CONDITION, + "items": STRING_OR_EXPRESSION_CONDITION, "uniqueItems": True, "minItems": 1 } diff --git a/mistral/workbook/v2/actions.py b/mistral/workbook/v2/actions.py index b97acb695..469c663b7 100644 --- a/mistral/workbook/v2/actions.py +++ b/mistral/workbook/v2/actions.py @@ -54,12 +54,12 @@ class ActionSpec(base.BaseSpec): # Validate YAQL expressions. inline_params = self._parse_cmd_and_input(self._data.get('base'))[1] - self.validate_yaql_expr(inline_params) + self.validate_expr(inline_params) - self.validate_yaql_expr(self._data.get('base-input', {})) + self.validate_expr(self._data.get('base-input', {})) if isinstance(self._data.get('output'), six.string_types): - self.validate_yaql_expr(self._data.get('output')) + self.validate_expr(self._data.get('output')) def get_name(self): return self._name diff --git a/mistral/workbook/v2/policies.py b/mistral/workbook/v2/policies.py index e817e43eb..e2d8b83da 100644 --- a/mistral/workbook/v2/policies.py +++ b/mistral/workbook/v2/policies.py @@ -19,11 +19,11 @@ from mistral.workbook.v2 import retry_policy RETRY_SCHEMA = retry_policy.RetrySpec.get_schema(includes=None) -WAIT_BEFORE_SCHEMA = types.YAQL_OR_POSITIVE_INTEGER -WAIT_AFTER_SCHEMA = types.YAQL_OR_POSITIVE_INTEGER -TIMEOUT_SCHEMA = types.YAQL_OR_POSITIVE_INTEGER -PAUSE_BEFORE_SCHEMA = types.YAQL_OR_BOOLEAN -CONCURRENCY_SCHEMA = types.YAQL_OR_POSITIVE_INTEGER +WAIT_BEFORE_SCHEMA = types.EXPRESSION_OR_POSITIVE_INTEGER +WAIT_AFTER_SCHEMA = types.EXPRESSION_OR_POSITIVE_INTEGER +TIMEOUT_SCHEMA = types.EXPRESSION_OR_POSITIVE_INTEGER +PAUSE_BEFORE_SCHEMA = types.EXPRESSION_OR_BOOLEAN +CONCURRENCY_SCHEMA = types.EXPRESSION_OR_POSITIVE_INTEGER class PoliciesSpec(base.BaseSpec): @@ -59,11 +59,11 @@ class PoliciesSpec(base.BaseSpec): super(PoliciesSpec, self).validate_schema() # Validate YAQL expressions. - self.validate_yaql_expr(self._data.get('wait-before', 0)) - self.validate_yaql_expr(self._data.get('wait-after', 0)) - self.validate_yaql_expr(self._data.get('timeout', 0)) - self.validate_yaql_expr(self._data.get('pause-before', False)) - self.validate_yaql_expr(self._data.get('concurrency', 0)) + self.validate_expr(self._data.get('wait-before', 0)) + self.validate_expr(self._data.get('wait-after', 0)) + self.validate_expr(self._data.get('timeout', 0)) + self.validate_expr(self._data.get('pause-before', False)) + self.validate_expr(self._data.get('concurrency', 0)) def get_retry(self): return self._retry diff --git a/mistral/workbook/v2/retry_policy.py b/mistral/workbook/v2/retry_policy.py index aa52f073b..b46176588 100644 --- a/mistral/workbook/v2/retry_policy.py +++ b/mistral/workbook/v2/retry_policy.py @@ -26,15 +26,15 @@ class RetrySpec(base.BaseSpec): "properties": { "count": { "oneOf": [ - types.YAQL, + types.EXPRESSION, types.POSITIVE_INTEGER ] }, - "break-on": types.YAQL, - "continue-on": types.YAQL, + "break-on": types.EXPRESSION, + "continue-on": types.EXPRESSION, "delay": { "oneOf": [ - types.YAQL, + types.EXPRESSION, types.POSITIVE_INTEGER ] }, @@ -74,10 +74,10 @@ class RetrySpec(base.BaseSpec): super(RetrySpec, self).validate_schema() # Validate YAQL expressions. - self.validate_yaql_expr(self._data.get('count')) - self.validate_yaql_expr(self._data.get('delay')) - self.validate_yaql_expr(self._data.get('break-on')) - self.validate_yaql_expr(self._data.get('continue-on')) + self.validate_expr(self._data.get('count')) + self.validate_expr(self._data.get('delay')) + self.validate_expr(self._data.get('break-on')) + self.validate_expr(self._data.get('continue-on')) def get_count(self): return self._count diff --git a/mistral/workbook/v2/task_defaults.py b/mistral/workbook/v2/task_defaults.py index 18446a66f..6cb48b0dc 100644 --- a/mistral/workbook/v2/task_defaults.py +++ b/mistral/workbook/v2/task_defaults.py @@ -32,7 +32,7 @@ class TaskDefaultsSpec(base.BaseSpec): _on_clause_type = { "oneOf": [ types.NONEMPTY_STRING, - types.UNIQUE_STRING_OR_YAQL_CONDITION_LIST + types.UNIQUE_STRING_OR_EXPRESSION_CONDITION_LIST ] } @@ -93,7 +93,7 @@ class TaskDefaultsSpec(base.BaseSpec): def _validate_transitions(self, on_clause): val = self._data.get(on_clause, []) - [self.validate_yaql_expr(t) + [self.validate_expr(t) for t in ([val] if isinstance(val, six.string_types) else val)] def get_policies(self): diff --git a/mistral/workbook/v2/tasks.py b/mistral/workbook/v2/tasks.py index a2a779060..b5668df0a 100644 --- a/mistral/workbook/v2/tasks.py +++ b/mistral/workbook/v2/tasks.py @@ -19,15 +19,15 @@ import re import six from mistral import exceptions as exc -from mistral import expressions as expr +from mistral import expressions from mistral import utils from mistral.workbook import types from mistral.workbook.v2 import base from mistral.workbook.v2 import policies - +_expr_ptrns = [expressions.patterns[name] for name in expressions.patterns] WITH_ITEMS_PTRN = re.compile( - "\s*([\w\d_\-]+)\s*in\s*(\[.+\]|%s)" % expr.INLINE_YAQL_REGEXP + "\s*([\w\d_\-]+)\s*in\s*(\[.+\]|%s)" % '|'.join(_expr_ptrns) ) RESERVED_TASK_NAMES = [ 'noop', @@ -62,8 +62,8 @@ class TaskSpec(base.BaseSpec): "pause-before": policies.PAUSE_BEFORE_SCHEMA, "concurrency": policies.CONCURRENCY_SCHEMA, "target": types.NONEMPTY_STRING, - "keep-result": types.YAQL_OR_BOOLEAN, - "safe-rerun": types.YAQL_OR_BOOLEAN + "keep-result": types.EXPRESSION_OR_BOOLEAN, + "safe-rerun": types.EXPRESSION_OR_BOOLEAN }, "additionalProperties": False, "anyOf": [ @@ -122,12 +122,12 @@ class TaskSpec(base.BaseSpec): # Validate YAQL expressions. if action or workflow: inline_params = self._parse_cmd_and_input(action or workflow)[1] - self.validate_yaql_expr(inline_params) + self.validate_expr(inline_params) - self.validate_yaql_expr(self._data.get('input', {})) - self.validate_yaql_expr(self._data.get('publish', {})) - self.validate_yaql_expr(self._data.get('keep-result', {})) - self.validate_yaql_expr(self._data.get('safe-rerun', {})) + self.validate_expr(self._data.get('input', {})) + self.validate_expr(self._data.get('publish', {})) + self.validate_expr(self._data.get('keep-result', {})) + self.validate_expr(self._data.get('safe-rerun', {})) def _transform_with_items(self): raw = self._data.get('with-items', []) @@ -149,11 +149,13 @@ class TaskSpec(base.BaseSpec): "%s" % self._data) raise exc.InvalidModelException(msg) - var_name, array = match.groups() + match_groups = match.groups() + var_name = match_groups[0] + array = match_groups[1] # Validate YAQL expression that may follow after "in" for the # with-items syntax "var in {[some, list] | <% $.array %> }". - self.validate_yaql_expr(array) + self.validate_expr(array) if array.startswith('['): try: @@ -223,7 +225,7 @@ class DirectWorkflowTaskSpec(TaskSpec): _on_clause_type = { "oneOf": [ types.NONEMPTY_STRING, - types.UNIQUE_STRING_OR_YAQL_CONDITION_LIST + types.UNIQUE_STRING_OR_EXPRESSION_CONDITION_LIST ] } @@ -271,7 +273,7 @@ class DirectWorkflowTaskSpec(TaskSpec): def _validate_transitions(self, on_clause): val = self._data.get(on_clause, []) - [self.validate_yaql_expr(t) + [self.validate_expr(t) for t in ([val] if isinstance(val, six.string_types) else val)] @staticmethod diff --git a/mistral/workbook/v2/workflows.py b/mistral/workbook/v2/workflows.py index 16312983f..40988eddf 100644 --- a/mistral/workbook/v2/workflows.py +++ b/mistral/workbook/v2/workflows.py @@ -77,9 +77,9 @@ class WorkflowSpec(base.BaseSpec): "Workflow doesn't have any tasks [data=%s]" % self._data ) - # Validate YAQL expressions. - self.validate_yaql_expr(self._data.get('output', {})) - self.validate_yaql_expr(self._data.get('vars', {})) + # Validate expressions. + self.validate_expr(self._data.get('output', {})) + self.validate_expr(self._data.get('vars', {})) def validate_semantics(self): super(WorkflowSpec, self).validate_semantics() diff --git a/requirements.txt b/requirements.txt index 1387393c2..910c141ca 100644 --- a/requirements.txt +++ b/requirements.txt @@ -7,6 +7,7 @@ Babel>=2.3.4 # BSD croniter>=0.3.4 # MIT License cachetools>=1.1.0 # MIT License eventlet!=0.18.3,>=0.18.2 # MIT +Jinja2>=2.8 # BSD License (3 clause) jsonschema!=2.5.0,<3.0.0,>=2.0.0 # MIT keystonemiddleware!=4.5.0,>=4.2.0 # Apache-2.0 mock>=2.0 # BSD diff --git a/setup.cfg b/setup.cfg index 9e9b662e8..21c7528f7 100644 --- a/setup.cfg +++ b/setup.cfg @@ -69,12 +69,16 @@ mistral.actions = std.javascript = mistral.actions.std_actions:JavaScriptAction std.sleep = mistral.actions.std_actions:SleepAction -mistral.yaql_functions = - json_pp = mistral.utils.yaql_utils:json_pp_ - task = mistral.utils.yaql_utils:task_ - execution = mistral.utils.yaql_utils:execution_ - env = mistral.utils.yaql_utils:env_ - uuid = mistral.utils.yaql_utils:uuid_ +mistral.expression.functions = + json_pp = mistral.utils.expression_utils:json_pp_ + task = mistral.utils.expression_utils:task_ + execution = mistral.utils.expression_utils:execution_ + env = mistral.utils.expression_utils:env_ + uuid = mistral.utils.expression_utils:uuid_ + +mistral.expression.evaluators = + yaql = mistral.expressions.yaql_expression:InlineYAQLEvaluator + jinja = mistral.expressions.jinja_expression:InlineJinjaEvaluator mistral.auth = keystone = mistral.auth.keystone:KeystoneAuthHandler