Merge "Documentation structure update"
This commit is contained in:
Jenkins
Gerrit Code Review
commit
0d13bf3aed
155
README.rst
155
README.rst
@ -1,101 +1,68 @@
|
||||
YAQL - Yet Another Query Language
|
||||
=================================
|
||||
YAQL: Yet Another Query Language
|
||||
================================
|
||||
|
||||
At the beginning of millennium the growing trend towards data formats standardization and application integrability made
|
||||
XML extremely popular. XML became lingua franca of the data. Applications tended to process lots of XML files ranging
|
||||
from small config files to very large datasets. As these data often had a complex structure with many levels of
|
||||
nestedness it is quickly became obvious that there is a need for specially crafted domain specific languages to query
|
||||
these data sets. This is how XPath and later XQL were born.
|
||||
YAQL (Yet Another Query Language) is an embeddable and extensible query
|
||||
language, that allows performing complex queries against arbitrary objects. It
|
||||
has a vast and comprehensive standard library of frequently used querying
|
||||
functions and can be extend even further with user-specified functions. YAQL is
|
||||
written in python and is distributed via PyPI.
|
||||
|
||||
With later popularization of REST services and Web 2.0 JSON started to take XML’s place. JSON’s main advantage (besides
|
||||
being simpler than XML) is that is closely reassembles data structures found in most programming languages (arrays,
|
||||
dictionaries, scalars) making it very convenient for data serialization. As JSON lacked all the brilliant XML-related
|
||||
technologies like XSLT, XML Schema, XPath etc. various attempts to develop similar languages for JSON were made. One of
|
||||
those efforts was JSONPath library developed in 2007 by Stefan Gössner. Initial implementation was for PHP and
|
||||
JavaScript languages, but later on ports to other languages including Python were written.
|
||||
Quickstart
|
||||
----------
|
||||
Install the latest version of yaql:
|
||||
|
||||
JSONPath allows navigation and querying, well, JSONs.
|
||||
Suppose we have JSON as in following:
|
||||
.. code-block:: console
|
||||
|
||||
{
|
||||
"customers": [
|
||||
{
|
||||
"customer_id": 1,
|
||||
"name": "John",
|
||||
"orders": [{
|
||||
"order_id": 1,
|
||||
"item": "Guitar",
|
||||
"quantity": 1
|
||||
}]
|
||||
},{
|
||||
"customer_id": 2,
|
||||
"name": "Paul",
|
||||
"orders": [ {
|
||||
"order_id": 2,
|
||||
"item": "Banjo",
|
||||
"quantity": 2
|
||||
},{
|
||||
"order_id": 3,
|
||||
"item": "Piano",
|
||||
"quantity": 1
|
||||
}]
|
||||
}
|
||||
]
|
||||
}
|
||||
pip install yaql>=1.0.0
|
||||
..
|
||||
|
||||
Run yaql REPL:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
yaql
|
||||
..
|
||||
|
||||
Load a json file:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
yaql> @load my_file.json
|
||||
..
|
||||
|
||||
Check it loaded to current context, i.e. `$`:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
yaql> $
|
||||
|
||||
..
|
||||
|
||||
Run some queries:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
yaql> $.customers
|
||||
...
|
||||
yaql> $.customers.orders
|
||||
...
|
||||
yaql> $.customers.where($.age > 18)
|
||||
...
|
||||
yaql> $.customers.groupBy($.sex)
|
||||
...
|
||||
yaql> $.customers.where($.orders.len() >= 1 or name = "John")
|
||||
..
|
||||
|
||||
Project Resources
|
||||
-----------------
|
||||
|
||||
* `Official Documentation <http://yaql.readthedocs.org>`_
|
||||
|
||||
* Project status, bugs, and blueprints are tracked on
|
||||
`Launchpad <https://launchpad.net/yaql>`_
|
||||
|
||||
|
||||
then
|
||||
License
|
||||
-------
|
||||
|
||||
`jsonpath(data, "$.customers[0].name") -> [‘John’]`
|
||||
`jsonpath(data, "$.customers[*].orders[*].order_id") -> [1, 2, 3]`
|
||||
|
||||
But what if we need, for example to find order having ID = 2? Here is how it done in JSONPath:
|
||||
|
||||
`jsonpath(data, "$.customers[*].orders[?(@.order_id == 2)") -> [{'order_id': 2, 'item': 'Banjo', 'quantity': 2}]`
|
||||
|
||||
The construct `[?(expression)]` allows to filter items using any Python expression in our case. `@` character is
|
||||
replaced with current value and then the whole expression is evaluated. Evaluation of arbitrary Python expression
|
||||
requires using `eval()` function unless one wants to develop his own complete parser and interpreter of Python
|
||||
programming language. Needless to say that `eval()` is a great security breach. If JSONPath expressions are used to
|
||||
simplify program logic it would not be a big deal, but what if JSONPath is written by program users?
|
||||
|
||||
JSONPath expression is just a plain string. There is no such concept as parameter. That is if one want to find order
|
||||
having ID = some variable value he has to dynamically construct expression string using string formatting or
|
||||
concatenation. And again that is might be okay for internal usage but would became difficult for external usage and also
|
||||
open the doors for injection attacks (remember SQL injection?)
|
||||
|
||||
Another limitation of JSONPath is JSON itself. Technically speaking JSONPath operates not on the JSON itself (i.e. text
|
||||
representation) but on a JSON-like object model that is mixture of arrays, dictionaries and scalar values. But what is
|
||||
one want to query object model consisting of custom objects? What if some parts of this model are dynamically computed?
|
||||
Or the model is a graph rather than a tree?
|
||||
|
||||
It seems like JSONPath is good enough to use in Python code when you can `eval()` things and have many helper function
|
||||
to work with data besides JSONPath capabilities but is not enough for external use when you need to have sufficient
|
||||
power to query model without manual coding and have it still secure.
|
||||
This is why we designed YAQL. YAQL follows the JSONPath ideas and has very similar syntax but offers much more for data
|
||||
querying.
|
||||
|
||||
Expressions are quite similar to JSONPath. Here is how examples above can be translated to YAQL:
|
||||
|
||||
`$.customers[0].name -> $.customers[0].name (no change)`
|
||||
`$.customers[*].orders[*].order_id -> $.customers.orders.order_id`
|
||||
|
||||
the main addition to JSONPath is functions and operators. Consider the following YAQL expressions:
|
||||
|
||||
`$.customers.orders[$.quantity > 0].quantity.sum() -> 4`
|
||||
`$.customers.orders.select($.quantity * $.quantity).sum() -> 6`
|
||||
`$.customers.orders.order_id.orderDesc($) -> [3, 2, 1]`
|
||||
`$.customers.orders.order_id.orderDesc($).take(2) -> [3, 2]`
|
||||
`$.customers.orders.order_id.orderDesc($).first() -> 3`
|
||||
|
||||
Does it mean that YAQL has large built-in function and operator library?. Yes, YAQL library has a out of the box large
|
||||
set of commonly used functions. But they are not built-in. All the functions and operators (which are also function:
|
||||
`a + b = operator_+(a, b)` etc) are user-supplied. User is free to add other functions that could be used in expressions
|
||||
and to remove standard ones.
|
||||
|
||||
JSONPath library needs 2 arguments - input JSON data and an a expression. YAQL library requires third
|
||||
parameter - context.
|
||||
|
||||
Context is a repository of functions and variables that can be used in expressions. So all the functions above are just
|
||||
ordinary Python functions that are registered in Context object. But because they all need to be registered in Context
|
||||
user can always customize them, add his own model-specific ones and have full control over the expression evaluation.
|
||||
Apache License Version 2.0 http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
@ -4,16 +4,33 @@
|
||||
contain the root `toctree` directive.
|
||||
|
||||
Welcome to yaql's documentation!
|
||||
========================================================
|
||||
================================
|
||||
|
||||
Contents:
|
||||
Introduction
|
||||
~~~~~~~~~~~~
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
readme
|
||||
installation
|
||||
what_is_yaql
|
||||
|
||||
Usage
|
||||
~~~~~
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
usage
|
||||
language_description
|
||||
standard_library
|
||||
|
||||
Contributing
|
||||
~~~~~~~~~~~~
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
contributing
|
||||
|
||||
Indices and tables
|
||||
|
||||
@ -1,12 +0,0 @@
|
||||
============
|
||||
Installation
|
||||
============
|
||||
|
||||
At the command line::
|
||||
|
||||
$ pip install yaql
|
||||
|
||||
Or, if you have virtualenvwrapper installed::
|
||||
|
||||
$ mkvirtualenv yaql
|
||||
$ pip install yaql
|
||||
19
doc/source/language_description.rst
Normal file
19
doc/source/language_description.rst
Normal file
@ -0,0 +1,19 @@
|
||||
Language description
|
||||
====================
|
||||
|
||||
This section is not ready yet
|
||||
|
||||
Syntax
|
||||
~~~~~~
|
||||
|
||||
Types
|
||||
~~~~~
|
||||
|
||||
Operators
|
||||
~~~~~~~~~
|
||||
|
||||
Context
|
||||
~~~~~~~
|
||||
|
||||
$ variable
|
||||
~~~~~~~~~~
|
||||
4
doc/source/standard_library.rst
Normal file
4
doc/source/standard_library.rst
Normal file
@ -0,0 +1,4 @@
|
||||
Standard Library
|
||||
================
|
||||
|
||||
This section is not ready yet.
|
||||
@ -1,7 +1,13 @@
|
||||
========
|
||||
Usage
|
||||
========
|
||||
=====
|
||||
|
||||
To use yaql in a project::
|
||||
This section is not ready yet.
|
||||
|
||||
import yaql
|
||||
REPL
|
||||
~~~~
|
||||
|
||||
Embedding YAQL
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
Extending YAQL
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
22
doc/source/what_is_yaql.rst
Normal file
22
doc/source/what_is_yaql.rst
Normal file
@ -0,0 +1,22 @@
|
||||
What is YAQL
|
||||
============
|
||||
|
||||
YAQL is a general purpose query language, that is designed to operate on
|
||||
objects of arbitrary complexity.
|
||||
YAQL has a large standard library of functions for filtering, grouping and
|
||||
aggregation of data. At the same time YAQL allows you to extend it by
|
||||
defining your own functions.
|
||||
|
||||
Why YAQL?
|
||||
=========
|
||||
|
||||
So why bother and create another solution for a task, that has been addressed
|
||||
by many before us? Obviously because we were not satisfied with flexibility
|
||||
and/or quality of any existing solution. Most notably we needed a tool for json
|
||||
data, that would support some complex data transformations.
|
||||
YAQL is a pure-python library and therefore is easily embeddable in any python
|
||||
application.
|
||||
YAQL is designed to be human-readable and has a SQL-like feel and look. It is
|
||||
inspired in part by LINQ for .NET.
|
||||
Since YAQL is extensible and embeddable it makes a perfect choice for becoming
|
||||
the basis for your DSLs.
|
||||
Loading…
x
Reference in New Issue
Block a user