This patch changes the Redis drivers so they use the oslo.utils
version of timeutils. This is consistent with the rest of Zaqar.
Change-Id: I56379b90414cbd8bcc6bca6998988dba800693b9
Partially-Implements: blueprint redis-storage-driver
Side counters are error-prone, and there are cases where they can
get out of sync if we lose a connection to the DB at the wrong
moment. Also, they are difficult to update when performing GC
operations.
This patch mostly removes the use of counters to track stats and
does the calculations on the fly. These calculations should still be
quite fast, since we are still using a per-claim counter (which
is easier to maintain than the per-queue counters).
Change-Id: I5d63a3791afd8d749890c5996208568c974f71d1
Partially-Implements: blueprint redis-storage-driver
It was decided to make Client-ID a required header. This patch adds a
new `before hook` that enforces this new rule.
Partially-Implements blueprint: api-v1.1-header-changes
Change-Id: I7b20d56812a0dff4f8ed3e6da75b62ef7c0f307e
We decided to make X-Project-ID a required header for every request. If
not present, a 400 error should be raised.
Partially-Implements: blueprint api-v1.1-header-changes
Change-Id: I6c6c21585e9a738ff7c0f74ce6a284229ebec2aa
This patch uses Redis where possible to automatically expire
claim and message data. Where it is not possible, manual
GC functions have been added in order to iterate through the
remaining data and clean it up.
Note that this patch does not include an implementation for
a command script that will call the Redis drivers' GC
method. That will be done in a followup patch.
Also, the GC functions in this patch do not decrement the
stats counters, since those are actually removed in the next
patch.
This change has been tested locally on my box using the
ZAQAR_TEST_REDIS=1 flag.
Change-Id: I38bfc52cf2de0ed078478aca80e6ad56b5bd55aa
Partially-Implements: blueprint redis-storage-driver
This patch implements the standard controllers for the redis
storage driver. It has been tested against a localhost Redis
server with ZAQAR_TEST_REDIS=1.
Change-Id: Ib7c100afd11a0410c3f241c1925d5aaf172ce6a8
Partially-Implements: blueprint redis-storage-driver
This patch adds the knowledge of flavors to the catalog. When a new
queue is registered, if a flavor is passed, it'll get the pool from the
flavor record. If flavor is not passed, it'll preserve the previous
behavior.
Partially Implements blueprint: marconi-queue-flavors
Change-Id: Idcf5a0215699269f29df793169ff21fc6435f3b2
This patch adds the control API for flavors. The patch allows users to
create, delete, update and list flavors. This API is considered an
admin-only API, hence it's registered as part of the private endpoints.
DocImpact
Partially-Implements blueprint marconi-queue-flavors
Change-Id: Id29d29940f2ecabab0531edefe018c0dd2f39811
All top-level JSON arrays in response bodies are now encapsulated within
an object. Endpoints changed:
messages_get_many: GET /v1.1/queues/{queue}/messages?ids=
claim_create: POST /v1.1/queues/{queue}/claims
Partially-implements: blueprint api-v1.1-response-document-changes
Change-Id: I81e64decc928be665b8b4f4d0e238fd701794bc9
This patch is a follow-up on the queue's flavor work. It adds support
for such feature to the MongoDB storage driver.
Partially-Implements blueprint: marconi-queue-flavors
Change-Id: Ib3231ab361896e1c08831d17899dd4111710ab9b
The patch adds a minimum requirement for mongodb. This will provide a
useful error message to users when the installed mongodb version doesn't
match the minimum required one.
Change-Id: Iff8bb1645e7276529230da58c408552bebd8647a
This patch renames every package, file, match of Marconi in the codebase
to Zaqar *except* for the .gitreview file, which will have to be updated
*after* I8e587af588d9be0b5ebbab4b0f729b106a2ae537 lands.
Implements blueprint: project-rename
Change-Id: I63cf2c680cead4641f3e430af379452058bce5b3
During Icehouse, we decided to pull the metadata out of the queue
endpoint by adding a new endpoint `/metadata`. This worked well because
we didn't have a clear view of what the scope of metadata is. Now that
we want to use it and make it a key piece of the queues, we've found
that the workflow is not as clear as we'd like. The reason being that
some key items of the metadata - flavor, for example - need to be
specified during the queue creation and not after.
Merging these endpoints together is enough to fix this issue.
DocImpact
Implements blueprint: api-v1.1-merge-queue-and-metadata
Change-Id: Icca1a2b2066a518f8f50cad20e55b39a8bff0f2d
Some mongodb tests were failing because the test suite was not
cleaning up the database in between runs. In the course of
fixing this bug, the tests in question were also DRY'd up between
the sqla and mongo drivers.
NOTE: This patch is a prerequisite for enabling mongo in the gate.
Change-Id: Iaa2a1cbd6a27c53a884c9e0260d48d77a7f198b9
This patch removes an unnecessary stream seek from one test,
and renames a couple other tests to reflect the refactoring
of the filter_stream function into two separate functions.
Change-Id: Ie6731766028dee2410193ae79ec4c6f985df1a92
README instructions on running functional tests without tox
points to nonexistent conf file, and also steps are not correct.
Change-Id: I51ce0a4a7f4bc3396a744ec1bedcf862dcc04688
Closes-Bug: #1347223
This patch implements one of the breaking changes specified in
the API v1.1 spec, which requires clients to encapsulate the
list of messages to enqueue in a JSON object, when posting
those messages to the API.
Partially-Implements: blueprint api-v1.1-request-document-changes
Change-Id: I72ae87e1c479b6be6fbc8ce1fbaee89b2729e41a
When posting messages, the "ttl" field may be omitted, in which
case the server defaults to the default TTL value configured by
the operator.
Likewise, when creating a claim, the "ttl" and "grace" fields
may be omitted to use the defaults configured for each field.
Change-Id: I38c6307377293b26370de28ffa13db22228c208c
Partially-Implements: blueprint api-v1.1-request-document-changes
This patch removes the queue exists check from the transport layer.
Implements: blueprint api-v1.1-remove-check-queue-existence
Change-Id: I5fbd3089d552c22ab59a49a7fadc743cbb884770
This patch implements the pop functionality for v1.1 of the Marconi
API. The POP functionality is implemented as an optional parameter
in the delete messages using id params endpoint. This includes the
following validations,
1. pop & id params cannot be used together in the request.
2. 0 < pop value <= max_messages_per_claim_or_pop
docImpact
Change-Id: I63b22e57ed5ab398b81cde2d0284767e9704ec11
Implements: blueprint api-v1.1-pop-operation
Given that pylint is raising a lot of false positives
and that we have alternative static code checkers,
we decided to remove pylint from the tox envlist.
This patch also fix some pylint true positives
and cleans up the disabled pylint tests in the codebase.
Change-Id: I6fb4b9b4c80af452796032736e39ef064b9bcd03
Closes-Bug: #1297994
Enable storage unit tests for pooled context. This patch adds check for
pooling in test config in test initialization and creates appropriate
wrappers.
Also changes to sqlalchemy pooling controllers were made: in pooled
context project cannot be None due to constraint in database.
Change-Id: I59a663cef5b761e9470b8870d07cea03f5dac98b
Marker's test used a local queue_name instead of the one defined in the
base class. This caused some tests to fail because the ran on a no clean
database.
The patch makes the test use the queue_name and the project defined in
the class.
Change-Id: I5a1682e835a07a452eb1114da8c87369fe2f48b4
Added tests for new V1.1 functions like pop and shard
Updated existing tests for V1.1 return calls
Added new json schemas
Edited functional test base to support multiple schemas
Change-Id: I35dfb92af22540609ecb50c1eb5c8fa65e6cadc8
Partially-Implements blueprint: api-v1.1-functional-tests
This commit contains last small fixes. In Python 3 None cannot be
compared with int. Also string can be added only to string, thus
NamedUnicodeStr should be subclass of string.
Change-Id: I3afaa24cd88861e56b906689870ca7f1b8e853bb
Implements: blueprint py3k-support
The newer hacking version is more strict about some things, so I
had to update a bunch of files with minor issues that previously
fell through the cracks.
Change-Id: I169bc6f8533bab28c7d6cf321ea83a8d7945f136
Shard turned out to be a quite overloaded/confusing term to use here,
although correct. We decided to rename our shard feature to pool.
P.S: I swear I didn't use sed O.O
Change-Id: Ic54f29a4da7d7690c9c9210b74876b96f0ae0eac
