Recently the elasticsearch schema was updated to include a
build_short_uuid field which has indexed the first 7 chars of the
build_uuid. This field is useful because it allows e-r to filter on that
field instead of searching on build_uuid.
Update e-r to filter on build_short_uuid which should make queries much
more performant. As part of this change replace variables named
short_build_uuid with build_short_uuid for consistency with the
elasticsearch schema.
Change-Id: Iae5323f3f5d2fd01f2c69f78b9403baf5ebafe85
In preperation for providing a web page that will just show hits on the
gate queue, add a '-q queue' option to elastic-recheck-graph.
Change-Id: I9217a2ceedf86ffe04851084df78238384fccd51
All parameters in query_builder.result_ready are needed, so remove
default value of None. Defaulting these values to None can hide bugs
where not enough parameters are passed in.
Change-Id: Ib6aee3ec1f39ff8fa9a221b118e157e5326d1c6b
In addition to searching by change and patch search by the short build_uuid.
This prevents us accidentally classifying multiple builds when we classify
a failure on gerrit. This can happen in the gate queue if there is a
gate reset, or if there are multiple 'recheck bug x' on a single patch
revision in the check queue.
Change-Id: I6356a971ca250ddf5f01a9734f13d0b080a62c89
Our elasticSearch cluster previously used '@message', but we have since
moved over to using just 'message'. The rest of the uses of '@message'
were removed in I6fb0aa87a291660df879282e9a7851bbb27e9ac2
Change-Id: I2b5d0f176deddb1b1ab9e831395c3216e927d8bf
this changes the interface to move the readiness check out of
the classifier and into the stream object. This massively
simplifies the logic connecting these pieces, as classifier is
now just a thin wrapper to elastic search.
This also adds unit testing for the stream processing through the
creation of a fake_gerrit mock class. That lets us run gerrit
event interactions in a sane way.
It also drops all the unit testing for the classifier which is now
largely useless, because all it tests is we can execute a for loop.
Change-Id: I1971c121276412e31f01eb5680b9c41fc7e442d3
* elastic_recheck/query_builder.py: Due to the way multiline events
work in logstash, the "Finished: FAILURE" line is not pushed into
elasticsearch until the next console.html file is processed or some
timeout is reached (I haven't actually confirmed that the timeout works
though). This happens because this is the last line in the file that we
do not drop. To better detect the end of indexing look for the "[SCP]
Copying console log" message which should also happen at the end of each
console log file.
A better long term solution to this problem may be to have the logstash
gearman workers insert and EOF event that can ensure each log file is
completely flushed without waiting for more data.
Change-Id: I7c51e019c774f9ba15e731aca01aaffc27de1e60
as part of trying to simplify the core elasticRecheck, refactor
the query creation into a separate set of query_builder routines.
This takes away some of the duplication between the queries, and
attempts to add documentation to the uses for each of them.
add elasticRecheck fake pyelasticsearch testing
build basic fixtures for unit testing that let us fake out the
interaction to pyelasticsearch. This uses the json samples added
for previous testing as the return results should an inbound
query match one of the queries we know about.
If the query is unknown to us, return an empty result set. Unit
testing for both cases included going all the way from the top
level Classifier class.
Change-Id: I0d23b649274b31e8f281aaac588c4c6113a11a47
