opendev/elastic-recheck
Code Issues Proposed changes

Better end of console log indexing detection.

Browse Source 
* elastic_recheck/query_builder.py: Due to the way multiline events
work in logstash, the "Finished: FAILURE" line is not pushed into
elasticsearch until the next console.html file is processed or some
timeout is reached (I haven't actually confirmed that the timeout works
though). This happens because this is the last line in the file that we
do not drop. To better detect the end of indexing look for the "[SCP]
Copying console log" message which should also happen at the end of each
console log file.

A better long term solution to this problem may be to have the logstash
gearman workers insert and EOF event that can ensure each log file is
completely flushed without waiting for more data.

Change-Id: I7c51e019c774f9ba15e731aca01aaffc27de1e60
This commit is contained in:
Clark Boylan 2013-10-28 15:25:26 -07:00
parent f05111a794
commit 604b6468ee
1 changed files with 3 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
elastic_recheck/query_builder.py
View File

@ -66,7 +66,9 @@ def result_ready(review=None, patch=None):
lets us know that we've got results waiting that we need to process.
"""
return generic('filename:"console.html" AND '
'(@message:"Finished: FAILURE" OR message:"Finished: FAILURE") '
'(@message:"[SCP] Copying console log" '
'OR message:"[SCP] Copying console log") '
'AND build_status:"FAILURE" '
'AND build_change:"%s" '
'AND build_patchset:"%s"' %
(review, patch))