The Kolla Ansible Fluentd config is not idempotent. In practise this
can be an issue if some logs are rejected by OpenSearch /
ElasticSearch bulk API. In this case, by default, the unprocessed
logs will be reprocessed from the start of the Fluentd pipeline,
leading to error messages.
The solution proposed here is to explicitly set the retry_tag as
documented in [1,2], and add a dedicated output for retried logs.
An alternative fix could be to make the pipeline idempotent. This
would require a compromise of either duplicating content, or having
non-standard fields for the log payload/message.
[1] https://github.com/fluent/fluent-plugin-opensearch?tab=readme-ov-file#retry_tag
[2] https://github.com/uken/fluent-plugin-elasticsearch?tab=readme-ov-file#retry_tag
Closes-Bug: #2064104
Change-Id: I310fc1b8e002ce9f0ba60f8bb67b7f372a589314
a2273026d7