[Q&A] Fluentd retry_times overflows to 0 causing retry forever

[yukuraga]

What is a problem?

Hello, we have a project setup where fluentd buffer configuration is as follows:

<buffer time>
      @type file
      path /fluentd/buffer/myproject
      flush_thread_count 8
      total_limit_size 16GB
      flush_mode interval
      flush_interval 15s
      retry_max_times 8
      timekey 300
      timekey_wait 30
      timekey_use_utc true
      flush_at_shutdown true
</buffer>

In this setup, we came across a situation where a buffer is failed to be flushed for some reason (that's another issue) but fluentd retries to flush this buffer forever causing accumulating logs in the database.

When I check fluentd logs, I see that retry_times overflows to 0 before reaching retry_max_times which is 8 in our setup.

Can you help me on this?

PS: For the sake of information security, I did not upload entire fluentd logs, if needed I'll try my best to provide.

Describe the configuration of Fluentd

No response

Describe the logs of Fluentd

No response

Environment

- Fluentd version: fluentd 1.18.0

[Watson1978]

I suspect that at least two events have caused the failed to flush the buffer error, not overflow.
The cause of your issue is difficult to guess from the given information....