Unable to use templates option in latest version of td-agent along with opensearch plugin.

[vikranth06]

Steps to replicate

We are using the latest version of td-agent along with Opensearch from AWS but as soon as we put templates section in the config file following error message is thrown "2023-01-16 13:47:39 +0000 [error]: #0 fluent/log.rb:372:error: unexpected error error_class=ArgumentError error="expected both :access_key_id and :secret_access_key options""

Expected Behavior or What you need to ask

As per the documentation templates is supported. Can the below config file be reviewed and let us know if there is a mistake?
...

Config file

<match **.*******.UPLOAD>
  @type copy
  <store>
    @type opensearch
    index_name xx-xxxx-xxxxx-data
    type_name "_doc"
    logstash_format true
    logstash_prefix xx-xxxx-xxxxx-data
    logstash_dateformat %Y.%m
    template_overwrite true
    templates {"xxxxxxxxxxxxxx_template": "/opt/aws/custom/xxxxx_index_template.json"}
    include_tag_key false
    buffer_type file
    buffer_chunk_limit 2m
    buffer_queue_limit 1000
    buffer_path /var/log/td-agent/uploadreportsupesbuffer
    disable_retry_limit true
    flush_interval 60s
    retry_wait 10s
    id_key MessageId
    reload_connections false
    remove_keys MessageId
    <endpoint>
      url xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      region us-east-1
      access_key_id xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      secret_access_key xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
	  </endpoint>
  </store>
</match>

Using Fluentd and OpenSearch plugin versions

• OS version - Ubuntu 22.04
• Bare Metal or within Docker or Kubernetes or others?
• Fluentd v1.0 or later
  • td-agent 4.4.2 fluentd 1.15.3
• OpenSearch plugin version
  • paste boot log of fluentd or td-agent
  • paste result of fluent-gem list, td-agent-gem list or your Gemfile.lock
• OpenSearch version (optional)
• OpenSearch template(s) (optional)

[Watson1978]

Thanks for the report, and sorry for the late response.

This isn't a mistake in your config — the config itself is essentially fine.
What you're hitting is a plugin-side limitation when index-template management is combined with AWS SigV4 auth (<endpoint>).

What happens:

• Template handling (templates, template_name/template_file, template_overwrite) runs synchronously while the plugin is starting up, before any buffered data is flushed.
• That startup path issues the _template requests, and in the SigV4 + <endpoint> case the credentials from <endpoint> aren't carried over to it, so aws-sigv4 raises ArgumentError: expected both :access_key_id and :secret_access_key options.
• With no template section, that startup path never runs, which is why plain ingestion (the _bulk path) works.

So this is a bug in the "SigV4 auth × template management" combination, not something wrong on your side.

Workaround to unblock you now: register the index template directly in OpenSearch once, and drop the template options from Fluentd.

1. Register the template out-of-band (OpenSearch Dashboards → Dev Tools, or curl/awscurl with signed requests):

   PUT _index_template/xxxxxxxxxxxxxx_template
   { ...contents of /opt/aws/custom/xxxxx_index_template.json... }
   

2. Remove these two lines from the <store> block:

   template_overwrite true
   templates {"xxxxxxxxxxxxxx_template": "/opt/aws/custom/xxxxx_index_template.json"}
   

Ingestion stays signed correctly and keeps working, and OpenSearch applies the pre-registered template to newly created indices.

If you want the plugin to manage templates later, also confirm that the IAM policy for those credentials allows es:ESHttp* on the domain's _template* / _index_template* resources, not only the index and _bulk paths.

To help us reproduce and fix the underlying issue, could you share:

• the exact plugin version (td-agent-gem list | grep opensearch, or the fluent-plugin-opensearch line from the boot log)
• the full stack trace at startup (run with @log_level debug)?