failed to flush the buffer. Mongo::Error::NoServerAvailable "No server is available matching preference: #<Mongo::ServerSelector::Primary:0x47353936741240"

[rockettomatooo]

Hey guys,

I'm trying to setup a pretty simple fluentd config. I simply want some logs to be sent to my replica set (which only consists of 1 primary which is a mongo:4.0.18 docker img) on my local kubernetes. The mongodb server is available on "mongodb.default:27017".

This is my fluentd config

<source>
  @type http
  port 8888
  bind 0.0.0.0
  <parse>
    @type json
    time_type unixtime
  </parse>
</source>

# <match **>
#   @type stdout
# </match>

<match mongo.*>
  @type mongo_replset

  nodes mongodb.default:27017,
  replica_set rs0
  database fluentd
  collection ${tag}
  

  <inject>
    time_key time
  </inject>

  <buffer>
    flush_interval 10s
  </buffer>
</match>

<system>
  log_level debug
</system>

I'm using this Dockerfile for my fluentd container:

FROM fluent/fluentd:latest

COPY setup.sh /tmp/
RUN /tmp/setup.sh

COPY fluentd.conf /fluentd/etc/

with setup.sh beeing:

apk add --no-cache --update --virtual .build-deps \
        sudo build-base ruby-dev 


# install plugins
sudo gem install fluent-plugin-mongo


# clean up 
sudo gem sources --clear-all
apk del .build-deps
rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

Apparently this doesn't work though. I'm always getting this error:

2020-10-21 13:20:17 +0000 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2020-10-21 13:20:18 +0000 chunk="5b22e330161c80e2e79154b8aea6ff51" error_class=Mongo::Error::NoServerAvailable error="No server is available matching preference: #<Mongo::ServerSelector::Primary:0x47186394554900 tag_sets=[] max_staleness=nil> using server_selection_timeout=30 and local_threshold=0.015"
  2020-10-21 13:20:17 +0000 [warn]: #0 /usr/lib/ruby/gems/2.5.0/gems/mongo-2.6.4/lib/mongo/server_selector/selectable.rb:115:in `select_server'
  2020-10-21 13:20:17 +0000 [warn]: #0 /usr/lib/ruby/gems/2.5.0/gems/mongo-2.6.4/lib/mongo/cluster.rb:263:in `next_primary'
  2020-10-21 13:20:17 +0000 [warn]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-mongo-1.4.1/lib/fluent/plugin/out_mongo.rb:255:in `list_collections_enabled?'
  2020-10-21 13:20:17 +0000 [warn]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-mongo-1.4.1/lib/fluent/plugin/out_mongo.rb:259:in `collection_exists?'
  2020-10-21 13:20:17 +0000 [warn]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-mongo-1.4.1/lib/fluent/plugin/out_mongo.rb:273:in `get_collection'
  2020-10-21 13:20:17 +0000 [warn]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-mongo-1.4.1/lib/fluent/plugin/out_mongo.rb:305:in `operate'
  2020-10-21 13:20:17 +0000 [warn]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-mongo-1.4.1/lib/fluent/plugin/out_mongo_replset.rb:40:in `block in operate'
  2020-10-21 13:20:17 +0000 [warn]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-mongo-1.4.1/lib/fluent/plugin/out_mongo_replset.rb:47:in `rescue_connection_failure'
  2020-10-21 13:20:17 +0000 [warn]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-mongo-1.4.1/lib/fluent/plugin/out_mongo_replset.rb:39:in `operate'
  2020-10-21 13:20:17 +0000 [warn]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-mongo-1.4.1/lib/fluent/plugin/out_mongo.rb:186:in `write'
  2020-10-21 13:20:17 +0000 [warn]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-mongo-1.4.1/lib/fluent/plugin/out_mongo_replset.rb:33:in `write'
  2020-10-21 13:20:17 +0000 [warn]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/plugin/output.rb:1123:in `try_flush'
  2020-10-21 13:20:17 +0000 [warn]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/plugin/output.rb:1423:in `flush_thread_run'
  2020-10-21 13:20:17 +0000 [warn]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/plugin/output.rb:452:in `block (2 levels) in start'
  2020-10-21 13:20:17 +0000 [warn]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'

Things I already tried:

1. I tried to use @type mongo instead. used connection_string instead of the standalone params but everything leads to the same error.
2. I downgraded my mongodb from v4.2 initially to 4.0.18 now. Exactly the same error.

This is my first time working with fluentd and the mongo-plugin. Also I don't have any experience with ruby. Any ideas / suggestions are appreciated!

[repeatedly]

I checked basic conf works on my environment.

<match mongo.*>
  @type mongo

  connection_string mongodb://127.0.0.1:27017/fluent
  collection ${tag}

  <buffer>
    flush_interval 5s
  </buffer>
</match>

So if basic configuration doesn't work on your environment, the problem is something wrong, e.g. network setting, port setting, dns setting or etc.

[rockettomatooo]

@repeatedly thanks for your answer. Did you check with a mongo replica set locally?
I basically start my mongodb container with mongod --replSet rs0 --bind_ip 0.0.0.0 and when the container starts the first time, you have to run rs.initiate() on the mongo shell to initialize the replica set. I only have a master though.

[repeatedly]

Yes. It worked on my local environment with only primary setting.

rs0:PRIMARY> show dbs;
admin      0.000GB
config     0.000GB
fluent     0.000GB
fluentdrs  0.000GB
local      0.000GB
rs0:PRIMARY> use fluentdrs
switched to db fluentdrs
rs0:PRIMARY> show tables;
mongo.test
rs0:PRIMARY> db.mongo.test.find();
{ "_id" : ObjectId("5fa02365b214b345422626df"), "msg" : "hi for rs", "time" : ISODate("2020-11-02T15:18:55.665Z") }

fluentd conf:

<source>
  @type forward
</source>

<match mongo.*>
  @type mongo_replset

  nodes 127.0.0.1:27017
  replica_set rs0
  database fluentdrs
  collection ${tag}

  <inject>
    time_key time
  </inject>
  <buffer>
    flush_interval 5s
  </buffer>
</match>

[ultryuan]

I have a similar issue.
mongodb is stood up as a container within a swarm, thus my fluentd container in the swarm should recognize it by its container name "mongodb". Which I verified using ping and nslookup (but I had to use "mongodb." with the nslookup.)

But when I set the host to mongodb (i've also tried mongodb.) I keep getting the error message described above. Actual IP address works though.
fluentd conf:

<mongo.log>
      @type mongo
      host mongodb
      port 27017
      database logs
      collection ${tag} 
      <buffer>
        flush_interval 5s
      </buffer>
      <inject>
        time_key time
      </inject>
  </match>

[arun11299]

Facing same issue. @rockettomatooo Did you figure out the issue you were having ?

[arun11299]

The solution is to use the same host name that comes in "rs.conf()" or "rs.status()" in fluentd conf. Since I was operating in kubernetes, "rs.initiate()" was taking the pod name as host instead of service name. Changed the hostname following https://www.mongodb.com/docs/manual/tutorial/change-hostnames-in-a-replica-set/#:~:text=To%20change%20the%20hostnames%2C%20edit,Switch%20to%20the%20local%20database.&text=Create%20a%20JavaScript%20variable%20for,to%20match%20your%20replica%20set.

cfg = rs.conf()
cfg.members[0].host = "mongodb0.example.net:27017"
rs.reconfig(cfg)