Fluentd Not Sending Data to MongoDB

[tech-talks-india]

I am trying to send the parsed data from Fluentd to MongoDB. My system configuration is as below:

And MongoDB plugin: fluent-plugin-mongo | 1.1.0

My data is being parsed but I am not able to send it to Mongo. Here is my config file:

<match *>
  @type mongo
  host 127.0.0.1
  port 27017
  user tti-prod
  password mypassword
  database fluent-testrun
  collection requests
  capped
  capped_size 100m
</match>

<source>
  @type tail
  path /home/my-app/logs/%Y/%b/app.log
  tag request.main
  format /^\[(?<time>[^\]]*)\] (?<ip>[^ ]*) (?<method>\w*) (?<url>[^ ]*) (?<format>[^ ]*) (?<size>\d*) (?<agent>[^ ]*) (?<status_code>\d*) (?<duration>\d*)$/
  time_format %Y-%m-%d %H:%M:%S 
  pos_file /tmp/fluentd--1516870436.pos
</source>

Can anyone explain this. There is no error log as such.

[tech-talks-india]

I tried using fluentd insead of td-agent. It did not work either.

[aferreira]

Notice that the * pattern just matches "a single tag part" – see https://docs.fluentd.org/v1.0/articles/config-file#match-pattern:-how-you-control-the-event-flow-inside-fluentd

That means the request.main tag won't be caught by your <match *> block. Use <match *.*> , <match request.*> , <match **> or something along these lines that makes sense to you.

[tech-talks-india]

Hi @aferreira , I tried the same. Below is my new config:

<match request.*>
    @type mongo
    host 127.0.0.1
    port 27017
    user tech-talks
    password testpassword
    database myapp
    collection ${tag} #tag_mapped is depricated
    capped
    capped_size 100m
</match>

<source>
    @type tail
    path /home/my-app/logs/%Y/%b/app.log
    tag request.main
    format /^\[(?<time>\s*[^\\\]]*)\] (?<ip-address>[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*) (?<request-method>\w*) (?<request-url>[^ ]*) (?<format>[^ ]*) (?<request-size>\d*) (?<user-agent>[^ ]*) (?<response-code>\d*) (?<response-duration>\d*)$/
    time_format %Y-%m-%d %H:%M:%S
    keep_time_key true
    pos_file /tmp/fluentd--1516882649.pos
</source>

Below is the sample Log:

[2018-01-26 10:42:42] 192.168.10.1 GET http://localhost.com/mypage html 0 Mozilla/5.0 200 369

[repeatedly]

I can insert your log, so the problem seems outside of fluentd:

db["request.main"].find()
{ "_id" : ObjectId("5a72d93add126f1ae9a7cb5e"), "time" : ISODate("2018-01-26T01:42:42Z"), "ip-address" : "192.168.10.1", "request-method" : "GET", "request-url" : "http://localhost.com/mypage", "format" : "html", "request-size" : "0", "user-agent" : "Mozilla/5.0", "response-code" : "200", "response-duration" : "369" }

• conf

<match request.*>
  @type mongo
  host 127.0.0.1
  port 27017
  database myapp
  collection ${tag}
  capped
  capped_size 100m
</match>

<source>
    @type tail
    path /tmp/mongo_test.log
    tag request.main
    format /^\[(?<time>\s*[^\\\]]*)\] (?<ip-address>[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*) (?<request-method>\w*) (?<request-url>[^ ]*) (?<format>[^ ]*) (?<request-size>\d*) (?<user-agent>[^ ]*) (?<response-code>\d*) (?<response-duration>\d*)$/
    time_format %Y-%m-%d %H:%M:%S
    keep_time_key true
    read_from_head true # for test
    #pos_file /tmp/fluentd--1516882649.pos
</source>