ruby - logstash elasticsearch output -

i'm having issues while i've tried change naming convention elasticsearch index in logstash conf file. need use part of file name passed through logstash pipeline, part set date of data contained file. so, instead of using standard naming convention, is, long can read: logstash-%{+yyyy.mm.dd}, need this: -.

i trying actual name of file passed through pipeline, not know how it. decided use year , month of current line being processed in filter section. grok pattern use:

  grok {     match => [ "message", "%{ip:client} %{notspace:sep} %{notspace:ident} %{notspace:inbracket}%{monthday:day}/%{month:month}/%{year:year}:%{hour:hour}:%{minute:minute}:%{second:second} %{iso8601_timezone:tz}%{notspace:outbracket} \"%{word:method} %{notspace:uri} %{notspace:http_version}\" %{number:code} %{number:size} %{notspace:action_hierarchy} %{notspace:content_type}" ]     remove_field => ["sep"]     remove_field => ["inbracket"]     remove_field => ["outbracket"]   } 

as can seen, "year"and "month"are 2 of fields can recover after applying grok pattern. thought this:

elasticsearch {     action => "index"     index => "myindexname-%{year}.%{month}"     index_type => "logs"     node_name => "node001" } 

but nor "year" nor "month" can used in section: there's no compilation problem in conf file, not way getting values. maybe using ruby way, attempts wrong. how can achieve this?

---

so, have same issue, solution works me.

the code plugin is:

# call file 'ordinalmonth.rb' (in logstash/filters, above) require "logstash/filters/base" require "logstash/namespace"  class logstash::filters::ordinalmonth < logstash::filters::base  # setting config_name here required. how # configure filter logstash config. # # filter { #   ordinalmonth { ... } # }   config_name "ordinalmonth"  # new plugins should start life @ milestone 1. milestone 2  # replace message value. config :month_field, :validate => :string, :default => "month"  public def register   # nothing end # def register  public def filter(event)   # return nothing unless there's actual filter event   return unless filter?(event)   if event[@month_field]     # replace event message our message configured in     # config file.     tmp = case event[@month_field]       when "jan" "01"       when "feb" "02"       when "mar" "03"       when "apr" "04"       when "may" "05"       when 'jun' '06'       when "jul" "07"       when "aug" "08"       when "sep" "09"       when "oct" "10"       when "nov" "11"       when "dec" "12"       else "unknown"       end     event["month"] = tmp   end   # filter_matched should go in last line of our successful code    filter_matched(event) end # def filter end # class logstash::filters::ordinalmonth 

basically, plugin receive name of filed contain name of month 3 letters, starting capital one. enters in case statement, updating can achieved. then, alters old value contained in field.

so, works in expected manner, had change code in configuration file logstash job:

filter {   if [type] == "nauta_navroom" {       grok {         match => [ "message", "%{ip:client} %{notspace:sep} %{notspace:ident} %{notspace:inbracket}%{notspace:day}/%{month:month}/%{year:year}:%{hour:hour}:%{minute:minute}:%{second:second} %{iso8601_timezone:tz}%{notspace:outbracket} \"%{word:method} %{notspace:uri} %{notspace:http_version}\" %{number:code} %{number:size} %{notspace:action_hierarchy} %{notspace:content_type}" ]         remove_field => ["sep"]         remove_field => ["inbracket"]         remove_field => ["outbracket"]       }       ordinalmonth {}       kv {         source => "@message"       }   } } 

check out invocation of ordinalmonth plugin, without parameters. other magic thing using kv filter, makes changes visible outside filter.

and that's it. hope can useful needs it.

logstash builds index name time in @timestamp field (which defaults "now"). want parser time file , use set timestamp.

for example, have pattern of %{monthday:day}/%{month:month}/%{year:year}:%{hour:hour}:%{minute:minute}:%{second:second} %{iso8601_timezone:tz} in file, in config file:

mutate {   add_field => [ "timestamp", "%{year}-%{month}-%{day}t%{hour}:%{minute}:%{second}%{tz}" ] } date {   match => [ "timestamp", "iso8601" ]   remove_field => ["timestamp" ] } 

which adds timestamp field event based on parsed out, sets @timstamp based on , removes field added.

then you'll need change elasticsearch output

elasticsearch {     action => "index"     index => "myindexname-%{+yyyy-mm}"     index_type => "logs"     node_name => "node001" }