Yu Watanabe Yu Watanabe - 1 month ago 37
Ruby Question

How to format digits in logstash?

How can I format digit in logstash?
I am using the '' % format expression in ruby code in filter plugin but I get nil as format result. I tried sprintf and format function but same result.

Below is my code snippet.

ruby {
code => "
event.set( 'positioning', event.get('branch_lat') + ',' + event.get('branch_lon') )
event.set( 'report_datetime', event.get('report_date') + '%04d' % event.get('report_time') )
"
}


As a format result, I get below error in the log.

[2016-10-28T12:31:43,217][ERROR][logstash.filters.ruby ] Ruby exception occurred: undefined method `+' for nil:NilClass


My platform information is below.

[root@elk-analytic logstash]# rpm -qi logstash
Name : logstash
Epoch : 1
Version : 5.0.0
Release : 1
Architecture: noarch
Install Date: Thu 27 Oct 2016 01:26:03 PM JST
Group : default
Size : 198320729
License : ASL 2.0
Signature : RSA/SHA512, Wed 26 Oct 2016 01:57:59 PM JST, Key ID d27d666cd88e42b4
Source RPM : logstash-5.0.0-1.src.rpm
Build Date : Wed 26 Oct 2016 01:10:26 PM JST
Build Host : packer-virtualbox-iso-1474648640
Relocations : /
Packager : <vagrant@packer-virtualbox-iso-1474648640>
Vendor : Elasticsearch
URL : http://www.elasticsearch.org/overview/logstash/
Summary : An extensible logging pipeline
Description :
An extensible logging pipeline


Added on 2016.10.28 14:32



My Goal is to parse below csv columns into timestamp field in elasticsearch.
Please notice that hour of time has mixed patterns of 1 and 2 digits.

date,time

20160204,1000

20160204,935

I tried using date function in filter plugin but it did not work properly by logging error.

[2016-10-28T11:00:10,233][WARN ][logstash.filters.date ] Failed parsing date from field {:field=>"report_datetime",
:value=>"20160204 935", :exception=>"Cannot parse \"20160204 935\": Value 93 for hourOfDay must be in the range [0,23]", :config_parsers=>"YYYYMMdd Hmm", :config_locale=>"default=en_US"}


Below is the code snippet when above error appeared.

ruby {
code => "
event.set( 'positioning', event.get('branch_lat') + ',' + event.get('branch_lon') )
event.set( 'report_datetime', event.get('report_date') + ' ' + event.get('report_time') )
"
}

# Set the @timestamp according to report_date and time
date {
"match" => ["report_datetime", "YYYYMMdd Hmm"]
}


I did some modification and ended up with the code I first posted.

Val Val
Answer

I suggest to do it like this without any ruby filter:

filter {
  # your other filters...

  # if 3-digit hours, pad the time with one zero
  if [time] =~ /^\d{3}$/ {
    mutate {
      add_field => { "report_datetime" => "%{date} 0%{time}" }
    }
  # otherwise just concat the fields
  } else {
    mutate {
      add_field => { "report_datetime" => "%{date} %{time}" }
    }
  }

  # match date and time
  date {
    "match"  => ["report_datetime", "yyyyMMdd HHmm"]
    "target" => "report_datetime"
  }
}