Infinite Infinite - 5 months ago 73
Ruby Question

Logstash - Adding execption for json escape character

I googled a solution to a for this issue and I added the Ruby code. Problem is if I use

gsub
I also get an error.

I please need a solution either to fix the Ruby error or another method to add an exception for the escape character.

Here is my Logstash filter:

filter {
if [type] == "apache" {
json {
source => "message"
}
ruby {
code => "str=event['url']; str=str.gsub('\\','\*'); event['url']=str;"
}
}
}


I am getting my Logs from Apache in json format. Here is the Logstash error message:

{:timestamp=>"2016-06-28T12:49:21.821000+0100", :message=>"Error
parsing json", :source=>"message", :raw=>"{ \"@version\": \"1\",
\"@timestamp\": \"2016-06-28T12:49:16.000+0100\", \"message\": \"GET
/yolo/q-ap%C3%Ablah/?search%5Bdist%5D=15&page=13 HTTP/1.1\", \"via\":
\"192.168.220.100\", \"client-ip\": \"123.123.123.123\",
\"remote-logname\": \"-\", \"remote-user\": \"-\", \"recv-time\":
\"[28/Jun/2016:12:49:16 +0100]\", \"serve-time-microsec\": \"88613\",
\"request\": \"GET /yolo/q-ap%C3%Ablah/?search%5Bdist%5D=15&page=13
HTTP/1.1\", \"status\": \"200\", \"size\": \"184985\", \"referer\":
\"-\", \"user-agent\": \"Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)\", \"url\": \"/yolo/q-ap\\xc3\\xablah/\", \"query\":
\"?search%5Bdist%5D=15&page=13\", \"method\": \"GET\", \"protocol\":
\"HTTP/1.1\", \"vhost\": \"www.site.com\", \"received-size\": \"1136\"
}", :exception=>#LogStash::Json::ParserError: Unrecognized character
escape 'x' (code 120) at Source: [B@2ebf051e; line: 1, column: 599>,
:level=>:warn}

{:timestamp="2016-06-28T12:49:21.821000+0100", :message=>"Ruby
exception occurred: undefined method `gsub' for nil:NilClass",
:level=>:error}

Answer

I was able to solve my issue thanks to the logstash team. https://discuss.elastic.co/t/add-execption-for-special-character-from-json-log/54285/3

Here is the correct config:

filter {
  if [type] == "apache" {

    ruby {
        code => " if event['message']
                  event['message'] = event['message'].gsub('\x','Xx')
                  event['message'] = event['message'].gsub('\\x','XXx')
                  end
                "
            }

    json {
          source => "message"
        }

 }
}

Note the ruby needs to be above the json formatting.

Comments