Shlomi Schwartz Shlomi Schwartz - 1 month ago 7
JSON Question

Logstash - Parsing and mutating JSON file

Having the following JSON file:

{
"count": 2,
"status": {"partial": true},
"records": [
{
"info": {
"startTime": "2016-07-17 08:42:40.212+0000",
"endTime": "2016-07-17 08:43:47.715+0000",
"id": "123456789"
},
"conversation": {
"lines": [
{
"time": "2016-07-17 08:42:32.533+0000",
"text": "Hi There",
"user": "user A"
},
{
"time": "2016-07-17 08:42:36.533+0000",
"text": "Hello",
"user": "user B"
}
]
}
},
{
"info": {
"startTime": "2016-07-18 08:42:40.212+0000",
"endTime": "2016-07-18 08:43:47.715+0000",
"id": "4567890"
},
"conversation": {
"lines": [
{
"time": "2016-07-17 08:42:32.533+0000",
"text": "Hi There",
"user": "user X"
},
{
"time": "2016-07-17 08:42:36.533+0000",
"text": "Hello",
"user": "user Y"
}
]
}
}
]
}


EDIT (raw format):

{"count": 20,"status": {"partial": true},"records": [{"info": {"startTime": "2016-07-17 08:42:40.212+0000","endTime": "2016-07-17 08:43:47.715+0000","id": "123456789"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user A"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user B"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}},
{"info": {"startTime": "2016-07-18 08:42:40.212+0000","endTime": "2016-07-18 08:43:47.715+0000","id": "4567890"},"conversation": {"lines": [{"time": "2016-07-17 08:42:32.533+0000","text": "Hi There","user": "user X"},{"time": "2016-07-17 08:42:36.533+0000","text": "Hello","user": "user Y"}]}}
]}


I would like to use logstash to import
conversation.lines
(ignoring the rest of the information like
info
) for each of the records, and maybe run some logic like removing some of the lines depending on the time property.

Is it possible to do that with Logstash alone, or should I preprocess the file?

Val Val
Answer

I think the easiest would be to use node.js.

  1. require your JSON file
  2. loop over the records
  3. loop over the conversation.lines
  4. apply your logic
  5. send each line to ES with the JS ES client

Logstash is very good at parsing text files line by line, but if you want to parse a multi-line JSON file, that would not be my first choice.