Abdullah Al Hafiz Abdullah Al Hafiz - 25 days ago 4
JSON Question

Add a header line before each json document

I have a json file with 1000 json object.
is there any way to add a header line before each json document ? Is there any easiest way ?

Example : I have 1000 object like this

{"id":58,"first_name":"Louis","last_name":"Jordan","email":"ljordan1l@nature.com","gender":"Male","Latitude":"-15.93444","Longitude":"-50.14028"}


i want to add index header like below for every json object so that i can use in Elasticsearch Bulk api

{ "index" : { "_index" : "test", "_type" : "type1", "_id" : "unique_id" } }
{"id":58,"first_name":"Louis","last_name":"Jordan","email":"ljordan1l@nature.com","gender":"Male","Latitude":"-15.93444","Longitude":"-50.14028"}

Val Val
Answer

If you are willing to leverage Logstash, you don't need to modify your file and can simply read it line by line and stream it to ES using the elasticsearch output which leverages the Bulk API.

Store the following Logstash configuration in a file named es.conf (make sure the file path and ES hosts match your settings):

input {
  file {
    path => "/path/to/your/json"
    sincedb_path => "/dev/null"
    start_position => "beginning"
    codec => "json"
  }
}
filter {
  mutate {
    remove_fields => ["@version", "@timestamp"]
  }
}
output {
  elasticsearch {
    hosts => "localhost:9200"
    index => "test"
    document_type => "type1"
    document_id => "%{id}"
  }
}

Then, you need to install logstash and you'll be able to run the following command in order to load your JSON files to your ES server:

bin/logstash -f es.conf