Sofo56 Sofo56 - 4 months ago 41x
JSON Question

Importing Data into Elastic Search from R

Hej dear community,

I am right now trying to import data from an API call (and processed the JSON Output in R) into an index at elastic search.

"stored" is a dataframe containing 20 obs. along 113 variables. However, elastic search copies only 7 out of 20 obs. into the index. Those are correctly transfered in terms of values.

Still, I cannot explain where and why I am missing the other 13 observations. The code I am using, see below

stored <- fromJSON(API_URL)
stored <- stored[['results']]

connect(es_base = "", es_port = xxxx)

docs_bulk(stored, index="data", raw = FALSE, chunk_size = 100000)

Thanks in advanced :-)


Thanks to Sckott, we were able to solve the problem.

The Json file from the API Call were not 100% - UTF8 encoded. By using fromJSON for the URL-Call it entered additional characters to the data. However, adding a readLines avoids the problem. The final code i used were:

Output_FT <- fromJSON(readLines(BWURL_x), flatten = TRUE)
stored <- Output_FT[['results']]
connect(es_base = "", es_port = xxxx)
docs_bulk(stored, index="data")