Hej dear community,
I am right now trying to import data from an API call (and processed the JSON Output in R) into an index at elastic search.
"stored" is a dataframe containing 20 obs. along 113 variables. However, elastic search copies only 7 out of 20 obs. into the index. Those are correctly transfered in terms of values.
Still, I cannot explain where and why I am missing the other 13 observations. The code I am using, see below
stored <- fromJSON(API_URL)
stored <- stored[['results']]
connect(es_base = "xxx.xxx.x.xx", es_port = xxxx)
docs_bulk(stored, index="data", raw = FALSE, chunk_size = 100000)
Thanks to Sckott, we were able to solve the problem.
The Json file from the API Call were not 100% - UTF8 encoded. By using fromJSON for the URL-Call it entered additional characters to the data. However, adding a readLines avoids the problem. The final code i used were:
Output_FT <- fromJSON(readLines(BWURL_x), flatten = TRUE) stored <- Output_FT[['results']] connect(es_base = "xxx.xxx.x.xx", es_port = xxxx) connection() docs_bulk(stored, index="data")