hgl hgl - 9 months ago 26
JSON Question

How to merge a remote json referenced by a url string into the current json

Given

[
{"json1": "http://example.com/remote1.json"},
{"json2": "http://example.com/remote2.json"}
]


with
remote1.json
and
remote2.json
containing
[1]
and
[2]
respectively

How to turn it into

[{"json1": [1], "json2": [2]}]


using jq? I think other CLI tools like bash and curl are needed. But I have no idea how to merge the responses back.

Answer Source

First, our test framework:

curl() {
  case $1 in
    http://example.com/remote1.json) echo "[1]" ;;
    http://example.com/remote2.json) echo "[2]" ;;
    *) echo "IMABUG" ;;
  esac
}
input_json='[
  {"json1": "http://example.com/remote1.json"},
  {"json2": "http://example.com/remote2.json"}
]'

Then, our actual code:

# defines the "walk" function, which is not yet included in a released version of jq
# ...in the future, this will not be necessary.
walk_fn='
def walk(f):
  . as $in
  | if type == "object" then
      reduce keys[] as $key
        ( {}; . + { ($key):  ($in[$key] | walk(f)) } ) | f
  elif type == "array" then map( walk(f) ) | f
  else f
  end;
'

get_url_keys() {
  jq -r "$walk_fn
    walk(
      if type == \"object\" then
        to_entries
      else . end
    )
    | flatten
    | .[]
    | select(.value | test(\"://\"))
    | [.key, .value]
    | @tsv"
}

operations=( )
options=( )
i=0
while IFS=$'\t' read -r key url; do
  options+=( --arg "key$i" "$key" --argjson "value$i" "$(curl "$url")" )
  operations+=(
    " walk(
        if type == \"object\" then
          if .[\$key$i] then .[\$key$i]=\$value$i else . end
        else . end
      ) "
  )
  (( ++i ))
done < <(get_url_keys <<<"$input_json")

IFS='|' # separate operations with a | character
jq -c "${options[@]}" "${walk_fn} ${operations[*]}" <<<"$input_json"

Output is properly:

[{"json1":[1]},{"json2":[2]}]