Jeff Learman Jeff Learman - 1 year ago 63
JSON Question

How can I completely sort arbitrary JSON using jq?

I want to diff two JSON text files. Unfortunately they're constructed in arbitrary order, so I get diffs when they're semantically identical. I'd like to use jq (or whatever) to sort them in any kind of full order, to eliminate differences due only to element ordering.

--sort-keys solves half the problem, but it doesn't sort arrays.

I'm pretty ignorant of jq and don't know how to write a jq recursive filter that preserves all data; any help would be appreciated.

I realize that line-by-line 'diff' output isn't necessarily the best way to compare two complex objects, but in this case I know the two files are very similar (nearly identical) and line-by-line diffs are fine for my purposes.

Using jq or alternative command line tools to diff JSON files answers a very similar question, but doesn't print the differences. Also, I want to save the sorted results, so what I really want is just a filter program to sort JSON.

Answer Source

Here is a solution using the generic function walk/1, which is a built-in in versions of jq > 1.5, and can therefore be omitted if your jq includes it, but there is no harm in including it redundantly in a jq script.


# Apply f to composite entities recursively, and to atoms
def walk(f):
  . as $in
  | if type == "object" then
      reduce keys[] as $key
        ( {}; . + { ($key):  ($in[$key] | walk(f)) } ) | f
  elif type == "array" then map( walk(f) ) | f
  else f

def normalize: walk(if type == "array" then sort else . end);


Example using bash:

diff <(jq -S -f normalize.jq FILE1) <(jq -S -f normalize.jq FILE2)