Create JSON using jq from pipe-separated keys and values in bash

I know this is an old post, but the tool you seek is called jo: https://github.com/jpmens/jo

A quick and easy example:

$ jo my_variable="simple"
{"my_variable":"simple"}

A little more complex

$ jo -p name=jo n=17 parser=false
{
  "name": "jo",
  "n": 17,
  "parser": false
}

Add an array

$ jo -p name=jo n=17 parser=false my_array=$(jo -a {1..5})
{
  "name": "jo",
  "n": 17,
  "parser": false,
  "my_array": [
    1,
    2,
    3,
    4,
    5
  ]
}

I've made some pretty complex stuff with jo and the nice thing is that you don't have to worry about rolling your own solution worrying about the possiblity of making invalid json.


Prerequisite

For all of the below, it's assumed that your content is in a shell variable named s:

s='CONTAINER|CPU%|MEMUSAGE/LIMIT|MEM%|NETI/O|BLOCKI/O|PIDS
nginx_container|0.02%|25.09MiB/15.26GiB|0.16%|0B/0B|22.09MB/4.096kB|0'

What (modern jq)

# thanks to @JeffMercado and @chepner for refinements, see comments
jq -Rn '
( input  | split("|") ) as $keys |
( inputs | split("|") ) as $vals |
[[$keys, $vals] | transpose[] | {key:.[0],value:.[1]}] | from_entries
' <<<"$s"

How (modern jq)

This requires very new (probably 1.5?) jq to work, and is a dense chunk of code. To break it down:

  • Using -n prevents jq from reading stdin on its own, leaving the entirety of the input stream available to be read by input and inputs -- the former to read a single line, and the latter to read all remaining lines. (-R, for raw input, causes textual lines rather than JSON objects to be read).
  • With [$keys, $vals] | transpose[], we're generating [key, value] pairs (in Python terms, zipping the two lists).
  • With {key:.[0],value:.[1]}, we're making each [key, value] pair into an object of the form {"key": key, "value": value}
  • With from_entries, we're combining those pairs into objects containing those keys and values.

What (shell-assisted)

This will work with a significantly older jq than the above, and is an easily adopted approach for scenarios where a native-jq solution can be harder to wrangle:

{
   IFS='|' read -r -a keys # read first line into an array of strings

   ## read each subsequent line into an array named "values"
   while IFS='|' read -r -a values; do

    # setup: positional arguments to pass in literal variables, query with code    
    jq_args=( )
    jq_query='.'

    # copy values into the arguments, reference them from the generated code    
    for idx in "${!values[@]}"; do
        [[ ${keys[$idx]} ]] || continue # skip values with no corresponding key
        jq_args+=( --arg "key$idx"   "${keys[$idx]}"   )
        jq_args+=( --arg "value$idx" "${values[$idx]}" )
        jq_query+=" | .[\$key${idx}]=\$value${idx}"
    done

    # run the generated command
    jq "${jq_args[@]}" "$jq_query" <<<'{}'
  done
} <<<"$s"

How (shell-assisted)

The invoked jq command from the above is similar to:

jq --arg key0   'CONTAINER' \
   --arg value0 'nginx_container' \
   --arg key1   'CPU%' \
   --arg value1 '0.0.2%' \
   --arg key2   'MEMUSAGE/LIMIT' \
   --arg value2 '25.09MiB/15.26GiB' \
   '. | .[$key0]=$value0 | .[$key1]=$value1 | .[$key2]=$value2' \
   <<<'{}'

...passing each key and value out-of-band (such that it's treated as a literal string rather than parsed as JSON), then referring to them individually.


Result

Either of the above will emit:

{
  "CONTAINER": "nginx_container",
  "CPU%": "0.02%",
  "MEMUSAGE/LIMIT": "25.09MiB/15.26GiB",
  "MEM%": "0.16%",
  "NETI/O": "0B/0B",
  "BLOCKI/O": "22.09MB/4.096kB",
  "PIDS": "0"
}

Why

In short: Because it's guaranteed to generate valid JSON as output.

Consider the following as an example that would break more naive approaches:

s='key ending in a backslash\
value "with quotes"'

Sure, these are unexpected scenarios, but jq knows how to deal with them:

{
  "key ending in a backslash\\": "value \"with quotes\""
}

...whereas an implementation that didn't understand JSON strings could easily end up emitting:

{
  "key ending in a backslash\": "value "with quotes""
}

Tags:

Docker

Bash

Json

Jq