Iterating through JSON array in Shell script

Just use a filter that would return each item in the array. Then loop over the results, just make sure you use the compact output option (-c) so each result is put on a single line and is treated as one item in the loop.

jq -c '.[]' input.json | while read i; do
    # do stuff with $i
done

By leveraging the power of Bash arrays, you can do something like:

# read each item in the JSON array to an item in the Bash array
readarray -t my_array < <(jq --compact-output '.[]' input.json)

# iterate through the Bash array
for item in "${my_array[@]}"; do
  original_name=$(jq --raw-output '.original_name' <<< "$item")
  changed_name=$(jq --raw-output '.changed_name' <<< "$item")
  # do your stuff
done

jq has a shell formatting option: @sh.

You can use the following to format your json data as shell parameters:

cat data.json | jq '. | map([.original_name, .changed_name])' | jq @sh

The output will look like:

"'pdf_convert' 'pdf_convert_1'"
"'video_encode' 'video_encode_1'",
"'video_transcode' 'video_transcode_1'"

To process each row, we need to do a couple of things:

• Set the bash for-loop to read the entire row, rather than stopping at the first space (default behavior).
• Strip the enclosing double-quotes off of each row, so each value can be passed as a parameter to the function which processes each row.

To read the entire row on each iteration of the bash for-loop, set the IFS variable, as described in this answer.

To strip off the double-quotes, we'll run it through the bash shell interpreter using xargs:

stripped=$(echo $original | xargs echo)

Putting it all together, we have:

#!/bin/bash

function processRow() {
  original_name=$1
  changed_name=$2

  # TODO
}

IFS=$'\n' # Each iteration of the for loop should read until we find an end-of-line
for row in $(cat data.json | jq '. | map([.original_name, .changed_name])' | jq @sh)
do
  # Run the row through the shell interpreter to remove enclosing double-quotes
  stripped=$(echo $row | xargs echo)

  # Call our function to process the row
  # eval must be used to interpret the spaces in $stripped as separating arguments
  eval processRow $stripped
done
unset IFS # Return IFS to its original value

From Iterate over json array of dates in bash (has whitespace)

items=$(echo "$JSON_Content" | jq -c -r '.[]')
for item in ${items[@]}; do
    echo $item
    # whatever you are trying to do ...
done

Here is a simple example that works in zch shell:

DOMAINS='["google","amazon"]'

arr=$(echo $DOMAINS | jq -c '.[]')
for d in $arr; do
    printf "Here is your domain: ${d}\n"
done

Try Build it around this example. (Source: Original Site)

Example:

jq '[foreach .[] as $item ([[],[]]; if $item == null then [[],.[0]]     else [(.[0] + [$item]),[]] end; if $item == null then .[1] else empty end)]'

Input [1,2,3,4,null,"a","b",null]

Output [[1,2,3,4],["a","b"]]

None of the answers here worked for me, out-of-the-box.

What did work was a combination of a few:

projectList=$(echo "$projRes" | jq -c '.projects[]')

IFS=$'\n' # Read till newline

for project in ${projectList[@]}; do
  projectId=$(jq '.id' <<< "$project")
  projectName=$(jq -r '.name' <<< "$project")
  ...
done

unset IFS

NOTE: I'm not using the same data as the question does, in this example assume projRes is the output from an API that gives us a JSON list of projects, eg:

{
  "projects": [ 
    {"id":1,"name":"Project"}, 
    ... // array of projects
  ] 
}

For the general case, @Jeff's answer is the way to go. It uses jq's --compact-output (or -c) flag to print each iteration result on its own single line, then uses the shell's read function in a while loop to linewise read the results into a shell variable.

jq -c '.[]' input.json | while read i; do
    # do stuff with $i
done

But utilizing that flag comes at the cost of sacrificing the pretty-printing otherwise present in jq's non-compact outputs. If you needed that formatting, the proximate attempt would be to subsequently run other instances of jq on each iteration step to (re-)establish the formatting for each output. However, this can be expensive, especially on large input arrays, and could be generally avoided by retaining the initial formatting while using a delimiter other than the newline character (because the pretty-printed, multi-line output items themselves already do contain newlines characters).

As bash is tagged, one way would be using read's (non-POSIX) -d option to provide a custom delimiter. With an empty string, it defaults to "terminate a line when it reads a NUL character", which can be added to jq's output using "\u0000". As for the jq filter, opening a new context (with |) after the iteration ensures it's printed with every array item. Finally, jq's --join-output (or -j) flag decodes the JSON-encoded NUL character while suppressing the otherwise appended newline characters after each item.

jq -j '.[] | ., "\u0000"' input.json | while read -d '' i; do
    # do stuff with pretty-printed, multi-line "$i"
done

Edit: With jq 1.7, a new flag --raw-output0 (note the 0 at the end) was introduced, which behaves just like the regular --raw-output (or -r) flag, but emits NUL characters instead of newlines after each output printed. It thus perfectly covers the last approach while rendering all its manual arrangements (adding "\u0000" and using the -j flag) unnecessary.

jq --raw-output0 '.[]' input.json | while read -d '' i; do
    # do stuff with pretty-printed, multi-line "$i"
done

An earlier answer in this thread suggested using jq's foreach, but that may be much more complicated than needed, especially given the stated task. Specifically, foreach (and reduce) are intended for certain cases where you need to accumulate results.

In many cases (including some cases where eventually a reduction step is necessary), it's better to use .[] or map(_). The latter is just another way of writing [.[] | _] so if you are going to use jq, it's really useful to understand that .[] simply creates a stream of values. For example, [1,2,3] | .[] produces a stream of the three values.

To take a simple map-reduce example, suppose you want to find the maximum length of an array of strings. One solution would be [ .[] | length] | max.

In case if you are receiving some problems, it might be related to the transformed escape character \" being parsed to just ", which will result in error:

parse error: Invalid escape at line ...

My solution to this problem is using sed. So, if we would transform the query from one of the previous answers, the command would look like this:

jq -c '.[]' input.json | sed 's/\\"/\\'\''/g' | while read i; do
    # do stuff with $i | jq
done

where the command sed 's/\\"/\\'\''/g' would replace all \" with '.

There might be different parsing error scenarios when iterating through the objects. If there would I would adjust the sed command to make the JSON syntax valid.

I stopped using jq and started using jp, since JMESpath is the same language as used by the --query argument of my cloud service and I find it difficult to juggle both languages at once. You can quickly learn the basics of JMESpath expressions here: https://jmespath.org/tutorial.html

Since you didn't specifically ask for a jq answer but instead, an approach to iterating JSON in bash, I think it's an appropriate answer.

Style points:

1. I use backticks and those have fallen out of fashion. You can substitute with another command substitution operator.
2. I use cat to pipe the input contents into the command. Yes, you can also specify the filename as a parameter, but I find this distracting because it breaks my left-to-right reading of the sequence of operations. Of course you can update this from my style to yours.
3. set -u has no function in this solution, but is important if you are fiddling with bash to get something to work. The command forces you to declare variables and therefore doesn't allow you to misspell a variable name.

Here's how I do it:

#!/bin/bash
set -u

# exploit the JMESpath length() function to get a count of list elements to iterate
export COUNT=`cat data.json | jp "length( [*] )"`

# The `seq` command produces the sequence `0 1 2` for our indexes
# The $(( )) operator in bash produces an arithmetic result ($COUNT minus one)
for i in `seq 0 $((COUNT - 1))` ; do

     # The list elements in JMESpath are zero-indexed
     echo "Here is element $i:"
     cat data.json | jp "[$i]"

     # Add or replace whatever operation you like here.

done

Now, it would also be a common use case to pull the original JSON data from an online API and not from a local file. In that case, I use a slightly modified technique of caching the full result in a variable:

#!/bin/bash
set -u

# cache the JSON content in a stack variable, downloading it only once
export DATA=`api --profile foo compute instance list --query "bar"`

export COUNT=`echo "$DATA" | jp "length( [*] )"`
for i in `seq 0 $((COUNT - 1))` ; do
     echo "Here is element $i:"
     echo "$DATA" | jp "[$i]"
done

This second example has the added benefit that if the data is changing rapidly, you are guaranteed to have a consistent count between the elements you are iterating through, and the elements in the iterated data.

I think in most cases, using JSON and jq is not useful.

But it's normal to try to use what we learnt in other languages.

Instead we could use BASH arrays and iterators, to get the best of best world, so we could craft advanced stuffs and roll math, while keeping the sugar syntax $fruit.

LIST=("orange" "apple" "banana")

I=$(("0")) # String to number
for fruit in "${LIST[@]}"; do
    echo "$I: $fruit"
    I=$(( $I + 1 )) # Iterator: Make your own
done

# Or direct read by key
echo "${LIST[2]}"

Output:

0: orange
1: apple
2: banana
banana

This is what I have done so far

 arr=$(echo "$array" | jq -c -r '.[]')
            for item in ${arr[@]}; do
               original_name=$(echo $item | jq -r '.original_name')
               changed_name=$(echo $item | jq -r '.changed_name')
              echo $original_name $changed_name
            done