bash scripting - how to loop through results called within a function

Question

I have a function that gives me a list of IPs and for each IP in my list, I want to run a query. The problem I'm having is its only looping through (1) of the results and not the rest.

getPartition ()
{
    _knife=$(which knife);
    _grep=$(which grep);
    _awk=$(which awk);
    cd ~/home/foo/.chef
    local result=$(${_knife} search "chef_environment:dev AND role:myapp AND ec2_region:us-east-1" | ${_grep} IP | ${_awk} '{ print $2 }');
   read -a servers <<< $result; 
   echo "Checking ${#servers[@]} servers";
   for i in ${servers[@]};
   do
       local host='10.1.2.123'
       local db='mystate'
       _mongo=$(which mongo);
       echo -n "$i";
       local exp="db.foobarcluster_servers.find(
       {\"node_host\":\"${i}\",\"node_type\":\"PROCESS\",\"region\":\"us-east-1\",\"status\":\"ACTIVE\"},{\"partition_range_start\":1,\"partition_range_end\":1, _id:0}).pretty();";
       ${_mongo} ${host}/${db} --eval "$exp" | grep -o -e "{[^}]*}";
  done
}

So, I tried using for, but its only running the query for (1) of the (5) hosts listed.

I can see in my output for result that the list of IPs look like this:

+ local 'result=10.8.3.34
10.8.2.161
10.8.3.514
10.8.4.130
10.8.2.173'

So, I'm just returning results for (1) of the IPs it should be (5) of them because I have 5 IPs:

+ read -a servers
+ echo 'Checking 1 servers'
Checking 1 servers
+ for i in ${servers[@]}
+ local host=10.1.2.130
+ local db=mystate
++ which mongo
+ _mongo=/usr/local/bin/mongo
+ echo -n 10.8.3.34
10.8.3.34+ local 'exp=db.foobarcluster_servers.find(
       {"node_host":"10.8.3.34","node_type":"PROCESS","region":"us-east-1","status":"ACTIVE"},{"partition_range_start":1,"partition_range_end":1, _id:0}).pretty();'
+ /usr/local/bin/mongo 10.8.3.34/mystate --eval 'db.foobarcluster_servers.find(
       {"node_host":"10.8.3.34","node_type":"PROCESS","region":"us-east-1","status":"ACTIVE"},{"partition_range_start":1,"partition_range_end":1, _id:0}).pretty();'
+ grep -o -e '{[^}]*}'
{ "partition_range_start" : 31, "partition_range_end" : 31 }
+ set +x

Results:

{ "partition_range_start" : 31, "partition_range_end" : 31 }

I'm expecting:

{ "partition_range_start" : 31, "partition_range_end" : 31 }
{ "partition_range_start" : 32, "partition_range_end" : 32 }
{ "partition_range_start" : 33, "partition_range_end" : 33 }
{ "partition_range_start" : 34, "partition_range_end" : 34 }
{ "partition_range_start" : 35, "partition_range_end" : 35 }

How do I effectively loop through my IPs? Did I set up result properly as a variable to hold that list of IPs?

What's the output of ${servers[@])?

Shammel Lee
– Shammel Lee

2017-06-06 06:55:45 +00:00
Commented Jun 6, 2017 at 6:55 — Shammel Lee
– Shammel Lee, Commented Jun 6, 2017 at 6:55

dimo414 · Accepted Answer · 2017-06-07 16:19:27Z

Good idea using set -x - another good debugging tactic (that also makes reading set -x easier) would be to comment out parts that aren't relevant to the issue (e.g. make the for loop simply print its iterations, hard-code the value of result, etc.) to try to narrow down the issue.

If I try to replicate what you're doing myself:

demo() {
  local result='10.8.3.34
10.8.2.161
10.8.3.514
10.8.4.130
10.8.2.173'
  read -a servers <<< $result
  echo "Checking ${#servers[@]} servers"
  for i in ${servers[@]}; do
    echo "$i"
  done
}

Which outputs (with set -x):

$ demo
+ demo
+ local 'result=10.8.3.34
10.8.2.161
10.8.3.514
10.8.4.130
10.8.2.173'
+ read -a servers
+ echo 'Checking 5 servers'
Checking 5 servers
+ for i in '${servers[@]}'
+ echo 10.8.3.34
10.8.3.34
+ for i in '${servers[@]}'
+ echo 10.8.2.161
10.8.2.161
+ for i in '${servers[@]}'
+ echo 10.8.3.514
10.8.3.514
+ for i in '${servers[@]}'
+ echo 10.8.4.130
10.8.4.130
+ for i in '${servers[@]}'
+ echo 10.8.2.173
10.8.2.173

In other words, the code you shared appears to be working as expected. Perhaps there's a typo you corrected while transcribing?

A key thing to note (per help read) is that read "Reads a single line from the standard input ... the line is split into fields as with word splitting". In other words, a multi-line input does not all get read by a call to read, only the first line does. We can test this by tweaking the demo function above to use:

read -a servers <<< "$result"

Which causes the output you describe:

$ demo
+ demo
+ local 'result=10.8.3.34
    10.8.2.161
    10.8.3.514
    10.8.4.130
    10.8.2.173'
+ read -a servers
+ echo 'Checking 1 servers'
Checking 1 servers
+ for i in '${servers[@]}'
+ echo 10.8.3.34
10.8.3.34

So that's likely the source of your issue - by quoting $result (which generally is a good idea) read respects the newlines separating the elements, and stops reading after it sees the first one.

Instead use the readarray command, which has more sane behavior for tasks like this. It will "read lines from a file into an array variable", rather than stopping after the first line.

You can then skip the indirection of writing to result, as well, and just pipe directly into readarray:

readarray -t servers < <(
  ${_knife} search "chef_environment:dev AND role:myapp AND ec2_region:us-east-1"
    | ${_grep} IP | ${_awk} '{ print $2 }')

thanks for the breakdown and showing readarray as an example. that works well for me.

Nahuel Fouilleul · Accepted Answer · 2017-06-06 07:06:52Z

1

it's because read reads only until first input line delimiter "\n" adding option -d '', reads until end of input

result=serv1$'\n'serv2$'\n'serv3

read -a servers <<< $result
printf "<%s>\n" "${servers[@]}"

read -d '' -a servers <<< $result
printf "<%s>\n" "${servers[@]}"

There's also readarray builtin which can be used to read an array

readarray -t servers <<< $result
printf "<%s>\n" "${servers[@]}"

-t to remove the newlines for each element of the array

edited Jun 6, 2017 at 7:06

answered Jun 6, 2017 at 6:58

Nahuel Fouilleul

19.5k2 gold badges36 silver badges37 bronze badges

Comments

Shammel Lee · Accepted Answer · 2017-06-06 07:29:03Z

1

The problem is that…

read -a servers <<< $result

is only reading the first line into the array

Change that line to…

servers=( $result )

This converts every whitespace-delimited value in $result into an array element. Effectively servers=( <ip> <ip> <ip> <ip> <ip> )

edited Jun 6, 2017 at 7:29

answered Jun 6, 2017 at 7:18

Shammel Lee

4,5351 gold badge19 silver badges21 bronze badges

1 Comment

dimo414 Over a year ago

result isn't an array.

Collectives™ on Stack Overflow

bash scripting - how to loop through results called within a function

3 Answers 3

1 Comment

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related