How to pipe input to a Bash while loop and preserve variables after loop ends

Question

Bash allows to use: cat <(echo "$FILECONTENT")

Bash also allow to use: while read i; do echo $i; done </etc/passwd

to combine previous two this can be used: echo $FILECONTENT | while read i; do echo $i; done

The problem with last one is that it creates sub-shell and after the while loop ends variable i cannot be accessed any more.

My question is:

How to achieve something like this: while read i; do echo $i; done <(echo "$FILECONTENT") or in other words: How can I be sure that i survives while loop?

Please note that I am aware of enclosing while statement into {} but this does not solves the problem (imagine that you want use the while loop in function and return i variable)

Related: stackoverflow.com/questions/37229058/… . Explains all options including the below-mentioned process substitution and lastpipe and their pros and cons. — ivan_pozdeev
– ivan_pozdeev, Commented Jun 21, 2017 at 23:02
Related: A variable modified inside a while loop is not remembered. — codeforester
– codeforester, Commented Aug 1, 2017 at 21:10
@codeforester there is more the one dup... Related: Why piping input to "read" only works when fed into "while read ..." construct? and Read values into a shell variable from a pipe — F. Hauri - Give Up GitHub
– F. Hauri - Give Up GitHub, Commented Jun 13, 2022 at 13:21

Community · Accepted Answer · 2020-06-20 09:12:55Z

163

The correct notation for Process Substitution is:

while read i; do echo $i; done < <(echo "$FILECONTENT")

The last value of i assigned in the loop is then available when the loop terminates. An alternative is:

echo $FILECONTENT | 
{
while read i; do echo $i; done
...do other things using $i here...
}

The braces are an I/O grouping operation and do not themselves create a subshell. In this context, they are part of a pipeline and are therefore run as a subshell, but it is because of the |, not the { ... }. You mention this in the question. AFAIK, you can do a return from within these inside a function.

Bash also provides the shopt builtin and one of its many options is:

lastpipe

If set, and job control is not active, the shell runs the last command of a pipeline not executed in the background in the current shell environment.

Thus, using something like this in a script makes the modfied sum available after the loop:

FILECONTENT="12 Name
13 Number
14 Information"
shopt -s lastpipe   # Comment this out to see the alternative behaviour
sum=0
echo "$FILECONTENT" |
while read number name; do ((sum+=$number)); done
echo $sum

Doing this at the command line usually runs foul of 'job control is not active' (that is, at the command line, job control is active). Testing this without using a script failed.

Also, as noted by Gareth Rees in his answer, you can sometimes use a here string:

while read i; do echo $i; done <<< "$FILECONTENT"

This doesn't require shopt; you may be able to save a process using it.

edited Jun 20, 2020 at 9:12

CommunityBot

11 silver badge

answered Oct 24, 2013 at 16:07

Jonathan Leffler

759k145 gold badges961 silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Wakan Tanka Over a year ago

Pardon my ignorance. I know this is right solution and I've marked it as answer so it worked for me. But now when I run while read i; do echo $i; done < <(cat /etc/passwd); echo $i It did not return last line two times. What I am doing wrong?

Jonathan Leffler Over a year ago

@WakanTanka: I had to experiment a bit...I believe the answer is that the failing read resets i to empty, so the echo after the loop echoes a blank line.

Atcold Over a year ago

Kudos for the Process Substitution reference. I was unaware of.

yurenchen Over a year ago

@Wakan Tanka: got same result as yours, i use while read i; do x=$i; done < <(cat /etc/passwd); echo i=$i; echo x=$x worked, //maybe those days bash behavior changed ?

Pat Over a year ago

You're a life saver! Been looking for hours for a similar solution. Yours is the only one that worked.

|

Community · Accepted Answer · 2017-05-23 11:47:11Z

33

Jonathan Leffler explains how to do what you want using process substitution, but another possibility is to use a here string:

while read i; do echo "$i"; done <<<"$FILECONTENT"

This saves a process.

edited May 23, 2017 at 11:47

CommunityBot

11 silver badge

answered Oct 24, 2013 at 16:14

Gareth Rees

66.1k10 gold badges139 silver badges166 bronze badges

Comments

Steve · Accepted Answer · 2020-08-02 02:24:18Z

1

This function makes duplicates $NUM times of jpg files (bash)

function makeDups() {
NUM=$1
echo "Making $1 duplicates for $(ls -1 *.jpg|wc -l) files"
ls -1 *.jpg|sort|while read f
do
  COUNT=0
  while [ "$COUNT" -le "$NUM" ]
  do
    cp $f ${f//sm/${COUNT}sm}
    ((COUNT++))
  done
done
}

answered Aug 2, 2020 at 2:24

Steve

312 bronze badges

Comments

Alberto Salvia Novella · Accepted Answer · 2023-06-01 14:31:39Z

1

The fastest was:

while read -r ITEM; do
    OPERATION
done < <(COMMAND)

Or in one line:

while read -r ITEM; do OPERATION; done < <(COMMAND)

edited Jun 1, 2023 at 14:31

answered Jun 1, 2023 at 13:39

Alberto Salvia Novella

1,33915 silver badges17 bronze badges

Comments

ocodo · Accepted Answer · 2023-08-01 05:34:55Z

1

Bash provides var default behavior so:

while read -r line
do
  total=${total:-0}
  total=$((total + $(calculation code)))
done
echo $total

This allows the loop to easily capture stdin, for scripts and functions.

answered Aug 1, 2023 at 5:34

ocodo

30.5k19 gold badges111 silver badges127 bronze badges

Collectives™ on Stack Overflow

How to pipe input to a Bash while loop and preserve variables after loop ends

5 Answers 5

7 Comments

Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

7 Comments

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related