Bash shell script Nested while loop with IFS

Question

I'm trying to parse a set of csv files using bash shell script the files looks as below:

File1: /tmp/test.txt
key1,key1file.txt
key2,key2file.txt
key3,key3file.txt

Files: /tmp/inter/key1file.txt
abc,cdf,123,456

Files: /tmp/inter/key2file.txt
abc,cdf,123,456

Files: /tmp/inter/key3file.txt
abc,cdf,123,456

I've tried parsing these files using 2 while loops:

while IFS="," read keycol keyfile
do
    while IFS="," read keyval
     do
     echo "inside inner while loop"
     echo "$keycol|$keyval"
    done < "/tmp/inter/$keyfile"
done < /tmp/test.txt

and expecting this code to output

key1,abc
key1,cdf
key1,123
key1,456 and so on...

However, i'm not getting any output when i run this code which indicates the second loop is not being executed. Any pointers in the right direction would be of help. Thanks

You have /tmp/test.txt in one place and /tmp/test.csv in another. Is that it? — John Kugelman
– John Kugelman, Commented Jul 11, 2017 at 12:04
Add a printout to the outer loop. Is that loop even running? — John Kugelman
– John Kugelman, Commented Jul 11, 2017 at 12:05
In general, read is not up to the task of parsing a CSV file; it cannot accommodate commas that are escaped to be part of a field, rather than separating two fields. Use a language that has a proper CSV parser available. — chepner
– chepner, Commented Jul 11, 2017 at 13:17

randomir · Accepted Answer · 2017-07-11 12:20:11Z

4

You are not properly splitting by , in your second loop. read generally splits by IFS, and assigns values to variables, one field per variable, and the remaining goes into the last variable provided. But if you provide only one variable, everything just gets stored there.

Instead, let read split by , into an array, then loop over values in that array, like this:

#!/bin/bash
while IFS="," read keycol keyfile; do
    while IFS="," read -a values; do
        for val in "${values[@]}"; do
            echo "$keycol,$val"
        done
    done < "/tmp/inter/$keyfile"
done < /tmp/test.txt

You'll get:

key1,abc
key1,cdf
key1,123
key1,456
key2,abc
key2,cdf
key2,123
key2,456
key3,abc
key3,cdf
key3,123
key3,456

edited Jul 11, 2017 at 12:20

answered Jul 11, 2017 at 12:13

randomir

18.8k1 gold badge46 silver badges60 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Ara Over a year ago

Thanks randomir, your suggestion worked. For some reason the line feed was missing in the files under inter folder, i believe that caused the inner loop not work.

randomir Over a year ago

You're welcome. The problem was in inner loop read splitting. Each line read got from inter/$keyfile it "split" it into exactly one variable. The key was to use -a, to split the line into multiple values and store them inside an bash array.

KargWare Over a year ago

Great answer!. How is IFS used of nested for loop?

Collectives™ on Stack Overflow

Bash shell script Nested while loop with IFS

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related