shell script: nested while loop: not pickup the outer loop counter

Question

I have a file that has millions of records. I will just make it simple with two sample records here. My goal is to add delimiters, such as a comma, to make the file has the same number of delimiters, so I can upload the file to a database.

The issue is that the nested while loop simply added a fixed number of delimiters at the end of line. My goal is to add delimiters dynamically according to the number of delimiters the file already has. I changed the inner while loop to a if statement block, the same behavior happened as well. So I do not think a nested while loop is necessary.

Here is my sample text file:

1st,1,
2nd,2

Here is the script. First user input position is the text file, the second position is the number of delimiters I want.

#!/bin/bash

f="$1"
delim="$2"

while read line
do
cnt=`echo $line | tr -cd ',' | wc -c`
    while   [[ $cnt  -lt $delim ]];   
    do
            sed -i 's/$/,/' $f 
        cnt=`expr $cnt + 1`
    done
done  < $f

Here is my trace using bash -x:

bash -x csv.sh split_address_2.csv 3
+ f=split_address_2.csv
+ delim=3
+ read line
++ wc -c
++ tr -cd ,
++ echo 1st,1,
+ cnt=2
+ [[ 2 -lt 3 ]]
+ sed -i 's/$/,/' split_address_2.csv
++ expr 2 + 1
+ cnt=3
+ [[ 3 -lt 3 ]]
+ read line
++ wc -c
++ tr -cd ,
++ echo 2nd,2
+ cnt=1
+ [[ 1 -lt 3 ]]
+ sed -i 's/$/,/' split_address_2.csv
++ expr 1 + 1
+ cnt=2
+ [[ 2 -lt 3 ]]
+ sed -i 's/$/,/' split_address_2.csv
++ expr 2 + 1
+ cnt=3
+ [[ 3 -lt 3 ]]
+ read line

Here is the output of the text file. You can see the script simply added 3 commas at the end of each line.

1st,1,,,,
2nd,2,,,

Thanks so much for your kind response. Have a good day!

Lifu Tang · Accepted Answer · 2011-10-13 00:22:00Z

2

Why did you use sed -i here? By that command, you are doing a search and replace on your whole file there, on each line...

In your first line in your input file, there are already 2 commas, so your while-loop runs once, appending each line in your input file with an extra comma. On the second line, there's only one comma so your while-loop runs twice, appending each line in your your whole text with 2 extra comma. That's why there are 3 extra commas after each line when you have run your script.

I did some simple minor adjustment to your script. It works as your expected this time:

#!/bin/bash

f="$1"
delim="$2"

while read line
do
cnt=`echo $line | tr -cd ',' | wc -c`
    while   [[ $cnt  -lt $delim ]];   
    do
        line=`echo $line | sed 's/$/,/'`
        cnt=`expr $cnt + 1`
    done
    echo $line
done  < $f

answered Oct 13, 2011 at 0:22

Lifu Tang

1,3099 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

dave Over a year ago

very nice improvement. It worked like a charm! The change from sed -i to your line=echo $line | sed 's/$/,/' would probably increase speed a lot. I will have to measure the speed differences for fun. Thanks Mr. Fox!

Lifu Tang Over a year ago

Well, if you really care about the "runtime speed", you may try this: line="line$,". Using sed is quite slow. I stick to sed just because you used that before, and I wanted to show that sed also works.

Collectives™ on Stack Overflow

shell script: nested while loop: not pickup the outer loop counter

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related