Setting multiple field to awk variables at once

Question

I am trying to set an awk variable field to several field at once.

Right now I can only set the variables one by one.

for line in `cat file.txt`;do
    var1=`echo $line | awk -F, '{print $1}'`
    var2=`echo $line | awk -F, '{print $2}'`
    var3=`echo $line | awk -F, '{print $3}'`

    #Some complex code....
done

I think this is costly cause it parses the linux variable several times. Is there a special syntax to set the variable at once? I know that awk has a BEGIN and END block but the reason I am trying to avoid the BEGIN and END block is to avoid nested awk.

I plan to place another loop and awk code in the #Some complex code.... part.

for line in `cat file.txt`;do
    var1=`echo $line | awk -F, '{print $1}'`
    var2=`echo $line | awk -F, '{print $2}'`
    var3=`echo $line | awk -F, '{print $3}'`

    for line2 in `cat file_old.txt`;do
        vara=`echo $line2 | awk -F, '{print $1}'`
        varb=`echo $line2 | awk -F, '{print $2}'`

        # Do comparison of $var1,var2 and $vara,$varb , then do something with either
    done
done

Why not using awk for the comparison and 'doing something'? — Bernhard
– Bernhard, Commented Oct 5, 2012 at 6:34
Can you give a sample code on using two file compared with awk? note that the two files file.txt and file_old.txt can have different line size. — Nap
– Nap, Commented Oct 5, 2012 at 7:28
Sorry, I missed that you wanted to compare them with eachother. Depends on the specific problem you are trying to solve and the structure of the files, but I have the strong impression that we there are nicer solutions. — Bernhard
– Bernhard, Commented Oct 5, 2012 at 7:38

newfurniturey · Accepted Answer · 2012-10-05 04:50:18Z

4

You can use the IFS internal field separator to use a comma (instead of whitespace) and do the assignments in a while loop:

SAVEIFS=$IFS;
IFS=',';
while read line; do
    set -- $line;
    var1=$1;
    var2=$2;
    var3=$3;
    ...
done < file.txt

IFS=$SAVEIFS;

This will save a copy of your current IFS, change it to a , character, and then iterate over each line in your file. The line set -- $line; will convert each word (separated by a comma) into a numeric-variable ($1, $2, etc.). You can either use these variables directly, or assign them to other (more meaningful) variable names.

Alternatively, you could use IFS with the answer provided by William:

IFS=',';
while read var1 var2 var3; do
    ...
done < file.txt

They are functionally identical and it just comes down to whether or not you want to explicitly set var1=$1 or have it defined in the while-loop's head.

answered Oct 5, 2012 at 4:50

newfurniturey

38.6k10 gold badges99 silver badges104 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Nap Over a year ago

Is there any problem when nesting two while loop? Would the variable $1 $2 $3 have an effect?

newfurniturey Over a year ago

If you call set -- $variable; again inside a nested loop, the $1, $2, .., $n variables that are set will be overridden in the parent-loop as well. So, if you want to use that method, you may want to make sure you assign them to other variables first.

William Pursell · Accepted Answer · 2012-10-05 05:09:38Z

4

Why are you using awk at all?

while IFS=, read var1 var2 var3; do
  ...
done < file.txt

edited Oct 5, 2012 at 5:09

answered Oct 5, 2012 at 4:33

William Pursell

214k49 gold badges279 silver badges317 bronze badges

1 Comment

TaninDirect Over a year ago

If the data is comma-separated, the same approach would work with the input field seperator set to ','. Add IFS=',' before the while loop.

V H · Accepted Answer · 2012-10-05 10:20:01Z

#!/bin/bash    
FILE="/tmp/values.txt"

    function parse_csv() { 
    local lines=$lines;
    > $FILE
    OLDIFS=$IFS;
    IFS=","
    i=0
    for val in ${lines}
    do
      i=$((++i))
      eval var${i}="${val}"
    done
    IFS=$OLDIFS;
    for ((j=1;j<=i;++j))
    do
      name="var${j}"
      echo ${!name} >> $FILE
    done

    }

    for lines in `cat file_old.txt`;do
     parse_csv;
    done

The problem you have described has only got 3 values, would there be a chance that 3 values may differ and be 4 or 5 or undefined ?

if so the above will parse through the csv line by line and output each value at a time on a new line in a file called /tmp/values.txt

feel free to modify to match your requirements its far more dynamic than defining 3 values

Collectives™ on Stack Overflow

Setting multiple field to awk variables at once

3 Answers 3

2 Comments

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related