2

The question is really simple, I have read all the questions and still cannot make it! I have a plain file like this

$cat file1.txt
ALA
AJD
KSF

And I want awk to use each of the values as regex to print lines from another file to another one: 

$cat file2.txt
name,st,ed,le
ALA,10,12,12
ALA,2,5,4
ALA,6,5,8
AJD,5,8,7
KSF,5,8,7

So my script is

while read p;
awk -F"," 'NR==1{print $0}' file2.txt > $p.csv
awk -F"," '/$p/{print $0}' file2.txt >> $p.csv
done <file1.txt

And the desired output would be:

$cat ALA.csv
name,st,ed,le
ALA,10,12,12
ALA,2,5,4
ALA,6,5,8
$cat AJD.csv
name,st,ed,le
AJD,5,8,7
$cat KSF.csv
name,st,ed,le
KSF,5,8,7

Unfortunately, I only get the headers printed in each file. I have manually put each value from file1.txt replacing $p and it works perfectly. So I think the problem is that the variable $p is not been well interpreted. I tried with quotes, double simple. I tried also many different suggestions I found, but nothing seems to work!

Improve this question

1 Answer 1

Reset to default
4

While you could do:

awk "/$p/" file2.txt > "$p.csv"

That is have the shell expand the content of the $p shell variable in the code passed to awk, that's bad practice and basically amounts to a command injection vulnerability (for instance for a value of $p like ^/{system("reboot");/). Best is to pass the shell variable as is to awk and use awk's ~ operator for regexp matching. The best way is via an environment variable and awk's ENVIRON special array:

export P
while IFS= read -r P; do
  awk 'NR == 1 || $0 ~ ENVIRON["P"]' < file2.txt > "$P.csv"
done < file1.txt

But here, you could avoid the shell loop and do only one pass in the files:

awk 'NR == FNR {files[$0]; next}
     FNR == 1 {for (f in files) print > f ".csv"; next}
     {
       for (f in files)
         if ($0 ~ f) print > f ".csv"
     }' file1.txt file2.txt
Improve this answer
6
  • Is there really a difference if the variable is passed through the environment or via -vP="$P"? Can the second one choke on some weird special character? Commented Oct 28, 2016 at 11:09
  • @ikkachu, yes -vP="$P" does some backslash sequence expansion (typically a problem with regexps that often contain backslashes). See unix.stackexchange.com/search?q=user%3A22565+ENVIRON+ARGV+awk Commented Oct 28, 2016 at 11:23
  • By the way -vP="$P" doesn't work in many traditional implementations of awk where you need -v P="$P".(-v and P="$P" being separate arguments). Commented Oct 28, 2016 at 11:26
  • Much memory save if you iterate through fields not through array {for(i=1;i<=NF;i++)if($i in files)print > $i ".csv" (do not forget FS=OFS=",") Commented Oct 28, 2016 at 11:41
  • @Costas, that would be functionally different. I'm not sure how that would save memory though. And you could write the same line several times into a given file. Commented Oct 28, 2016 at 11:44

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.