Awk error inside bash shell script

Question

I am trying to sort a two column file based on order of another file. With the following command :

  awk  'FNR == NR {lineno[$1] = NR; next} {print lineno[$1], $0;}'  associativity4Way_cacheSize32KB_replacementPolicy1.outs  associativity4Way_cacheSize32KB_replacementPolicy2.outs   | sort -k  1,1n  | cut -d " " -f2- > sorted

When I use AWK command outside shell it is working perfectly. Below is the head content of my files and the output:

this is head of "associativity4Way_cacheSize32KB_replacementPolicy2.outs "

LONG_MOBILE-10.bt9.trace.gz_a4_b64_c32  4.516192
LONG_MOBILE-10.bt9.trace.gz_a4_b64_c32  4.147467
LONG_MOBILE-10.bt9.trace.gz_a4_b64_c64  2.040121
LONG_MOBILE-10.bt9.trace.gz_a4_b64_c64  1.837639
LONG_MOBILE-10.bt9.trace.gz_a4_b64_c64  3.068701
LONG_MOBILE-11.bt9.trace.gz_a4_b64_c32  0.474358
LONG_MOBILE-11.bt9.trace.gz_a4_b64_c32  0.545525
LONG_MOBILE-11.bt9.trace.gz_a4_b64_c32  0.469907
LONG_MOBILE-11.bt9.trace.gz_a4_b64_c64  0.19461
...

And this is "ssociativity4Way_cacheSize32KB_replacementPolicy1.outs"

SHORT_SERVER-2.bt9.trace.gz_a4_b64_c32  29.869599
SHORT_SERVER-2.bt9.trace.gz_a4_b64_c32  29.100611
SHORT_SERVER-2.bt9.trace.gz_a4_b64_c32  29.068284
SHORT_SERVER-2.bt9.trace.gz_a4_b64_c32  28.559002
SHORT_MOBILE-20.bt9.trace.gz_a4_b64_c32 21.332859
SHORT_MOBILE-22.bt9.trace.gz_a4_b64_c32 20.605805
SHORT_MOBILE-20.bt9.trace.gz_a4_b64_c32 20.256246
SHORT_MOBILE-20.bt9.trace.gz_a4_b64_c32 20.193713
SHORT_MOBILE-22.bt9.trace.gz_a4_b64_c32 20.119883
SHORT_MOBILE-22.bt9.trace.gz_a4_b64_c32 20.099358
...

output sorted perfectly as I wish:

SHORT_SERVER-2.bt9.trace.gz_a4_b64_c32  28.559002
SHORT_SERVER-2.bt9.trace.gz_a4_b64_c32  29.068284
SHORT_SERVER-2.bt9.trace.gz_a4_b64_c32  29.100611
SHORT_SERVER-2.bt9.trace.gz_a4_b64_c32  29.869599
SHORT_MOBILE-20.bt9.trace.gz_a4_b64_c32 20.193713
SHORT_MOBILE-20.bt9.trace.gz_a4_b64_c32 20.256246
SHORT_MOBILE-20.bt9.trace.gz_a4_b64_c32 21.332859
...

The problem appears when I try to place the awk command inside my bash script to be able do it for other files:

set lru = "associativity${a}Way_cacheSize${c}KB_replacementPolicy1.outs"


set policy = "associativity${a}Way_cacheSize${c}KB_replacementPolicy${r}.outs"


awk  "FNR == NR {lineno["$policy"] = NR; next} {print lineno["$policy"], "$lru";}" "$lru" "$policy" | sort -k 1,1n  | cut -d " " -f2- > "${policy}_sorted"

This is the error I get:

awk: cmd. line:1: FNR == NR {lineno[associativity4Way_cacheSize32KB_replacementPolicy3.outs] = NR; next} {print lineno[associativity4Way_cacheSize32KB_replacementPolicy3.outs], associativity4Way_cacheSize32KB_replacementPolicy1.outs;}
awk: cmd. line:1:                                                                                                                                                                                                                  ^ syntax error

I also tried this ( suggested by one answer here) but the output is not correct. It is printing the name of 'lru' file multiple times in the output file.

set lru = "associativity${a}Way_cacheSize${c}KB_replacementPolicy1.outs"
            set policy = "associativity${a}Way_cacheSize${c}KB_replacementPolicy${r}.outs"
            awk -v policy="$policy" -v lru="$lru" 'FNR == NR {lineno[policy] = NR; next} {print lineno[policy], lru}' "$lru" "$policy"  | sort -k 1,1n  | cut -d ' ' -f2- > "${policy}_sorted"


 associativity8Way_cacheSize64KB_replacementPolicy1.outs
associativity8Way_cacheSize64KB_replacementPolicy1.outs
associativity8Way_cacheSize64KB_replacementPolicy1.outs
associativity8Way_cacheSize64KB_replacementPolicy1.outs
associativity8Way_cacheSize64KB_replacementPolicy1.outs
associativity8Way_cacheSize64KB_replacementPolicy1.outs
associativity8Way_cacheSize64KB_replacementPolicy1.outs
associativity8Way_cacheSize64KB_replacementPolicy1.outs
associativity8Way_cacheSize64KB_replacementPolicy1.outs
associativity8Way_cacheSize64KB_replacementPolicy1.outs

Can someone advise where I am wrong here?

edit your question using the editors {} button to format your code properly so we know what your code really looks like. White space matters in awk scripts. — Ed Morton
– Ed Morton, Commented Mar 5, 2017 at 18:57
@James: yes I tried the complete path as well. Still the same issue. — samira
– samira, Commented Mar 8, 2017 at 20:14

Ed Morton · Accepted Answer · 2017-03-05 19:00:24Z

2

Never enclose any script (awk sed, whatever) in double quotes as it just introduces complexity and fragility.

Change this:

awk "FNR == NR {lineno["$policy"] = NR; next} {print lineno["$policy"], "$lru";}" "$lru" "$policy"

to this:

awk -v policy="$policy" -v lru="$lru" 'FNR == NR {lineno[policy] = NR; next} {print lineno[policy], lru}' "$lru" "$policy"

and see how it goes. Get the book Effective Awk Programming, 4th Edition, by Arnold Robbins to start to learn about awk.

answered Mar 5, 2017 at 19:00

Ed Morton

209k18 gold badges90 silver badges212 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

samira Over a year ago

Thanks, The error resolved with this. But the output is so weird. Now it is printing the name of lru file in output file instead of my data. ` set lru = "associativity${a}Way_cacheSize${c}KB_replacementPolicy1.outs" set policy = "associativity${a}Way_cacheSize${c}KB_replacementPolicy${r}.outs" awk -v policy="$policy" -v lru="$lru" 'FNR == NR {lineno[policy] = NR; next} {print lineno[policy], lru}' "$lru" "$policy" | sort -k 1,1n | cut -d ' ' -f2- > "${policy}_sorted"`

samira Over a year ago

I don't think it answers my question because I am able to do what I want to do with awk outside of bash script but it doesn't do the same thing inside the bash script. This answer made the awk script to do other function.

Ed Morton Over a year ago

Your problem is unrelated to the awk script being inside vs outside of a bash script, you are running a different awk script inside vs outside of the bash script and the one you are running inside the bash script is syntactically incorrect and can be corrected by using the syntax in my answer. If you ran that same awk script outside of your bash script you would have exactly the same problem you're having running it inside of it.

samira Over a year ago

What is incorrect about the script I use outside of the bash?

Ed Morton Over a year ago

I said the script you were using inside the bash script was incorrect and I already told you in what way. Please re-read my answer as it is THE answer to question you originally asked. To be clear, I'm in no way suggesting that my answer will produce the output you eventually want to get, just that it answers your question of why you get the syntax error you posted and how to solve that problem. As I also said earlier if you want help debugging your script due to output not being what you wanted, then post a new question with it's own concise, testable sample input and expected output.

Collectives™ on Stack Overflow

Awk error inside bash shell script

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related