using awk with arrays

Question

I am using below commands for getting details for each channel. Instead of grep with channel name can we combine below commands into one. So for all channels it will provide total success and failure for each service. I have given expected output below for more clarity.

$2 is channel $11 is the service type, $12 is either S(success) or F(failure), and $14 is elapsed time.

grep 'EXSTAT|BNK' sample.log|/usr/xpg4/bin/awk -F"|" '
          {a[$11]=1}
           $12=="S"{sc[$11]++;st[$11]+=$14}
           $12=="F"{fc[$11]++;ft[$11]+=$14}
           END{for(t in a){
               printf "%s,%u,",t,sc[t]+fc[t];
               if(sc[t])printf "%u,%g,",sc[t],st[t]/sc[t];else printf "0,0,";
               if(fc[t])printf "%u,%g\n",fc[t],ft[t]/fc[t];else print "0,0";
            }}'  
QCD,1,546,1,546,0,0
QASM,1,2460,1,2460,0,0
QBEND,1,247,1,247,0,0

grep 'EXSTAT|SBN' sample.log|/usr/xpg4/bin/awk -F"|" '
          {a[$11]=1}
           $12=="S"{sc[$11]++;st[$11]+=$14}
           $12=="F"{fc[$11]++;ft[$11]+=$14}
           END{for(t in a){
               printf "%s,%u,",t,sc[t]+fc[t];
               if(sc[t])printf "%u,%g,",sc[t],st[t]/sc[t];else printf "0,0,";
               if(fc[t])printf "%u,%g\n",fc[t],ft[t]/fc[t];else print "0,0";
            }}'  
QASM,3,10202,3,3400.67,0,0
QBSD,4,674,4,168.5,0,0

Expected Output

BNK,QCD,1,1,546,0,0
BNK,QASM,1,1,2460,0,0
BNK,QBEND,1,1,247,0,0
MEM,QBATI,10,10,289.4,0,0
SBN,QASM,3,3,3400.67,0,0
SBN,QBSD,4,4,168.5,0,0 

bash-3.2$ grep 'EXSTAT|' sample.log|/usr/xpg4/bin/awk -F"|" '
           {a[$11]=$2}
           $12=="S"{sc[$11]++;st[$11]+=$14}
           $12=="F"{fc[$11]++;ft[$11]+=$14}
           END{for(t in a){
               printf "%s,%s,%u,",a[t],t,sc[t]+fc[t];
               if(sc[t])printf "%u,%g,",sc[t],st[t]/sc[t];else printf "0,0,";
               if(fc[t])printf "%u,%g\n",fc[t],ft[t]/fc[t];else print "0,0";
           }}'
BNK,QCD,1,1,546,0,0
SBN,QBSD,4,4,168.5,0,0
MEM,QBATI,10,10,289.4,0,0
SBN,QASM,4,4,3165.5,0,0
BNK,QBEND,1,1,247,0,0
bash-3.2$

===========Log========

EXSTAT|BNK|2014|11|24|11|07|53|403056|E582783758|QBEND|S|000|247 EXSTAT|BNK|2014|11|24|11|07|59|403057|E582783759|QASM|S|000|2460 EXSTAT|MEM|2014|11|24|11|08|00|403059|24112014110702|QBATI|S|000|310 EXSTAT|MEM|2014|11|24|11|09|00|403063|24112014110802|QBATI|S|000|278 EXSTAT|SBN|2014|11|24|11|09|36|403064|142802|QBSD|S|000|181 EXSTAT|MEM|2014|11|24|11|10|00|403065|24112014110902|QBATI|S|000|316 EXSTAT|MEM|2014|11|24|11|11|00|403066|24112014111002|QBATI|S|000|242 EXSTAT|MEM|2014|11|24|11|12|01|403067|24112014111102|QBATI|S|000|330 EXSTAT|MEM|2014|11|24|11|13|01|403068|24112014111202|QBATI|S|000|273 EXSTAT|SBN|2014|11|24|11|13|35|403069|15785|QASM|S|000|1572 EXSTAT|SBN|2014|11|24|11|13|56|403073|15786|QASM|S|000|3543 EXSTAT|MEM|2014|11|24|11|14|01|403074|24112014111302|QBATI|S|000|249 EXSTAT|SBN|2014|11|24|11|14|16|403075|15787|QASM|S|000|5087 EXSTAT|MEM|2014|11|24|11|15|00|403076|24112014111402|QBATI|S|000|347 EXSTAT|BNK|2014|11|24|11|15|52|403077|E582783764|QCD|S|000|546 EXSTAT|SBN|2014|11|24|11|15|53|403078|142806|QBSD|S|000|160 EXSTAT|MEM|2014|11|24|11|16|00|403079|24112014111502|QBATI|S|000|294 EXSTAT|SBN|2014|11|24|11|16|57|403080|142807|QBSD|S|000|170 EXSTAT|MEM|2014|11|24|11|17|00|403081|24112014111602|QBATI|S|000|255 EXSTAT|SBN|2014|11|24|11|17|42|403082|142808|QBSD|S|000|163

kdhp · Accepted Answer · 2014-11-26 10:52:36Z

2

You can use the combined string $2","$11 as the array index.

awk -F"|" '!/EXSTAT\|/{next}
           {t=$2","$11;a[t]=1}
           $12=="S"{sc[t]++;st[t]+=$14}
           $12=="F"{fc[t]++;ft[t]+=$14}
           END{for(t in a){
               printf "%s,%u,",t,sc[t]+fc[t];
               if(sc[t])printf "%u,%g,",sc[t],st[t]/sc[t];else printf "0,0,";
               if(fc[t])printf "%u,%g\n",fc[t],ft[t]/fc[t];else print "0,0";
           }}' sample.log

!/EXSTAT\|/{next} skips any lines not containing EXSTAT, this replaces grep "EXSTAT|".
{t=$2","$11;a[t]=$2} sets the channel and service type in array a, separated by a comma.
$12=="S"{sc[t]++;st[t]+=$14} $12=="S" checks for success. The remainder increments the success count sc and sums the success time st.
$12=="F"{fc[t]++;ft[t]+=$14} $12=="F" checks for failure. The remainder increments the failure count fc and sums the failure time ft.
printf "%s,%u,",t,sc[t]+fc[t] prints the output fields that are always present.
if(sc[t])printf "%u,%g,",sc[t],st[t]/sc[t];else printf "0,0," prints the success count and average time, guarding from a division by zero error.
if(fc[t])printf "%u,%g\n",fc[t],ft[t]/fc[t];else print "0,0" prints the failure count and average time, guarding from a division by zero error.

edited Nov 26, 2014 at 10:52

answered Nov 18, 2014 at 7:19

kdhp

2,12416 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

14 Comments

Guru Over a year ago

Hi it is working fine.One more query. If I want to check for 2nd parameter EGP along with EXSTAT how to do it. EXSTAT|EGP|2014|09|21|12|02|31|1001|QSWR1236|QSWR|S|000|4385

kdhp Over a year ago

@Guru You con use boolean expressions in patterns. So you can add a second condition to the skipping line. $1!="EXSTAT"||$2!=EGP{next}

Guru Over a year ago

For some directories I am getting below error. /usr/xpg4/bin/awk: line 0 (NR=40945): Record too long (LIMIT: 19999 bytes) /usr/xpg4/bin/awk -F"|" '$1!="EXSTAT"||$2!=EGP{next}\ /\|F\|/{fc[$11]++;ft[$11]+=$14}\ /\|S\|/{sc[$11]++;st[$11]+=$14}\ END{for(t in sc){\ printf "%s,%u,%u,%g,",t,sc[t]+fc[t],sc[t],st[t]/sc[t];\ if(fc[t])printf "%u,%g\n",fc[t],ft[t]/fc[t];\ else print "0,0"}}' $(find $ectpath -mtime 0 -type f). Any solution for this or records are too high.

kdhp Over a year ago

@Guru You could try a different utility (perl, C, maybe replace filter line with a piped grep like the original), or a different version of awk. I had no problems getting nawk or gawk to read and write records longer than that. nawk is the default on *BSD and gawk on most Linux distributions, and OS X.

kdhp Over a year ago

@Guru Perhaps your find is finding some binary files in it's search. If so narrowing it down might help. Another solution, found here, is to use cut to trim lines to a sane size before it gets to awk.

|

Collectives™ on Stack Overflow

using awk with arrays

1 Answer 1

14 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

14 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related