1

I have a folder with some files (snippet of the contents of the folder)

PAT1.URGRSVP.50.WR786842JOB11632.WRS20140.FILE0005.DAT
PAT1.URGRSVP.50.WR786842JOB11643.WRS20140.FILE0003.DAT
PAT1.URGRSVP.51.WR786842JOB11643.WRS29232.FILE0003.DAT
PAT1.URGRSVP.50.WR786842JOB11694.WRS20140.FILE0002.DAT
...
...
...

My focus is on the 3rd (50,50,51,50) and the 5th (WRS20140,WRS20140,WRS29232,WRS20140) blocks. How can I write a script that displays the duplicate filenames with the same 3rd block AND 5th block (The duplicates of the combination of the 3rd and the 5th block strings)?

So the output should list the following in the above example

PAT1.URGRSVP.50.WR786842JOB11643.WRS20140.FILE0003.DAT
PAT1.URGRSVP.50.WR786842JOB11694.WRS20140.FILE0002.DAT
Improve this question

2 Answers 2

Reset to default
3 
ls *.DAT | awk -F. '{ if (c[$3$5]) print $0 ; c[$3$5]=$0}'

In the above, awk looks at each file name using . as a field separator. If it has seen the combination of the third and fifth fields before, it prints the file name. With your file names as input, the above produces:

PAT1.URGRSVP.50.WR786842JOB11643.WRS20140.FILE0003.DAT
PAT1.URGRSVP.50.WR786842JOB11694.WRS20140.FILE0002.DAT

MORE: Let's examine the awk commands in more detail:

if (c[$3$5]) print $0 ; c[$3$5]=$0

The above consists of two statements: one "if" statement and one assignment. The "if" statement is:

if (c[$3$5]) print $0

In this statement, c is an "associative array". This means that that you give it a key and it gives you back a value. We are using $3$5 as the key where $3 is the third "block" (what awk would call the third "field") and $5 is the fifth block. If that key was previously unassigned, then c[$3$5] returns an empty (false) value. So, if this combination of third and fifth blocks was seen before, then print $0 is executed, meaning that the whole of the file name is printed. If not, the print statement is skipped.

The second statement is:

c[$3$5]=$0

This assigns the name of the file ($0) to the associative array under the key of the third and fifth fields: $3$5. Thus, the next time that those fields are seen in the "if" statement, the print statement will execute.

Improve this answer
4
  • Thanks. Can you please explain the awk part of the script? Between the {} Commented Feb 3, 2014 at 6:37
  • 1
    What should I do when two nice folks give the right answer at the same time? Commented Feb 3, 2014 at 6:47
  • @user1324816 Be nice to both and recognize that I was first by two minutes (grin). Commented Feb 3, 2014 at 6:49
  • @user1324816 Does the new explanation answer your questions? Commented Feb 3, 2014 at 6:50
1

Here's how you could do it with awk: use a variable to count the number of times you've seen the same pair of 3rd and 5th field, and print out the filename if you've already seen that particular pair.

With those filenames in a file called input, this would look like:

$ awk -F. '{if (dups[$3$5]++) print $0}' input

If your filenames could contain whitespace or other funky characters, use find rather than ls to list them, with something like:

$ find . -name 'PAT1.*.DAT' -print0 | \
    awk -F. 'BEGIN{RS="\0"} {if (dups[$3$5]++) print $0}'

As a side benefit, you could inspect the dups variable in an END block to print out how many of each pair you saw in the input.

Improve this answer
5
  • What should I do when two nice folks give the right answer at the same time? Commented Feb 3, 2014 at 6:48
  • 1
    Wait for a third person to provide something better? :-) John actually answered first, and provided more details about how this works, so his answer is arguably better. Commented Feb 3, 2014 at 6:53
  • 1
    On the other hand, your answer handles hostile file names, and I like the ++ trick. Commented Feb 3, 2014 at 6:57
  • 1
    Can be shortened to: printf '%s\0' PAT1.*.DAT | awk -F. -vRS='\0' 'dups[$3$5]++'. Note that not all awk implementations support NUL characters in their input Commented Feb 3, 2014 at 11:57
  • Also note that while the ls output or the shell globs are lexicographically sorted, the find output is not, so which of the duplicates will be output will not necessarily be the same. Commented Feb 3, 2014 at 12:00

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.