23

I work on a cluster shared with other colleagues. The hard disk is limited (and has been full on some occasions), so I clean up my part occasionally. I want to do this quickly, so until now I do this by making a list of files larger than 100 MB older than 3 months, and I see if I still need them.

But now I am thinking that there could be a folder with >1000 smaller files that I miss, so I want to get an easy way to see if this is the case. From the way I generate data, it would help to get a list of total size per extension. In the context of this question, 'extension' as everything behind the last dot in the filename.

Suppose I have multiple folders with multiple files:

folder1/file1.bmp   40 kiB
folder1/file2.jpg   20 kiB
folder2/file3.bmp   30 kiB
folder2/file4.jpg    8 kiB

Is it possible to make a list of total filesize per file extension, so like this:

bmp 70 kiB
jpg 28 kiB

I don't care about files without extension, so they can be ignored or put in one category.

I already went through man pages of ls, du and find, but I don't know what is the right tool for this job...

Improve this question
2
  • This question wouldn't be amiss on codegolf.stackexchange.com :) Commented Mar 13, 2019 at 16:52
  • @DougMcLean: you are welcome to post it there. ;) Commented Mar 31, 2019 at 12:38

5 Answers 5

Reset to default
31

On a GNU system:

LC_ALL=C find . -name '?*.*' -type f -printf '%b.%f\0' |
  LC_ALL=C gawk -F . -v RS='\0' '
    {s[$NF] += $1; n[$NF]++}
    END {
      PROCINFO["sorted_in"] = "@val_num_asc"
      for (e in s) printf "%15d %4d %s\n", s[e]*512, n[e], e
    }'

Or the same with perl, avoiding the -printf extension of GNU find (still using a GNU extension, -print0, but this one is more widely supported nowadays):

LC_ALL=C find . -name '?*.*' -type f -print0 |
  perl -0ne '
    if (@s = lstat$_){
      ($ext = $_) =~ s/.*\.//s;
      $s{$ext} += $s[12];
      $n{$ext}++;
    }
    END {
      for (sort{$s{$a} <=> $s{$b}} keys %s) {
        printf "%15d %4d %s\n",  $s{$_}<<9, $n{$_}, $_;
      }
    }'

It gives an output like:

          12288    1 pnm
          16384    4 gif
         204800    2 ico
        1040384   17 jpg
        2752512   83 png

If you want KiB, MiB... suffixes, pipe to numfmt --to=iec-i --suffix=B.

%b*512 gives the disk usage¹, but note that if files are hard linked several times, they will be counted several times so you may see a discrepancy with what du reports.

¹ As an exception, on HP/UX, the block size reported by lstat()/stat() is 1024 instead of 512. GNU find adjusts for that so it's %b still represents the number of 512 byte units, but with perl, you'd need to multiply by 1024 instead there.

Improve this answer
4
  • Fails on MacOS (find: -printf: unknown primary or operator) Commented Apr 11, 2019 at 15:45
  • 1
    @MichaelCodes, yes -printf is specific to GNU find, which is why I said on a GNU system. Commented Apr 11, 2019 at 15:50
  • @MichaelCodes, see edit with a perl alternative that should work even on macOS. Commented Apr 11, 2019 at 16:12
  • What is 1,4,2,17? The ammount of files for each type? Commented Jan 24, 2020 at 22:19
10

Here is another solution:

find . -type f |  egrep -o "\.[a-zA-Z0-9]+$" | sort -u | LC_ALL=C xargs -I '%' find . -type f -name "*%" -exec du -ch {} + -exec echo % \; | egrep "^\.[a-zA-Z0-9]+$|total$" | uniq | paste - -

The part that gets the extensions is:

find . -type f |  egrep -o "\.[a-zA-Z0-9]+$" | sort -u

Next search for the files with an extension and print it on the screen as well:

LC_ALL=C xargs -I '%' find . -type f -name "*%" -exec du -ch {} + -exec echo % \;

Next we want to keep the extension and the total:

egrep "^\.[a-zA-Z0-9]+$|total$" | uniq

and keep it on the same line:

paste - -
Improve this answer
2
  • Works on MacOS. Commented Apr 11, 2019 at 15:46
  • If you get multiple numbers for the same extension, add them together. Commented May 5, 2024 at 22:08
9

Not as nice as Stephane's solution, but you could try

find . -type f -name "*.png" -print0 | xargs -0r du -ch | tail -n1

where you have to run this for each type of files.

Improve this answer
3
  • 2
    That assumes there are sufficiently few png files that only one du invocation is run. With GNU xargs, you'd want to add the -r flag so du is not run when there's no file (otherwise, you'd end up with the disk usage of the current directory). You may want to add a -type f or ! type d to avoid counting the files that are in directories whose name ends in .png. Commented Sep 9, 2016 at 11:02
  • this only looks for one specific extension. Commented Sep 9, 2016 at 11:14
  • That's what I wrote. One had to wrap it in a script that iterates over all applicable extensions in order to get a "complete" solution. Commented Sep 9, 2016 at 11:17
3

Since I do not yet have enough reputation points to write comments, I'll extend Stéphane Chazelas' answer here. To include in the list files with no extension, for example executables, one can use this command line:

find . -name '*' -type f -printf '%b.%f\0' | awk -F . -v RS='\0' '{if (NF==2) $(NF+1)=" "; s[$NF] += $1; n[$NF]++} END {for (e in s) printf "%15d %6d  %s\n", s[e]*512, n[e], e}' | sort -rn | numfmt --to=iec-i --suffix=B
Improve this answer
1
  • That doesn't work properly if there are extension-less files below directories with dots in there names. Commented Jun 26, 2024 at 18:03
0 
find . -type f -printf "%p %k\n" | sed 's|.*\.||' | awk '{
    arr[$1]+=$2
   }
   END {
     for (key in arr) printf("%s\t%s\n", key, arr[key])
   }' | sort -k1,1

Based on this and this

Improve this answer
1
  • That won't work properly for file paths that contain newline characters, or extensions that contain whitespace, or files with no extension, or file paths that can't be decoded as text in the locale. Commented Sep 26 at 15:30

You must log in to answer this question.