Powershell Script to find duplicate files

Question

I found a PowerShell script on TechNet to help locate duplicate files in folders. However, when I run it, I am getting an error on what appears to be every folder\file. Not sure what switch is supposed to be used in this.

$Path = '\\servername\Share\Folders' #define path to folders to find duplicate files
$Files=gci -File -Recurse -path $Path | Select-Object -property FullName,Length
$Count=1
$TotalFiles=$Files.Count
$MatchedSourceFiles=@()
ForEach ($SourceFile in $Files)
{
  Write-Progress -Activity "Processing Files" -status "Processing File $Count / $TotalFiles" -PercentComplete ($Count / $TotalFiles * 100)
  $MatchingFiles=@()
  $MatchingFiles=$Files |Where-Object {$_.Length -eq $SourceFile.Length}
  Foreach ($TargetFile in $MatchingFiles)
  {
    if (($SourceFile.FullName -ne $TargetFile.FullName) -and !(($MatchedSourceFiles |
      Select-Object -ExpandProperty File) -contains $TargetFile.FullName))
    {
      Write-Verbose "Matching $($SourceFile.FullName) and $($TargetFile.FullName)"
      Write-Verbose "File sizes match."
      if ((fc.exe /A $SourceFile.FullName $TargetFile.FullName) -contains "FC: no differences encountered")
      {
        Write-Verbose "Match found."
        $MatchingFiles+=$TargetFile.FullName
      }
    }
  }
  if ($MatchingFiles.Count -gt 0)
  {
    $NewObject=[pscustomobject][ordered]@{
      File=$SourceFile.FullName
      MatchingFiles=$MatchingFiles
    }
    $MatchedSourceFiles+=$NewObject
  }
  $Count+=1
}
$MatchedSourceFiles

Errors

FC: Insufficient number of file specifications fc.exe : FC: Invalid Switch At line:18 char:12

gci : Could not find a part of the path At line:2 char:8

fc.exe : FC: Invalid Switch At line:18 char:12

The Get-ChildItem error gci : Could not find a part of the path At line:2 char:8 indicates part of the path doesn't exist. The script won't work until you correct that. — Rich Moss
– Rich Moss, Commented Nov 27, 2019 at 18:34
Did you perhaps forget to also put the share name in there (\\server\share\Folder\Folders) — Theo
– Theo, Commented Nov 27, 2019 at 18:51

alain · Accepted Answer · 2022-05-30 18:08:54Z

1

To fix your fc.exe error and optimize tour script, I also recommend @rich-moss 's solution.

But if you only want to find duplicates, you can easily accomplish so by checking their hashes.

Example:

$Duplicates = Get-ChildItem -File -Recurse | Get-FileHash | Group-Object -Property Hash | Where-Object Count -gt 1

If ($duplicates.count -lt 1) {
  $null # 'No duplicates found. Do stuff ...'
} else {
$result = foreach ($d in $duplicates) {
  $d.Group | Select-Object -Property Path, Hash
}

edited May 30, 2022 at 18:08

answered May 26, 2022 at 11:14

alain

495 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Rich Moss · Accepted Answer · 2019-11-27 21:27:48Z

The script you provided is very inefficient and provides false positives in my tests. It's inefficient because it compares every file twice (Source->Target and Target->Source) and because it iterates through all files regardless of size. Here's a quicker version that gathers the files into groups of similarly sized files and only executes FC.EXE once per pair of files:

$Path = 'C:\Temp'     
$SameSizeFiles  = gci -Path $Path -File -Recurse | Select FullName, Length | Group-Object Length | ? {$_.Count -gt 1} #the list of files with same size
$MatchingFiles=@()
$GroupNdx=1
Foreach($SizeGroup in ($SameSizeFiles | Select Group)){
    For($FromNdx = 0; $FromNdx -lt $SizeGroup.Group.Count - 1; $FromNdx++){
        For($ToNdx = $FromNdx + 1; $ToNdx -lt $SizeGroup.Group.Count; $ToNdx++){
            If( (fc.exe /A $SizeGroup.Group[$FromNdx].FullName $SizeGroup.Group[$ToNdx].FullName) -contains "FC: no differences encountered"){
                $MatchingFiles += [pscustomobject]@{File=$SizeGroup.Group[$FromNdx].FullName; Match = $SizeGroup.Group[$ToNdx].FullName }
            }
        }
    }
    Write-Progress -Activity "Finding Duplicates" -status  "Processing group $GroupNdx of $($SameSizeFiles.Count)" -PercentComplete ($GroupNdx / $SameSizeFiles.Count * 100)
    $GroupNdx += 1
}
$MatchingFiles

Efficiency will be even more important if you're running it over the network. You may find it quicker to execute the script on the server itself, rather than from a share. There is some discussion here about the fastest way to compare files in .Net.

Collectives™ on Stack Overflow

Powershell Script to find duplicate files

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related