Just as a follow-up, following is the code for what I did. Mjolinor's suggestion of using a CSV file simplified (relatively speaking) the solution immensely (Thank YOU). I was definitely trying to make the solution too complex.
I created three CSV files that I used to drive the process. First is the RotateBackups_Masterlist.txt file. BackupName is the name of the backup as it's associated later on in the process. VersionsRetained is the number to keep on hand. BackupType is whether it's a file or folder backup.
RowNbr,BackupName,VersionsRetained,BackupType
1,TargetBackup,2,Folder
2,LstDefBackup,5,File
3,XMLBackup,3,File
4,SourceBackup,2,Folder
5,TXTBackup,8,File
Second file is RotateBackups_FolderList.txt. BackupName corresponds to the same field in the above file, Foldername is the folder that's being backed up.
RowNbr,BackupName,FolderName
1,TargetBackup,c:\MyBooks\target
2,SourceBackup,c:\MyBooks\source
Third file is RotateBackups_FileExtensions.txt. BackupName corresponds to the same field in the first file, FileExtension is the wildcarded name of the files to be backed up (it will accept patterns also), and FolderLoc is the folder where these items are to be found.
RowNbr,BackupName,FileExtension,FolderLoc
1,LstDefBackup,*.def,c:\MyBooks\target
2,LstDefBackup,*.lst,c:\MyBooks\target
3,XMLBackup,*.xml,c:\MyBooks\target
4,TXTBackup,*.txt,c:\MyBooks\rfiles
Please note that the same file extension can be backed up to multiple BackupNames. Also, Multiple Folders can be assigned to the same BackupName and FileExtension.
What it does:
- It creates a folder with zip files in it that are named after the
BackupName field (master file) plus a datetime stamp (all within the
same batch) in the filename.
- The filenames all have the same BATCH date-time stamp so you will
know that all the backups with ending with 20120401_091500.zip were
all created out of the same run.
- It gives you the ability to version backups, and gives you a
point-in-time to restore to.
- It gives you some flexibility in managing your backup space
requirements. It gives you the ability to maintain a longer retention
for critical backups or a shorter retention for less critical ones.
- Deleted items can be destructively deleted or recycled.
- Your backups do not age out based on the creation date. The only
thing that causes a backup to be marked for deletion is the creation
of a newer one that puts the count over the number you want retained.
What it doesn't do:
- I don't have a lot of error handling in it, other than checking for
empty arrays or the LASTEXITCODE immediately prior to deleting files.
- It's not meant for huge numbers of the same type of files to be
backed up using the File Extension, as the backup is a one-at-a-time
function call to create/update a zip file.
Lastly, the code that does the work (posted in entirety after comments). I grabbed pieces of code for numerous parts of this from many sources, and thank the original poster/developer of each of them. Use of this utility is at your own risk. I welcome any improvements others might wish to add to this.
This is the function that creates the backup.
function create-7zip([String] $aDirectory, [String] $aZipfile){
[string]$pathToZipExe = "C:\Program Files\7-zip\7z.exe";
[Array]$arguments = "a", "-tzip", "$aZipfile", "$aDirectory", "-r";
& $pathToZipExe $arguments;
}
# Call it by using:
#create-7zip "c:\temp\myFolder" "c:\temp\myFolder.zip"
Variables & Arrays are initialized here, Text files from above are imported here. The variable $KillOrRecycle is set to Recycle, so anything deleted goes to recycle bin. The Kill is a destructive, non-recoverable file deletion.
#************************************************************************************
#************************************************************************************
# Initialize variables
$zipFolder = "C:\ZipFiles"
$nameConv = "{0:yyyyMMdd_HHmmss}" -f (Get-Date) + ".zip"
$fileList = @{}
$FileCountArray = @()
$bkupTypeArr = @()
$myDocFolder = "c:\Documents and Settings\MyPC\My Documents\"
# Import text files for master, folder and file backup information
$bkupRotateMasterArr = Import-Csv $myDocFolder"RotateBackups_MasterList.txt"
$fldrBkupArray = Import-Csv $myDocFolder"RotateBackups_FolderList.txt"
$fileExtBkupArr = Import-Csv $myDocFolder"RotateBackups_FileExtensions.txt"
# Switch to delete Item or to send to recycle bin
# delete is destructive and cannot recover file.
# Recycle setting removes file from folder, but sends to recycle bin
# and can be restored if needed.
# Must be either "Kill" or "Recycle"
$KillOrRecycle = "Recycle"
#************************************************************************************
#************************************************************************************
Load the contents of the imported CSV files into hashtables.
$bkup_Counts is a hashtable of the BackupName & Number of files to be retained.
$bkupTypeArr is an array of the BackupNames (used later on)
$fArray is a hashtable of the BackupName and the filename generated that's associated with it.
$nameConv is done at the top to give all files in this run the same date-time stamp.
# Load contents of master backup array
$bkup_Counts = @{}
$b = $null
foreach($b in $bkupRotateMasterArr)
{
$bkup_Counts[$b.BackupName] = $b.VersionsRetained
}
#set Backup Types from the array we just defined
$type = $null
foreach ($type in $bkup_Counts.Keys) {
$bkupTypeArr += $type
}
#create array of our filenames for this batch
$type = $null
$fArray = @{}
foreach ($type in $bkupRotateMasterArr) {
$fArray[$type.BackupName] = ($type.BackupName + $nameConv)
}
Next, if we have any data in the fileExtension array, run through it using the folder Location & file extension (recursively) to grab everything matching it from that path, associating it with the appropriate BackupName. Add that to the $filelist hashtable that has the fullname property as the key, and the BackupName as the value it's associated with.
# if extension array not null, get list of files to back up
if ($fileExtBkupArr) {
# Gather the list of files to be backed up
$f = $null
foreach ($f in $fileExtBkupArr) {
$arr = @()
$arr = (Get-ChildItem $f.FolderLoc -Recurse -Include $f.FileExtension | Select-Object fullname)
foreach ($a in $arr) {
if ($a) {
$fileList[$a] = $f.BackupName
} # if $a not null
} # end inner foreach
} # end outer foreach
} # if FileExtension Backup Array not null
Then, if there's anything on the filelist hashtable, zip it to the appropriately named backup.
# if filelist count gt zero, then create zip file of them for appropriate backup
if ($fileList.Count -gt 0) { # must have entries in hashtable
$f = $null
#Loop thru file list & associate file with the appropriate backup
foreach ($f in $fileList.Keys) {
$arcFile = $null
if ($fileList.ContainsKey($f)) {
if ($fArray.ContainsKey($fileList[$f])) {
$arcFile = $fArray[$fileList[$f]]
create-7zip $f.FullName $zipFolder\$arcFile
} #if key in fArray
} # if key in Filelist
} # end foreach
} # if hastable not empty
Next, if we have folders to back up, then use a similar process to back those folders up using a similar method.
# if folder backup not null then back up folders
if ($fldrBkupArray) { # check if array not null (no entries)
$f = $null
#Backup Folders now
foreach ($f in $fldrBkupArray) {
$arcFldr = $null
#if ($fArray.ContainsKey($f[1])) {
if ($fArray.ContainsKey($f.BackupName)) {
$arcFldr = $fArray[$f.BackupName]
create-7zip $f.FolderName $zipFolder\$arcFldr
} #end if
} # end foreach
} # end if $fldrBkupArray not null
Next item is to make sure that we had no failure prior to deleting anything. We delete based on whether the archive bit is set, and send the deleted file to the recycle bin (default) or the bit bucket.
# if 7zip succeeded, we'll continue
if ($LASTEXITCODE -gt 0)
{Throw "7Zip failed" }
ELSE { # if Exitcode = 0 then continue with job
# Remove any files with Archive bit = False
# we marked it for deletion in previous run
Add-Type -AssemblyName Microsoft.VisualBasic
$files=get-childitem -path $zipFolder
# we'll delete all files that don't have the archive bit set
Foreach($file in $files) {
If((Get-ItemProperty -Path $file.fullname).attributes -band [io.fileattributes]::archive)
{ Write-output "$file is set to be retained" }
ELSE {
if ($KillOrRecycle = "Recycle") {
Write-output "$file does not have the archive bit set. Deleting (Sent to recycle bin)."
[Microsoft.VisualBasic.FileIO.Filesystem]::DeleteFile($file.fullname,'OnlyErrorDialogs','SendToRecycleBin')
$output = $_.ErrorDetails
}
ELSE {
Write-output "$file does not have the archive bit set. Deleting."
remove-item -recurse $file.fullname
$output =$_.ErrorDetails
}
}
} #end Foreach
Next, export the BackupName & Counts to an xml file, Count the files (by BackupName type) in the archive folder, and then set the archive bit on ALL files so we can mark excess ones for deletion.
# Export BackupCounts to XML
$bkup_counts | Export-Clixml bkup_counts.xml
# Get Number of ZIP files in folder
$btype = $null
foreach ($btype in $bkupTypeArr) {
$FileCountArray += ,@(($btype),(dir $zipFolder\$btype"*.zip").count)
}
# Import BkupCounts from XML
$bkup_Counts= Import-Clixml bkup_counts.xml
# set Attribute byte on ALL files in zipfolder so we know we'll get the right ones
attrib $zipFolder"\*" +a
Now, go back thru the bkup_Counts hashtable, getting the number of files matching the BackupName pattern, and subtracting the days of retention from the count of the number of files. If the difference is greater than 0, then we have to delete that many entries from that folder.
We do that by piping the folder using that same BackupName pattern, sorting it on the Creation Time property, then selecting the first N files, where N is the number greater than zero on the retention value above.
Then, for each object we selected, go back & set the Archive bit to FALSE to mark it for deletion on the next run.
$row = $null
# Get LST & DEF filenames in array & display count
foreach ($row in $bkup_Counts.Keys) {
Get-ChildItem -Path $zipFolder -Include $row"*" -Recurse #|
(dir $zipFolder\$row"*".zip).count - $bkup_Counts[$row]
$delfiles = 0
$delfiles = (dir $zipFolder\$row"*".zip).count - $bkup_Counts[$row]
if ($delfiles -gt 0) { #sort folder by createdtime
# if more than specified nbr of backups present, un-archive excess ones to delete next run.
dir $zipFolder\$row"*" | sort-object -property {$_.CreationTime} |
select-object -first $delfiles |
foreach-object { attrib $_.FULLNAME -A}
} # end if delfiles gt 0
} # End foreach in bkup_counts
} # End Else Last ExitCode = 0
Here is the code in its entirety.
function create-7zip([String] $aDirectory, [String] $aZipfile){
[string]$pathToZipExe = "C:\Program Files\7-zip\7z.exe";
[Array]$arguments = "a", "-tzip", "$aZipfile", "$aDirectory", "-r";
& $pathToZipExe $arguments;
}
# Call it by using:
#create-7zip "c:\temp\myFolder" "c:\temp\myFolder.zip"
#************************************************************************************
#************************************************************************************
# Initialize variables
$zipFolder = "C:\ZipFiles"
$nameConv = "{0:yyyyMMdd_HHmmss}" -f (Get-Date) + ".zip"
$fileList = @{}
$FileCountArray = @()
$bkupTypeArr = @()
$myDocFolder = "c:\Documents and Settings\MyPC\My Documents\"
# Import text files for master, folder and file backup information
$bkupRotateMasterArr = Import-Csv $myDocFolder"RotateBackups_MasterList.txt"
$fldrBkupArray = Import-Csv $myDocFolder"RotateBackups_FolderList.txt"
$fileExtBkupArr = Import-Csv $myDocFolder"RotateBackups_FileExtensions.txt"
# Switch to delete Item or to send to recycle bin
# delete is destructive and cannot recover file.
# Recycle setting removes file from folder, but sends to recycle bin
# and can be restored if needed.
# Must be either "Kill" or "Recycle"
$KillOrRecycle = "Recycle"
#************************************************************************************
#************************************************************************************
# Load contents of master backup array
$bkup_Counts = @{}
$b = $null
foreach($b in $bkupRotateMasterArr)
{
$bkup_Counts[$b.BackupName] = $b.VersionsRetained
}
#set Backup Types from the array we just defined
$type = $null
foreach ($type in $bkup_Counts.Keys) {
$bkupTypeArr += $type
}
#create array of our filenames for this batch
$type = $null
$fArray = @{}
foreach ($type in $bkupRotateMasterArr) {
$fArray[$type.BackupName] = ($type.BackupName + $nameConv)
}
# if extension array not null, get list of files to back up
if ($fileExtBkupArr) {
# Gather the list of files to be backed up
$f = $null
foreach ($f in $fileExtBkupArr) {
$arr = @()
$arr = (Get-ChildItem $f.FolderLoc -Recurse -Include $f.FileExtension | Select-Object fullname)
foreach ($a in $arr) {
if ($a) {
$fileList[$a] = $f.BackupName
} # if $a not null
} # end inner foreach
} # end outer foreach
} # if FileExtension Backup Array not null
# if filelist count gt zero, then create zip file of them for appropriate backup
if ($fileList.Count -gt 0) { # must have entries in hashtable
$f = $null
#Loop thru file list & associate file with the appropriate backup
foreach ($f in $fileList.Keys) {
$arcFile = $null
if ($fileList.ContainsKey($f)) {
if ($fArray.ContainsKey($fileList[$f])) {
$arcFile = $fArray[$fileList[$f]]
create-7zip $f.FullName $zipFolder\$arcFile
} #if key in fArray
} # if key in Filelist
} # end foreach
} # if hastable not empty
# if folder backup not null then back up folders
if ($fldrBkupArray) { # check if array not null (no entries)
$f = $null
#Backup Folders now
foreach ($f in $fldrBkupArray) {
$arcFldr = $null
#if ($fArray.ContainsKey($f[1])) {
if ($fArray.ContainsKey($f.BackupName)) {
$arcFldr = $fArray[$f.BackupName]
create-7zip $f.FolderName $zipFolder\$arcFldr
} #end if
} # end foreach
} # end if $fldrBkupArray not null
# if 7zip succeeded, we'll continue
if ($LASTEXITCODE -gt 0)
{Throw "7Zip failed" }
ELSE { # if Exitcode = 0 then continue with job
# Remove any files with Archive bit = False
# we marked it for deletion in previous run
Add-Type -AssemblyName Microsoft.VisualBasic
$files=get-childitem -path $zipFolder
# we'll delete all files that don't have the archive bit set
Foreach($file in $files) {
If((Get-ItemProperty -Path $file.fullname).attributes -band [io.fileattributes]::archive)
{ Write-output "$file is set to be retained" }
ELSE {
if ($KillOrRecycle = "Recycle") {
Write-output "$file does not have the archive bit set. Deleting (Sent to recycle bin)."
[Microsoft.VisualBasic.FileIO.Filesystem]::DeleteFile($file.fullname,'OnlyErrorDialogs','SendToRecycleBin')
$output = $_.ErrorDetails
}
ELSE {
Write-output "$file does not have the archive bit set. Deleting."
remove-item -recurse $file.fullname
$output =$_.ErrorDetails
}
}
} #end Foreach
# Export BackupCounts to XML
$bkup_counts | Export-Clixml bkup_counts.xml
# Get Number of ZIP files in folder
$btype = $null
foreach ($btype in $bkupTypeArr) {
$FileCountArray += ,@(($btype),(dir $zipFolder\$btype"*.zip").count)
}
# Import BkupCounts from XML
$bkup_Counts= Import-Clixml bkup_counts.xml
# set Attribute byte on ALL files in zipfolder so we know we'll get the right ones
attrib $zipFolder"\*" +a
$row = $null
# Get LST & DEF filenames in array & display count
foreach ($row in $bkup_Counts.Keys) {
Get-ChildItem -Path $zipFolder -Include $row"*" -Recurse #|
(dir $zipFolder\$row"*".zip).count - $bkup_Counts[$row]
$delfiles = 0
$delfiles = (dir $zipFolder\$row"*".zip).count - $bkup_Counts[$row]
if ($delfiles -gt 0) { #sort folder by createdtime
# if more than specified nbr of backups present, un-archive excess ones to delete next run.
dir $zipFolder\$row"*" | sort-object -property {$_.CreationTime} |
select-object -first $delfiles |
foreach-object { attrib $_.FULLNAME -A}
} # end if delfiles gt 0
} # End foreach in bkup_counts
} # End Else Last ExitCode = 0
