Creating n folders with equal number of files from a large folder with all files

#!/bin/sh
batchsize=5
batchcount=10

tdir=1
buffer=0
for i in *; do
    [ $buffer -eq $batchsize ] && tdir=$((tdir + 1)) && buffer=0
    [ $tdir -gt $batchcount ] && break
    [ -d "$tdir" ] || mkdir -p $tdir
    if [ -f "$i" ]; then
        buffer=$((buffer + 1))
        cp "$i" $tdir/
    fi
done

Running this script in a directory, will make a directory called 1, put the first 5 files in it, then make a directory called 2 and put the next 5 files in that and so on until it's created and filled 10 directories.

If you want to test this, replace the cp command with echo cp so it'll show you the copy commands instead of actually running them, and comment out the mkdir command so it won't create the directories.

Undoing it should be real simple too, just run

rm [1-10]/* ./ && rmdir [1-10]

But if you had any other folders named 1-10 that weren't created by the command, all data in them would be lost so careful with this one.

I don't think there's any more convenient way to do this, this kind of stuff is what bash scripts are for.

Here is an alternative version that automatically fills in either batchcount or batchsize if one of them is not set.

#!/bin/sh
batchsize=auto
batchcount=5

ceil(){ #  This function rounds a number up
    [[ $@ == *"."* ]] && [[ "${@##*\.}" -gt 0 ]] && echo $(("${@%%\.*}" + 1)) || echo "${@%%\.*}"
}

if [[ ! "${batchsize}${batchcount}" =~ ^-?[0-9]+$ ]]; then #One of the variables is not an integer
    filecount=$(ls -Ap | grep -v '/' | wc -l)
    if [[ ! "$batchsize" =~ ^-?[0-9]+$ ]] && [[ ! "$batchcount" =~ ^-?[0-9]+$ ]]; then #Neither of the variables is set.
        echo "Error: Batchsize and Batchcount are both unset, please set at least one of them."
        exit
    elif [[ ! "$batchsize" =~ ^-?[0-9]+$ ]]; then #Only batchsize is unset
        batchsize=$(ceil $(bc -l <<< "$filecount / $batchcount"))
    elif [[ ! "$batchcount" =~ ^-?[0-9]+$ ]]; then #Only batchcount is unset
        batchcount=$(ceil $(bc -l <<< "$filecount / $batchsize"))
    fi
fi

tdir=1
buffer=0
for i in *; do
    [ $buffer -eq $batchsize ] && tdir=$((tdir + 1)) && buffer=0
    [ $tdir -gt $batchcount ] && break
    [ -d "$tdir" ] || mkdir -p $tdir
    if [ -f "$i" ]; then
        buffer=$((buffer + 1))
        cp "$i" $tdir/
    fi
done

So in the above configuration's case, it will divide the files into 5 batches, so if you have 23 files, it will divide them into 5 batches of 5, with the last batch only having 3.

If you set batchcount to auto, and set batchsize to 3, it will divide the files evenly into 7 batches of 3.

This solution spreads the files as equally as it can, numerically speaking, across the number of required batches. It does this by taking the first nrBatches files and putting one into each of the target directories, and then taking the next nrBatches files and spreading those, and so on until we run out of files.

# How many batches
nrBatches=5

# Delete the target batch directories
rm -rf -- "/dump/merged_batches/batch"*

# And off we go…
batch=1
for file in /dump/merged/*.sorted.labeled.bam*
do
    # Only files
    [ -f "$file" ] || continue

    # Target batch directory
    tbDir="/dump/merged_batches/batch$batch"

    echo "Putting $file into batch $batch ($tbDir)" >&2
    mkdir -p -- "$tbDir"
    cp -- "$file" "$tbDir"

    # Move to the next batch
    batch=$(( (batch % nrBatches) +1 ))
done

For testing purposes you may want to comment out the mkdir and cp commands. If you want a silent solution comment out or omit the echo.

Here's yet another option, using an array to handle arbitrary filenames as NUL-terminated strings, and taking into consideration the size of the files:

#!/usr/bin/env bash

batches=5

declare -a files

readarray -d '' files < <(
    find merged/*.sorted.labeled.bam* -type f -print0
)

N="${#files[@]}"                # N files found
 
n=$(((N+batches-1) / batches))  # n files per batch
 
rm -rf merged_batches/
eval mkdir -p merged_batches/batch{1..$batches}

for k in "${!files[@]}"
do
    printf 'cp -p %s "merged_batches/batch%d/"\n' "${files[k]@Q}" $((k / n + 1))
done

That script will print the commands to copy your files into the batched directories. Pipe it to less or your favorite pager and inspect the proposed command list:

./test.sh | less

If you approve of that work, exit the pager, and this time, pipe the script into a shell:

./test.sh | bash

If the files are large, or so very numerous that they total up to a significant amount of space, consider creating hard links instead. Change the printf statement to be:

printf 'ln %s "merged_batches/batch%d/"\n' "${files[k]@Q}" $((k / n + 1))

Using ln instead of cp means that the batched files won't occupy any space except the space the original files take up. If the files are large, this will also make the script run faster. If your plan is to eventually delete the original files and move forward with the batched files, then further down the road, you can eventually:

rm -rf merged/*.sorted.labeled.bam*

and still retain the batched files under the merged_batches/ directory.