Im working on a small program where I have three threads with each a task to run: copy a file from A to B using a bash script. One thread copies a 100GB file, then another 2x a 10GB file.
The 100GB file copy starts then with a delay I start the copy of the first 10GB file when that is done the last 10GB copy starts. From the network speed I see that the copy of the first file starts with 120MB/s (more or less), then the 10GB file starts and we see some noise on the line but no real drop in transfer speed. When the 10GB files are finished, the 100GB file continues at a significant lower rate:
import subprocess
import threading
import time
def run(arguments):
process = subprocess.Popen(
arguments,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True, # For string output instead of bytes
)
# Print each line as it comes
for line in process.stdout:
print(line, end="\n") # `line` already includes newline
print(f"Done for thread {threading.current_thread()}")
def create_threads(experiment_paths, experiment_name):
machine_name = "test"
threads = []
for i, path_name in enumerate(experiment_paths):
cli_args = [
"bash",
mount_script,
f"--folder={experiment_name}",
f"--thread={i}",
f"--file-to-copy={path_name}",
f"--machine-name={machine_name}",
]
new_thread = threading.Thread(target=run, daemon=False, args=(cli_args,))
threads.append(new_thread)
return threads
experiment_name = 'test'
experiment_paths = ['/my/path/to/file/100gb', '/my/path/to/file/10gb', '/my/path/to/file/10gb']
threads = create_threads(experiment_paths, experiment_name=experiment_name)
t0 = time.time()
for t in threads:
print(f"Starting thread {t.name}")
t.start()
time.sleep(80)
for t in threads:
print(f"Joining thread {t.name}")
t.join()
how can I ensure that the copy speed of the 100GB file resume at maximum speed? Im working on a linux system btw, copying to a mounted cifs share (samba)
EDIT: when only transfering the 100GB file it goes at full speed the whole time. The idea of the 80second delay was to see if the 100GB transfer at least starts with full speed

fragmentationsmight be that the first 20GB, closer together than the rest of file.