0

I have two external HDDs with enough space to accommodate a 2TB device file.

I never tried to copy a file in two places simultaneously...

I have a very fast NVMe disk drive and want to make a backup copy of it on those two external HDDs which are slow.

Is it possible to copy it (/dev/nvme0n1) on both external HDDs at once with one command?

Improve this question
7
  • Are you sure you can? en.wikipedia.org/wiki/Von_Neumann_architecture Commented Jun 23, 2024 at 17:43
  • 1
    @oldfred not quite sure how copying from a DMA-capable permanent storage to some (presumable partially DMA-capable) permanent storage interacts with the idea of von-Neumann architectures? (this is an honest question, sounds like you have some thoughts on that, but I can't really put something together) Commented Jun 23, 2024 at 17:44
  • I do not know system design, but unless CPU can turn the copy process over to two separate sub-systems, the CPU just switches back and forth between the two copy processes. Even if NVMe is DMA capable is external USB? And external USB often is the limiting factor although now there are so many versions, it is hard to keep track of what speed port & device combined may be. Commented Jun 23, 2024 at 19:24
  • 1
    @oldfred USB usually isn't, but the way you can put data into USB packets has gotten improved a lot. With USB UASP support from USB device through host controller through host driver, which is pretty standard these days, yep, a lot USB storage devices can be DMA'd to! it's really just "SCSI over USB", and host controllers that have a mode that makes them act as DMA-capable SCSI controllers. It's technologically pretty interesting. However, point being here: yep, the CPU would still control two transfers, but Linux buffers reads like that, so reading happens only once (unless one target is … Commented Jun 23, 2024 at 19:54
  • 1
    … much slower than the other), and the copy processes would really just trigger two write syscalls, which the kernel internally translates into a list of transfer commands to set off – and that's it, it sets up the transfers, and lets the NVMe transfer to RAM using DMA once, and otherwise enqueues DMA-aided transfers to the targets. The interaction needed after that is only making pages as "not used anymore" once both targets signalled back they're done with that segment, and to occassionally enqueue new reads once one of the targets gets close to the end of the currently available Commented Jun 23, 2024 at 19:58

1 Answer 1

Reset to default
2

Is it possible to copy it (/dev/nvme0n1) on both external HDDs

Yes

at once

Yes

with one command?

I suppose so:

tee /path/to/destination/1 >/path/to/destination/2 </dev/nvme0n1

But if the two devices are slow you probably want to decouple the writing for each, so a couple of separate commands would seem to make more sense:

cat /dev/nvme0n1 >/path/to/destination/1 &
cat /dev/nvme0n1 >/path/to/destination/2 &
wait
Improve this answer
4
  • it makes a lot of sense to decouple two slow writers – especially because storage devices can be slower in different regions, and you'd otherwise would be writing at the speed of the slower one. However, I believe especially if they were fast, and of the same file system type, and you're using a modern kernel, and a lot of stars aligned, cat file > file would cause GNU coreutils cat to copy_file_range, and that would reduce the context switches to a minimum and do all the data movement scheduling in-kernel. In reality: I've yet to witness copy_file_range working on anything but the Commented Jun 23, 2024 at 20:13
  • same file system (as in: on the same volume). Sadly, I couldn't figure out why coreutils cat and cp implement detection and usage of copy_file_range but not sendfile, though the latter has been around since forever (linux 2.4?) and should in theory be more efficient than the read + write done when copy_file_range fails. It's only mentioned in the coreutils TODO as something to implement. Maybe someone did and it wasn't significantly faster? Who knows. Commented Jun 23, 2024 at 20:16
  • Very likely the copy bottleneck will be the slower disk devices, not the cpu, so likely these fancy suggestions won't help much. Running the copies in parallel might be faster because as long as the copies stay relatively in sync, it will read from the source disk once and one of the copies gets the data from cache. But if the source disk is faster than the target(s) anyway, it might not make much difference either. If the disks are USB2, bus contention writing to two disks at once might slow it down. If you are using a faster bus (? sata, usb3, usbc) then this wouldn't be an issue. Commented Jun 24, 2024 at 4:53
  • 1
    It took almost 5 hours with my 2.5-inch external HDDs (4TB + 5TB), one was ~5 minutes faster. As recommended, I did two simultaneous transfers using pv and Live Mint USB. Cheers to all. Commented Jun 24, 2024 at 5:50

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.