2

My backup strategy currently primarily consists of daily backups of all of my machines with Borg Backup, stored on different storage devices in different locations, following the 3-2-1 strategy. These file-level backups are the important ones that matter most to me. My question is not about these backups, I mention them for context.

Next to Borg Backup I also sporadically create full disk image backups with dd. After zero'ing the disk's free space (using zerofree on ext4, dd if=/dev/zero otherwise) I usually create a SquashFS image of the raw disk image (e.g. sda.img becomes the only file in disks.sqfs). This allows me to store the raw disk image in compressed form, while still allowing me to access the data without the need to decompress everything first.

These full disk image backups are stored on a single storage device (a NAS to be more precise), i.e. don't follow the 3-2-1 strategy. Creating a second copy of the data is out of scope, simply because they take too much space and because I consider investing into more storage a waste of money due to my Borg Backup backups. So, I'm fine with loosing these backups per-se, but I want to protect them a little better. Thus I'm thinking about adding some sort of error correction mechanism.

I read through a lot of resources and found that Reed–Solomon error correction seems to be the way to go. It adds some overhead to the data stored and provides safety in most, even though not all cases.

My question is the following: How do I do that in practice? What tools are available and how would I use them in my case? I found this 10 years old Stack Exchange question listing a whole bunch of tools, but many of the projects are apparently dead. Plus, they don't seem to fit my needs:

Storing the data in compressed form and yet being able to access the data without the need to decompress it first is a must-have for me. So, unless there's another solution, I'm stuck with SquashFS. However, according to the resources I read, combining ECC with compression is hard: One apparently shouldn't calculate ECCs from compressed data, but from the original data, because ECC doesn't guarantee a 100% correction and even a single remaining corruption could yield all compressed data useless. However, calculating ECCs from original data and then compressing it wouldn't help either, because I might not be able to decompress the data due to the corruptions. So, apparently one needs software that does both at the same time: compression and ECC. Per ddrescue I found that lzip can actually do that by creating forward error correction (fec) files alongside compressing the data, but AFAIK I can't tell SquashFS to create these files.

So, I'm kinda stuck with this chicken-and-egg problem... How can I combine SquashFS with ECC, or is there an alternative to SquashFS that allows this?

Any suggestions?

Improve this question
11
  • One possible alternative to SquashFS is DwarFS however even that relies on par2cmdline for your use case, see DwarFS: Dealing with bit rot. Commented Apr 1 at 19:52
  • This question is similar to: What are the self-healing file formats?. If you believe it’s different, please edit the question, make it clear how it’s different and/or how the answers on that question are not helpful for your problem. Commented Apr 1 at 19:59
  • By the way, Reed-Solomon codes as used in disk-level ECC are a sensible compromise between error correction capabilities and block length, and decoding complexity. Since the block size for adding error correction information to an image file can, without causing any issue, easily be hundreds of kilobits, it makes little sense to constrain oneself to such (typically) low-rate (i.e. much data overhead for error protection) codes; a sensible design could use long LDPC codes as known from highest-rate fiberoptical comms or satellite TV downlinks, and apply even longer interleavers to become robust Commented Apr 1 at 20:08
  • … against much more likely failure modes (failure of complete device blocks of data, i.e., burst erasure, and clustered bit errors in multiple places). Generally, "I apply some error correction" is much less useful then "I apply error correction that specifically deals with the errors that are likely to occur". Commented Apr 1 at 20:10
  • I have never used it but a nice solution (as it could be transparently used below the squashfs image) seems to be the forward error correction (FEC) feature of dm-verity: docs.kernel.org/admin-guide/device-mapper/verity.html Commented Apr 1 at 20:19

0

Reset to default

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.