We are currently backing up all of our databases to a series S3-Compatibable storageGRID buckets hosted by our data center provider. Some of our larger databases backups run quite long (upwards of 8 hours for fulls) and our largest backup (2,200Gb database) frequently receives the following error:
Write to S3 object storage device failed. Device has reached its limit of 10000 allowed parts.
From reading documentation I understand that this is because the maximum file size in S3 is a function of the max transfer size and the maximum number of parts (10000 in this case), however we are using MAXTRANSFERSIZE = 20971520, which should allow ~200Gb file sizes, much larger than we are writing (after compression). Currently we are seeing that differentials are failing with this error some of the time, even if the backups are 4 files as small as 40Gb each. Full backups for this database are written across 8 files, each ~115Gb after compression, and those succeed. We don't have visibility into the utilization of the hardware hosting our storage target since it's hosted by a 3rd party, but support has told us we are not pushing it past 50%, which we just have to trust is true.
The full set of backup parameters were using are:
WITH CHECKSUM, COMPRESSION, <DIFFERENTIAL>, BUFFERCOUNT = 20, MAXTRANSFERSIZE = 20971520
