How to create multiple CSV files to avoid 10MB file content limit using map reduce script?

Question

I hope you are welll. I am trying to store dataset results in a csv file. but I am getting error as, SSS_FILE_CONTENT_SIZE_EXCEEDED error. So my plan is to create multiple files. But, I need to check file size, if its 10MB then create second file and store subsequent data in that. My concern is, before saving the file we couldn't get the file size. Thanks in advance!

function summarize(summary) {
                let csvContent = [];
                let datasetObj = dataset.load({ id: 'custdataset70' }); 
                let headers = datasetObj.columns.map(col => col.label || col.id);
                csvContent.push(headers);
    
                summary.output.iterator().each((key, value) => {
                    csvContent.push(value); // Keep value as string
                    return true;
                });
    
                if (csvContent.length <= 1) { // Only headers, no data
                    log.debug('No Data', 'No unique dataset records found.');
                    return;
                }
    
                let fileId = saveCSVFile(csvContent.join('\n'));
                if (fileId) {
                    sendEmail(fileId);
                }
                          
       
        }
        function saveCSVFile(csvContent) {
                let today = format.format({ value: new Date(), type: format.Type.DATE });
                let csvFile = file.create({
                    name: 'DatasetResults_' + today + '.csv', // Dynamic filename
                    fileType: file.Type.CSV,
                    contents: csvContent,
                    folder: 5081389 
                });
                return csvFile.save();
                }

Arek Kubiński · Accepted Answer · 2025-02-28 10:31:32Z

0

You can check how much space takes up empty file (since it seems like you are putting some data inside by specifying file type), then just take you array I'm guessing line by line and add the value of new Blob(["myString"]).size, this should output the size in bytes for the line of data you are putting inside csv.

Then if the size is bigger or equal to threshold just put it inside new file.

answered Feb 28 at 10:31

Arek Kubiński

2695 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to create multiple CSV files to avoid 10MB file content limit using map reduce script?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related