How to parse multiple Large XML files with good performance and memory usage balance?

I was able to solve my problem, following what jdweng suggested with his ps script, I then went back to my first solution of the parsing which looks similar to his.

At first I decided to get the files from a folder directly from node.js and not to transfer them from front-end to back-end because it would take very long to package them and send them.

Then I tried to read the full files but those files where too big and my memory would crash, so I implemented a read by chunk logic.

So for anyone of who's struggling with something that looks like my case I did this:

Getting files from the folder and start to read them one by one:

async function getExcelRows(filterNome, filterStato){
        let result = [];
    
        const fileNames = await readFolder();
    
        for(let fileName of fileNames){
            const response = await readAndParseFile(folderPath+fileName, filterNome, filterStato);
    
            result.push(...response);
        }
        return JSON.stringify(result);
    }

Implementation for reading the folder:

async function readFolder(){
    const files = await fsp.readdir(folderPath);
    return files.filter(file => file.endsWith('.xml'));
}

Implementation of reading files in chunks:

async function readAndParseFile(filePath, filterNome, filterStato){

    return new Promise((resolve, reject) => {
        let keeper = {};
        let isToErase = true;
        let result = [];

        const re = new RegExp("<"+objTag+">", 'g');

        const stream = fs.createReadStream(filePath, { encoding: 'utf-8' });

        stream.on('data', async (chunk) => {
            
            if(isToErase){
                keeper = {};
                keeper.remain = chunk;
            }
            else{
                if(keeper.remain.indexOf("<"+objTag+">") != -1)
                    testFileOfferteNumber--;
                keeper.remain +=chunk;
            }

            testFileOfferteNumber += keeper.remain.match(re).length;

            if(keeper.remain.indexOf("<"+objTag+">") !== -1){
                keeper = await parseFile(keeper.remain, filterNome, filterStato);
                if(keeper?.result.length > 0){
                    result.push(...keeper.result);
                    keeper.result = [];
                }
                if(keeper?.remain != "")
                    isToErase = false;
                else
                    isToErase = true;
            }
        });

        stream.on('end', () => {
            resolve(result);
        });

        stream.on('error', (error) => {
            reject(error);
        });
    });
}

testFileOfferteNumber variable was to check if i was actually processing all tags without errors nothing more then that.

So here we read the file in chunk and if after parsing I still have some uncomplete content left i return that back to this function and I set the isToErase to false if that's the case, so that the next chunk will be appended to my current remaining content. Otherwize I just take the new chunk and process it.

Implementation of the parsing logic:

// objTag = "Offerte"
// filterNome = ["andrea"]
// filterStato = "italia"
function parseFile(source, filterNome, filterStato){
    return new Promise((resolve, reject)=>{
        try{
            const endTagLen = ("</"+objTag+">").length;
            let result = [];
            while(source.indexOf("<"+objTag+">") !== -1){

                if(source.indexOf("</"+objTag+">") === -1)
                    return resolve({result: result, remain: source});

                testResultOfferteNumber++;

                if(
                    !checkNomeFilter(filterNome, checkField(source, 'nome') ? getField(source, "nome") : "") ||
                    !checkStato(filterStato, checkField(source, 'stato') ? getField(source, "stato") : "")
                ){
                    source = source.slice(source.indexOf("</"+objTag+">")+endTagLen);
                    continue;
                }
                
                const obj = {
                    codice: checkField(source, 'codice') ? getField(source, "codice") : "",
                    nome: checkField(source, 'nome') ? getField(source, "nome") : "",
                    stato: checkField(source, 'stato') ? getField(source, "stato") : "",
                }

                source = source.slice(source.indexOf("</"+objTag+">")+endTagLen);

                result.push(obj);
            }
            resolve({result: result, remain: source});
        }
        catch(e){
            reject(e);
        }
    });
}

testResultOfferteNumber is the variable to compare with the previous one to check if all tags have been processed correctly.

So here I loop through all the tags "Offerte" as long as there are and in this "while" I check if there is a closing tag , if I don't, I return all what's left to the reading function which will append the next chunk of text to this text, doing so it will be able to check this content afterwards.

After, I apply my filters to avoid parsing data which is not accepted before parsing it, then because I know how the xml structure is, I will simply extract all 3 tags I need at once.

After this parsing I slice the already parsed part of the file away and restart the loop with the new content in the <Offerte>...</Offerte>.

I left the chunk size to the default one which worked for me perfectly, 3000 files were processed in about 20-30 minutes with no crashes.

For my case I needed to return the stringify of the result so I could create an Excel file.

I hope this solution helps someone struggling like me, thanks to everyone who helped me with this problem expecially jdweng.