Angular - Efficient way to parse 10000 rows of csv/tsv file

Question

I'm looking for a more sufficient way (or lib) to parse csv/tsv that contains around 5000 ~ 10000 rows (using it to render a table with cdk virtual scroll for previewing the file). My current implementation is quite banana for that amount of rows.

this.httpClient.get(this.dataSrc, {
  headers: {
    Accept: 'text/plain'
  },
  responseType: 'text'
}).subscribe(res => {
  // handles tsv or csv content
  const lines = res.split(/\r|\n/);
  const separator = lines[0].indexOf('\t') !== -1 ? '\t' : ',';
  this.parsedCSV = lines.map(l => l.split(separator));
});

fast-csv is a pretty lightweight package for that.

ahsan
– ahsan

2021-11-18 10:48:38 +00:00
Commented Nov 18, 2021 at 10:48 — ahsan
– ahsan, Commented Nov 18, 2021 at 10:48

halfer · Accepted Answer · 2021-12-10 23:24:24Z

1

It looks good, but parsing big amount of data will freeze the thread. You should sleep and await the thread so the parser won't freeze.

Here is how I would do it, have a look.

const sleep = (msTime) => {
 return new Promise((resolve, reject) => {
    setTimeout(() => {
      resolve();
    }, msTime);
  });
}

const parse(csv, onProgress) => {
  const lineSplitter = RegExp('\r|\n', 'g');
  const result = []
  var index =0;
  let match;
  // split one each time, so the thread won't freeze
  while ((match = lineSplitter.exec(csv)) !== null) {
    const line = match[0];
    const separator = line.indexOf('\t') !== -1 ? '\t' : ',';
    result.push(line.split(separator))
    if (index % 30 === 0)
       await sleep(10); // This will make sure the thread won't freeze
    if (onProgress)
       await onProgress((index / lines.length) * 100);
    index++;
  }
 return result;
}

edited Dec 10, 2021 at 23:24

halfer

20.2k20 gold badges110 silver badges207 bronze badges

answered Nov 18, 2021 at 11:00

Alen.Toma

4,8782 gold badges19 silver badges37 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

halfer Over a year ago

Hi there Alen. I wonder, would you be able to use an English spelling checker on your work? It would help volunteer editors out enormously.

Collectives™ on Stack Overflow

Angular - Efficient way to parse 10000 rows of csv/tsv file

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related