Read only the first chunk of a file in Javascript

Question

We have an browser-based interface where a user selects a file to upload.

When a file is selected, it is loaded into a FileReader object, and the contents is loaded into memory and validated by calling FileReader.readAsText().

This works fine on 50MB files (takes 2-3 seconds). However when you go up to 200MB in size, and even after 20 mins, the file never finished reading and a processor remains pegged at 100% utilization.

I believe this is due to memory exhaustion, because the entirety of the file is being loaded into a single variable.

Is there any way to load the contents of the file in chunks, rather than loading the entire file into one enormous, memory sucking monster?

All we need is the first 1KB or so of the file to validate, so all we'd need to do is read the first chunk, valid, and be done.

Proof of concept, to test for yourself:

<form>
  <input type="file" /> File
</form>
<script type="text/javascript">
  var input = document.getElementsByTagName('input')[0];
  input.onchange = function(e) {
    var file = input.files[0];
    var reader = new FileReader();
    reader.onload = function() {
      var file_contents = reader.result;
      alert('File Read. Size:' + file_contents.length);
    };
    reader.readAsText(file);
  };
</script>

Maybe implement a FileReader.onprogress handler, accumulate the count of bytes read so far, and call FileReader.abort() when you want to stop. — jarmod
– jarmod, Commented Sep 22, 2020 at 19:52

backtick · Accepted Answer · 2020-09-22 20:03:52Z

2

I would leverage the Blob interface that File implements, whose .stream() method returns a ReadableStream.

async function validateFile(file) {
  const stream = file.stream();
  const reader = stream.getReader();
  let bytesRead = 0;
  let exhausted = false;
  while (bytesRead < 1e3 && !exhausted) {
    const {value, done} = await reader.read();

    // value is a Uint8Array
    bytesRead += value ? value.length : 0;
    exhausted = done;
  }
}

answered Sep 22, 2020 at 20:03

backtick

2,79612 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Joshua Burns Over a year ago

file.stream() pushed me in the right direction. file.slice() ended up fitting our needs better, but stream() could have works as well.

Collectives™ on Stack Overflow

Read only the first chunk of a file in Javascript

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related