Provide ReadableStream as a download source for streamed download

Question

Searching for "download ReadableStream", I find two questions:

Both have answers ending up in collecting all the data in a Blob and using URL.createObjectURL. Yet having a ReadableStream, it is a waste to first create a huge Blob in memory before the browser can start the download operation.

Is there a way to let the client download directly from a ReadableStream, without first collecting all data in memory?

NOTE: this ReadableStream is created by the client from lots of data, so relying on fetch Response objects will hardly be a solution.

Your question 2 has a "what about this" comment which appears to offer a more direct solution. — James
– James, Commented Mar 29 at 18:32

guest271314 · Accepted Answer · 2025-03-29 21:43:12Z

1

This is how to use WHATWG Streams ReadableStream from WHATWG Fetch Response on Chromium-based browsers (Chrome, Chromium, Brave, Opera, Edge, et al.) and write that file (it's also possible to write folders) to the filesystem using WICG File System Access API - not be be confused with WHATWG File System (Standard), which share some of the same interfaces, such as FileSystemWritableFileStream

let handle = await showSaveFilePicker({
  startIn: "downloads",
  suggestedName: "qjs"
});

let writable = await handle.createWritable();

let readable = (await fetch(
  "https://corsproxy.io?url=https://github.com/quickjs-ng/quickjs/releases/latest/download/qjs-linux-x86_64",
)).body;

await readable.pipeTo(writable);

console.log("Done downloading file");

answered Mar 29 at 21:43

guest271314

1

Sign up to request clarification or add additional context in comments.

2 Comments

Harald Mar 30 at 7:00

To summarize: showSaveFilePicker -> FilesystemHandle -> createWriteable -> pipeTo and, as you say, the odd numbered in this list are Chromium-based only so far. Pity :-/ But I assume this strongly suggests it is impossible for now to get going on Safari or Firefox.

guest271314 Mar 30 at 13:57

Not impossible. You can use fetch() or WebTransport and a local HTTP/2 or HTTP/3 server.

guest271314 · Accepted Answer · 2025-03-29 18:40:26Z

Yes.

Here's how to do that using Deno, from the browser with Native Messaging https://github.com/guest271314/native-messaging-file-writer

var readable = ... // WHATWG ReadableStream
var fs = new FileWriter({
  fileName: "/home/user/Downloads/node", // File path to write to
  mode: 0o764 // Mode 
}, "/home/user/native-messaging-file-writer"); // Path to unpacked extension directory
fs.write(readable).then(console.log).catch(console.warn);
// Abort writing to the file
fs.abort("reason");

And here's how to do that using QuickJS NG https://github.com/guest271314/native-messaging-file-writer/tree/quickjs. Still, from the browser, using Native Messaging.

Fetch and write latest QuickJS NG qjs to file system



var {
  externalController,
  progressStream
} = await connectExternalFileWriter(
  "/home/user/native-messaging-file-writer-quickjs",
  "/home/user/Downloads/qjs",
  ["O_RDWR", "O_CREAT", "O_TRUNC"],
  "0o744"
).catch(console.error);
// externalController.error("a reason");
// externalController.close();
progressStream.pipeTo(new WritableStream({
  start() {
    console.groupCollapsed("FileWriter progress");
  },
  write(v) {
    console.log(v);
  },
  close() {
    console.groupEnd("FileWriter progress");
  },
  abort(reason) {
    console.log(reason);
    console.groupEnd("FileWriter progress");
  }
}), ).catch(console.error);

var writeStream =
  fetch(
    "https://corsproxy.io?url=https://github.com/quickjs-ng/quickjs/releases/latest/download/qjs-linux-x86_64",
  ).then((r) => r.body.pipeTo(new WritableStream({
    write(v) {
      console.log(externalController);
      externalController.enqueue(v);
    },
    close() {
      externalController.close();
    },
  })));

Fetch and write node nightly to file system

const {
  UntarFileStream
} = await import(URL.createObjectURL(new Blob([await (await fetch("https://gist.githubusercontent.com/guest271314/93a9d8055559ac8092b9bf8d541ccafc/raw/022c3fc6f0e55e7de6fdfc4351be95431a422bd1/UntarFileStream.js")).bytes()], {
  type: "text/javascript"
})));

const cors_api_host = "corsproxy.io/?url=";
const cors_api_url = "https://" + cors_api_host;
let osArch = "linux-x64";
let file;

let [node_nightly_build] = await (await fetch("https://nodejs.org/download/nightly/index.json")).json();
let {
  version,
  files
} = node_nightly_build;
let node_nightly_url = `https://nodejs.org/download/nightly/${version}/node-${version}-${osArch}.tar.gz`;
let url = `${cors_api_url}${node_nightly_url}`;
console.log(`Fetching ${node_nightly_url}`);
const request = (await fetch(url)).body.pipeThrough(new DecompressionStream('gzip'));
// Download gzipped tar file and get ArrayBuffer
const buffer = await new Response(request).arrayBuffer();
// Decompress gzip using pako
// Get ArrayBuffer from the Uint8Array pako returns
// const decompressed = await pako.inflate(buffer);
// Untar, js-untar returns a list of files
// (See https://github.com/InvokIT/js-untar#file-object for details)
const untarFileStream = new UntarFileStream(buffer);
while (untarFileStream.hasNext()) {
  file = untarFileStream.next();
  if (/\/bin\/node$/.test(file.name)) {
    break;
  }
}

var stream = new Blob([file.buffer]).stream();

var {
  externalController,
  progressStream
} = await connectExternalFileWriter(
  "/home/user/native-messaging-file-writer-quickjs",
  "/home/user/Downloads/node",
  ["O_RDWR", "O_CREAT", "O_TRUNC"],
  "0o744"
).catch(console.error);
// externalController.error("a reason");
// externalController.close();
progressStream.pipeTo(new WritableStream({
  start() {
    console.groupCollapsed("FileWriter progress");
  },
  write(v) {
    console.log(v);
  },
  close() {
    console.groupEnd("FileWriter progress");
  },
  abort(reason) {
    console.log(reason);
    console.groupEnd("FileWriter progress");
  }
}), ).catch(console.error);

var writeStream = stream.pipeTo(new WritableStream({
    write(v) {
      externalController.enqueue(v);
    },
    close() {
      externalController.close();
    },
  }));

writeStream.catch(console.error);

Technically, this is also possible using WICG File System Access API in the browser (Chromium-based browsers such as Chrome, Opera, Brave, Edge, Chromium) alone, without using a Web extension or Native Messaging.

Here's one way I fetch the Node.js nightly archive, extract only the node executable, and write that file to the filesystem https://github.com/guest271314/download-node-nightly-executable/blob/main/index.html#L29-L68. Hint: If you do something like

cd ~/Downloads
touch node
chmod u+x node

to create an empty file named node and set executable permission on that file, the file permission will be retained, see https://issues.chromium.org/issues/40742294

      try {
        let [node_nightly_build] = await (
          await fetch('https://nodejs.org/download/nightly/index.json')
        ).json();
        let {
          version,
          files
        } = node_nightly_build;
        let node_nightly_url = `https://nodejs.org/download/nightly/${version}/node-${version}-${osArch}.tar.gz`;
        let url = `${cors_api_url}${node_nightly_url}`;
        console.log(`Fetching ${node_nightly_url}`);
        const request = (await fetch(url)).body.pipeThrough(
          new DecompressionStream('gzip')
        );
        // Download gzipped tar file and get ArrayBuffer
        const buffer = await new Response(request).arrayBuffer();
        // Decompress gzip using pako
        // Get ArrayBuffer from the Uint8Array pako returns
        // const decompressed = await pako.inflate(buffer);
        // Untar, js-untar returns a list of files
        // (See https://github.com/InvokIT/js-untar#file-object for details)
        const untarFileStream = new UntarFileStream(buffer);
        while (untarFileStream.hasNext()) {
          file = untarFileStream.next();
          if (/\/bin\/node$/.test(file.name)) {
            break;
          }
        }
        writable = await fileSystemHandle.createWritable();
        writer = writable.getWriter();
        await writer.write(file.buffer);
        await writer.close();
        new Notification('Download complete.', {
          body: `Successfully downloaded node executable ${version}`
        });
      } catch (e) {
        console.log(e);
      } finally {
        console.log('Done');
      }

Now, you can do something like

await readable.pipeTo(writable);

to pipe a WHATWG ReadableStream to the file system on chrome, and that atomic pipe automatically gets rid of the .crswap file chrome writes to originally before moving the data to the actual file.

I wrote the above approaches using deno and qjs to fix the currently won't fix status of the linked Chromium bug https://issues.chromium.org/issues/40743502.

Collectives™ on Stack Overflow

Provide ReadableStream as a download source for streamed download

2 Answers 2

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related