VixinG VixinG - 3 months ago 17
jQuery Question

Prompt to download a stream of multiple files as one

Assuming there's a server storing multiple files (not necessarily text documents):

http://<server>/<path>/file0001.txt ... http://<server>/<path>/file9999.txt


If user was to download all of those files as one, how would I do it in javascript?

Normally user would have to download 9999 files and join them on his drive.
How can I prompt a download of a file and stream the data of multiple files while javascript gets them, just like it's a stream of one, big file.

I imagine it would be something like this (excuse me for lack of javascript, just trying to explain):

With (download prompt of 'onefile.txt') as connection:
While connection is open:
For file in file_list:
get file
return file.contents
connection close


Downloading each file and storing it in memory until the last one is retrieved is not a good idea, since overall size of that file can be quite big.

I'm wondering if that's even possible. I can write it in python, but that's another story. I wanted to make it a javascript function on a website.

Answer

I'm surprised javascript can't just create a "virtual localhost connection" where it uses some generator to "yield" the contents of each file...

Well, if you use a service worker then you can manipulate the response and give it a readableStream which you can "yield" the content of each file...


This is what the streamSaver dose internally but takes away all hassle...
I will show you an example using es6/7 and StreamSaver.js

It's not tested it's just a ruffly idea. This will consume very little memory, but it's limited to only Blink ATM if you wanna use StreamSaver

let download = Promise.coroutine(function* (files) {
    const writeStream = streamSaver.createWriteStream('onefile.txt')

    // Later you will be able to just simply do
    // yield res.body.pipeTo(writeStream) instead of pump

    for (let file of files) {
        let res = yield fetch(file)
        let reader = res.body.getReader()

        let pump = () => reader.read()
        .then(({ value, done }) => !done &&
            // Write one chunk, then get the next one
            writeStream.write(value).then(pump)
        )

        yield pump()
    }

    // Close the stream when you are done writing
    writeStream.close()
}

download([
    'http://<server>/<path>/file0001.txt',
    'http://<server>/<path>/file9999.txt'
]).then(() => {
    alert('all done')
})