Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Does not support sending very large files #10

Open
aswinshenoy opened this issue Jul 11, 2020 · 4 comments
Open

Does not support sending very large files #10

aswinshenoy opened this issue Jul 11, 2020 · 4 comments

Comments

@aswinshenoy
Copy link
Owner

Chunking takes some while and fails while trying to send very large files.

@aswinshenoy
Copy link
Owner Author

One of the cause can be #14 due to memory limit caused due to flawed chunking algorithm.

@aswinshenoy
Copy link
Owner Author

aswinshenoy commented Aug 4, 2020

One of the cause can be #14 due to memory limit caused due to flawed chunking algorithm.

The chunking algorithm on the sender's side seems to have been fixed (check #14 for stress test done by @MidhunSureshR) with the introduction of new chunking algorithm that only produces the next chunk and stores instead of creating and storing whole file at the beginning.

New algorithm

        const slice = this._file.slice(
            this._offset,
            this._offset + this._chunkSize
        );
        return await new Blob([slice]).arrayBuffer().then((buffer) => buffer);
    };

The following script is now obsolete and should be removed.
https://github.com/aswinshenoy/BayJDO/blob/master/functions/getChunksFromFile.js


export default async function getChunksFromFile(file, chunkSize = 6 * 1024 * 1024) {
    if(file) {
        return await file.arrayBuffer().then((buffer) => {
            let chunks = [];
            while(buffer.byteLength) {
                chunks.push(buffer.slice(0, chunkSize));
                buffer = buffer.slice(chunkSize, buffer.byteLength);
            }
            return chunks;
        });
    }
    return [];
};

@aswinshenoy
Copy link
Owner Author

Now there is another critical issue causing trouble in sending very large files as pointed out by @MidhunSureshR on the receiver's client, and a new issue has been opened #18.

@aswinshenoy
Copy link
Owner Author

Also #19 can be a potential issue faced while sending large files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant