Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add gzip to the protocol #88

Closed
blahah opened this issue Apr 13, 2017 · 9 comments
Closed

Add gzip to the protocol #88

blahah opened this issue Apr 13, 2017 · 9 comments

Comments

@blahah
Copy link

blahah commented Apr 13, 2017

@mafintosh and I were looking at why some hyperdrive feeds were taking a long time to sync metadata, and found that there's a lot of redundancy in the metadata. @mafintosh gzipped it and reduced ~6MB to a few hundred KB. The conclusion was that, whatever other optimisations might be added to the indexing etc., a simple optimisation that will help in almost all cases is to gzip everything.

@mafintosh - are we talking about gzipping on the fly during transport, or gzipping the metadata on disk?
@mafintosh
Copy link
Contributor

Just transport for now. Disk is cheap.

@blahah
Copy link
Author

blahah commented Apr 13, 2017 •
edited
Loading

Recording discussion from IRC:

  • @mafintosh suggested trying other compression algorithms - e.g. snappy
  • @yoshuawuyts suggested looking at brotli
  • we probably want to optimise for slower connections, since that's where large metadata will impact the user experience most
  • that likely means preferring a higher compression ratio over faster compression speed
  • benchmark all of this

@emilbayes
Copy link
Contributor

Just throwing zstd in the mix :)

@blahah
Copy link
Author

blahah commented Apr 13, 2017

@emilbayes nice

Also looking at:

@blahah
Copy link
Author

blahah commented Apr 13, 2017

Here's a pair of hyperdrive v8 keys that can be used to test: sciencefair-land/sciencefair-elife#1

One is ~3.6k small JSON files, the other is ~3.6k academic papers in JATS xml format with figures in jpg.

@blahah
Copy link
Author

blahah commented Apr 13, 2017

@mafintosh some v8 feeds that have flat dir structure:

  • json 538ff422e03f2b748421a347cb547b9d547297cdecfb267548c0ae32f907bdf7
  • articles 4e3f9a83f8b5bdd193b0c805196fa8db9878ef5efcfafced86fbbd7013d32585

@blahah
Copy link
Author

blahah commented Apr 30, 2017 •
edited
Loading

@mafintosh you said compression is now added to append-tree right?

does that make this obsolete?

@joehand
Copy link
Contributor

joehand commented Jul 11, 2017

To update: this hasn't been added anywhere yet. Bigger priorities like multiwriter =).

mafintosh pushed a commit that referenced this issue Aug 15, 2022
@chm-diederichs @kasperisager
Introduce general auth abstraction (#88)
f9612bc 
* implement pluggable auth and add multisig test

* add test using hypercore extension

* correct multisig extension test

* add proof-of-work test

* support passing opts.sign for backwards compat

* pass auth instance to core methods

* uint8array friendly tests

* add test for custom sign backwards compat

* typo in test

* use b4a instead of Buffer

* missing require in test

* add instance example to test

* use b4a.alloc in test

Co-authored-by: Kasper Isager Dalsgarð <[email protected]>

* ensure writable is a boolean

Co-authored-by: Kasper Isager Dalsgarð <[email protected]>
@mafintosh
Copy link
Contributor

Still interesting! Join the discord if y'all wanna discuss further.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@mafintosh @emilbayes @joehand @blahah