Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve metadata storage #37

Merged
merged 6 commits into from
Feb 24, 2024
Merged

Improve metadata storage #37

merged 6 commits into from
Feb 24, 2024

Conversation

kieraneglin
Copy link
Owner

What's new?

  • Adds post-download step to fetch thumbnails for our app's purposes
  • Adds a module for basic HTTP calls

What's changed?

  • Removes metadata from the DB itself, instead compressing and storing on-disk. This results in a 10-20x space reduction (~700KB -> ~38KB) per record
  • Shuffles around storage directories so (hopefully) disambiguate the metadata directory that stores critical files and the temporary metadata stored by yt-dlp calls

What's fixed?

N/A

Any other comments?

Although not a huge issue now, the metadata storage was going to be an issue in the future and it's something I REALLY want to fix before releasing since this would be a pain to retcon. In the worst case, the metadata could take up nearly 10% the space of a short which is not acceptable. I was also storing this in the DB which is nice for querying, but I don't know how SQLite handles that kind of storage as it scales up. I just feel better putting it on-disk.

@kieraneglin kieraneglin self-assigned this Feb 24, 2024
@kieraneglin kieraneglin merged commit 52c0a17 into master Feb 24, 2024
1 check passed
@kieraneglin kieraneglin deleted the ke/improve-metadata-storage branch February 24, 2024 21:40
kieraneglin added a commit that referenced this pull request Mar 29, 2024
* Updated config path to specify metadata storage path

* Removed metadata column from DB, replacing it with filepath columns

* Updated app to store compressed metadata; automatically download thumbnails

* Ensured metadata is deleted when other files are deleted

* Updated docs

* Added inets to application start so http calls work in prod
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant