Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Streaming media item creation during indexing #49

Merged
merged 3 commits into from
Mar 4, 2024

Conversation

kieraneglin
Copy link
Owner

@kieraneglin kieraneglin commented Mar 4, 2024

What's new?

  • Adds behaviour to indexing that listens to the file yt-dlp writes to and creates MediaItem records as they come in
    • This HUGELY improves the indexing process since it used to run indexing -> creation -> downloading in serial
  • Adds behaviour to the above to enqueue download of media items post-index
  • Adds FileSystemUtils module with generate_metadata_tmpfile method to generate a blank tmpfile of a given type
  • Adds a FileFollowerServer method for listening to file changes and performing actions on just the changes
  • Adds Media.pending_download? to determine if an individual MediaItem should be downloaded

What's changed?

  • Updates command runner to optionally take a filepath of the file it'll write its output to
  • Removes command_opts arg from VideoCollection.get_media_attributes since it was unused
  • Renames index_media_items in SourceTasks to index_and_enqueue_download_for_media_items to better reflect its purpose
  • Updates index_and_enqueue_download_for_media_items to now kickoff downloads for pending media itself without needing to call enqueue_pending_media_tasks specifically

What's fixed?

N/A

Any other comments?

N/A

@kieraneglin kieraneglin added documentation Improvements or additions to documentation enhancement New feature or request media downloads refactor media indexing labels Mar 4, 2024
@kieraneglin kieraneglin self-assigned this Mar 4, 2024
@kieraneglin kieraneglin merged commit 63e5058 into master Mar 4, 2024
1 check passed
@kieraneglin kieraneglin deleted the ke/stream-indexing branch March 4, 2024 18:14
kieraneglin added a commit that referenced this pull request Mar 29, 2024
* Implemented streaming during indexing

* Updated file watcher to enqueue download; refactored download worker methods

* Updated File Follower Server timeout
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation enhancement New feature or request media downloads media indexing refactor
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant