Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Triage] Restarts break ongoing initial indexings #573

Open
krafs opened this issue Jan 16, 2025 · 0 comments
Open

[Triage] Restarts break ongoing initial indexings #573

krafs opened this issue Jan 16, 2025 · 0 comments
Assignees
Labels

Comments

@krafs
Copy link

krafs commented Jan 16, 2025

Describe the bug
Restarting Pinchflat breaks ongoing initial indexings of large sources.

To Reproduce

  1. Add new channel source with more than 20 videos, e.g. 30.
  2. Wait until 20 videos have been indexed.
  3. Restart Pinchflat.
  4. See that channel indexing restarts but stops after 20 videos.

Expected behavior
Restarting Pinchflat resumes ongoing initial indexings and fully indexes all videos in the source.

Notes
Firstly, I understand this is likely due to the indexing change in #540, and that Pinchflat can't tell the difference between an initial slow indexing and an additional indexing. The additional indexing greatly benefits from stopping after seeing no changes after 20 videos, but someone trying to archive an entire channel presumably wants indexings to complete fully even after a service restart.

Secondly, I personally don't want this bug fixed yet, because it currently exposes a workaround for indexing large sources quickly.
Adding a large channel takes days to fully index the first time (e.g. LinusTechTips has >7000 videos). But with the workaround I can add the source, wait until it has indexed 20 videos, restart Pinchflat and let the indexing resume and finish after re-indexing the first 20 videos. After this the source is fast-indexed regularly.

I've got 50+ sources, most of which have 1000+ videos, and this workaround just saved me weeks of waiting before Pinchflat became usable. Maybe I missed some obvious setting or official way to onboard quickly, but this is what I ended up doing.

For the bug - I think it should be fixed so people archiving entire channels don't miss videos due to restarting Pinchflat (e.g. by updates via WatchTower). But perhaps we can wait until we have some way to do fast initial indexings?

Some ideas off the top of my head:

  • Add option to only index back to cutoff date.
  • Add option to only ever do fast indexing, never slow.

What do you think?

Diagnostic info

  • App Version: 2025.1.14
  • yt-dlp Version: 2025.01.12
  • Apprise Version: v1.9.2
  • System Architecture: x86_64-pc-linux-gnu
  • Timezone: Europe/Stockholm
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants