[Triage] YouTube returning a 403 (forbidden)

[mayoko185]

Describe the bug
I recently set up pinchflat last week and everything was working great, yesterday I went to add a new channel and now all videos even those ive already downloaded have the title youtube video #randomstuff

To Reproduce
Ive restarted the docker container, also tried to start a manual index/refresh metadata.

Expected behavior
Should have the current videos title

Screenshots

Diagnostic info

• App Version: 2025.1.3
• yt-dlp Version: 2024.12.23
• Apprise Version: v1.9.1
• System Architecture: x86_64-pc-linux-gnu
• Timezone: America/New_York

Additional context

pinchflat-logs-2025-01-05.txt

Thanks for any help, loving the app so far its perfect for what I need it for (archiving videos to watch on emby).

[kieraneglin]

Hey there! Thanks for the report (:

This is what happens when yt-dlp can't extract any data from the video it's indexing/downloading. This is, unfortunately, something that's out of my hands and usually happens when YouTube introduces breaking changes to the way yt-dlp works. YouTube always slowly rolls out these changes and in this case it seems like you were one of the (un)lucky first recipients. The good news is that the yt-dlp team is typically extremely quick to roll out a fix for these kinds of issues 🤞

I'll keep this issue open for a few days just to remind myself to check back in on any update from the yt-dlp team

[mayoko185]

Ahhh good old cat and mouse situation, thanks for taking a look.

[Shayviri]

Hi, Sorry to step in like that, I have some questions about this.

Let's imagine there is a problem like that and YouTube make some breaking change. yt-dlp is not updated yet, and pinchflat index the data as they come from yt-dlp. Some data will be wrong because of the breaking changes.

Does this metadata be updated later when the yt-dlp is updated? If yes, does it occur during the indexing? Do we need to force refresh metadata?
What about the videos downloaded during the breaking changes? If the titles were wrong, I guess the files name might be wrong if we use the title name and the metadata aside will be wrong too. What can we do about this? Do we need to redownload everything?

It is just questions to know how to handle it when this happens.

[aklferris]

I, too, am getting the same issue.

[kieraneglin]

Hey all! I've spent some more time looking into this and it seems like this is a widespread issue for some people and yt-dlp. For some people using cookies helped and for others changing their residential IP helped. The root cause seems yet unknown but I'll see if I can push out some changes today to change how rate limiting works in-app

@Shayviri good questions! The next successful index should update the in-app metadata. As for filenames, those will be screwed up until I come up with a solution to #408. I've been working on that issue recently, but it's harder than I expected to get everything to work in a sane way. In the meantime, if your filename is screwed up the best option is to delete the media item in-app and re-download it

[chaosbib]

Thank God I'm not the only one. Another thing that is seems related to this issue is if you're set to download music (adding metadata to the files), it only downloads the album cover (jpg).

[Maratzz]

First time installing Pinchflat, and I stumbled upon this as well :') I also got the "sign in to confirm you're not a bot" error which prevents me from downloading the videos, they're all pending with the # identifier in the meantime. I tried to use cookies as specified in your link, but it's not working.

[kieraneglin]

I've been looking into this a lot and unfortunately it doesn't seem like anything can be done to mitigate these errors outright. What I'm working on currently is getting a rate limiting system in place (which is proving to be harder than I had expected to get it just how I want) and to improve the app's behaviour to these kinds of errors

[kieraneglin]

I have an attempted fix for this in #559. I really recommend reading through that but here's the TL;DR:

• Adds rate limiting through yt-dlp's built in mechanisms
• Prevents the "Sign in to confirm..." error from retrying the download. That download will be dropped until the next indexing pass
• Stops indexing from creating/updating media items if the app detects it's being throttled

These changes haven't been released yet since I need to do some testing. If you want to help me test, here's how:

• Switch your Docker tag from latest to issue-549
• In the app, go to the settings page and set your Sleep Interval to some non-zero value. I don't know the optimal value here so I'd start with ~5s and go up or down from there. If you find a value that works for you, please share it!

Thank you!

[maulwurf87]

Hi,
I will switch to the Issue Tag and test it with you. Is there any data that would be valueable for you that I can collect?

[kieraneglin]

Thank you! I'm mostly looking for two things:

• Can you download media without getting throttled? These changes won't actually change anything if YouTube is throttling your IP from the offset so what I am actually interested in is the transition from "can download" to "can't download"
• (less important) If you are currently throttled, confirmation that new media items aren't created or updated would be nice! It won't do anything for past media items until it can start scanning without error again, but preventing creation of new media items is a win

[maulwurf87]

Hi,
yes, I am throttled at the moment.

2025-01-11 22:45:56.236 [error] | Error creating media item 'TaPbMCe3iBM' from URL: {:error, "ERROR: [youtube] TaPbMCe3iBM: Sign in to confirm you’re not a bot. Use --cookies-from-browser or --cookies for the authentication. See  https://github.com/yt-dlp/yt-dlp/wiki/FAQ#how-do-i-pass-

And yes, new media is not created. I have a new source, that still has 0 pending media and 0 downloaded media because it cannot even scan the channel.
I'm running pinchflat connected to a tailscale VPN. If I can find the time tomorrow I will try to use one of my servers as an exit node to get a fresh IP and see how long it will last until youtube is throttling again.

[Maratzz]

Hello,
I also switched to the issue-549 tag on a clean new install, and I have the same results as @maulwurf87 just above, 0 pending and 0 downloaded. I can see in the log that I have one "sign in" error though, instead of several.

13:19:33.083 [debug] [command_wrapper]: /usr/local/bin/yt-dlp called with: https://www.youtube.com/@HomoLudensjv --simulate --skip-download --ignore-no-formats-error --playlist-end 1 --print-to-file %(.{channel,channel_id,playlist_id,playlist_title,filename})j /tmp/pinchflat/data/97/b5/97b5c724a6d3afdd262b518e8b6e3663d0f49e3fe7e58ba0b75128a159ae6b69.json --windows-filenames --quiet --cache-dir /tmp/pinchflat/data/yt-dlp-cache exited: 0 with: WARNING: [youtube] Sign in to confirm you’re not a bot. Use --cookies-from-browser or --cookies for the authentication. See  https://github.com/yt-dlp/yt-dlp/wiki/FAQ#how-do-i-pass-cookies-to-yt-dlp  for how to manually pass cookies. Also see  https://github.com/yt-dlp/yt-dlp/wiki/Extractors#exporting-youtube-cookies  for tips on effectively exporting YouTube cookies
WARNING: No video formats found!
WARNING: Requested format is not available

[maulwurf87]

Hi,
still the cookie banner from Youtube. But there was also no wrongly named Indexes or videos. I found the time today to use one of my servers as an exit node and get a fresh IP. Now everything seems to be working correctly.
I also added a Youtube API key and activated fast indexing for all sources to offload the requests to some form of official Endpoint. Let's see how long this will work.

I noticed to channels being wrongly identified as playlists but I believe at least one was present before the change to the Issue Image, so that seems to be another problem. I try to open an issue for that in the next days.

Until now I would call this a success.

[SenorFusion]

@kieraneglin Just dropping in to say the PipePipe dev (fork of NewPipe) claims to have solved this issue.

The way his repos are strangely organized I haven't been able to track down the exact commits or understand what he did, but here is the relevant discussion: InfinityLoop1308/PipePipe#613

Maybe this isn't relevant because its single video / playback speed and not mass downloading, but he is the first person to claimed to have really solved it for now - yt-dlp devs still don't seem to have a solution or even understand the problem.

[kieraneglin]

Thanks for that link! This may also be resolved by the latest yt-dlp - I'm running some tests right now and I'll whip up a release things look good!

If anyone wants to test, you can update yt-dlp with these instructions

[jhotmann]

Updating yt-dlp alone on the stable docker tag worked for me

[kieraneglin]

I've pushed up a new release that includes an update to yt-dlp which should be live within 30 mins. Please let me know if that fixes things!

[ciavland]

@kieraneglin For me downloads work again. Thanks a lot!

[krog]

When using cookies there is still a problem with yt-dlp 2025-01-12 as it is using web_creator that seems to be require PO_TOKEN now.

I was able to download using the following in config/extras/yt-dlp-configs/base-config.txt
--extractor-args "youtube:player_client=default,-web_creator"

If I read the changes correctly this might be fixed in yt-dlp 2025-01-15

[maulwurf87]

Looks like I'm still getting 403:

2025-01-16 15:04:14.686 [error] | [command_wrapper]: /usr/local/bin/yt-dlp called with: https://www.youtube.com/watch?v=6fSzay-pFMA --no-simulate --no-progress --no-force-overwrites --parse-metadata %(upload_date>%Y-%m-%d)s:(?P<meta_date>.+) --write-subs --convert-subs srt --sub-langs de --write-thumbnail --convert-thumbnail jpg --output thumbnail:/downloads/shows/DieMaus/%(upload_date>%Y-%m-%d)S_Season %(upload_date>%Y)S/s%(upload_date>%Y)Se%(upload_date>%m%d)S99 - %(title)S-thumb.%(ext)S --write-info-json --clean-info-json --remux-video mp4 --format-sort res:2160,+codec:avc:m4a --format bestvideo*+bestaudio/best --sponsorblock-mark sponsor,intro,outro,selfpromo,preview,filler,interaction --output /downloads/shows/DieMaus/%(upload_date>%Y-%m-%d)S_Season %(upload_date>%Y)S/s%(upload_date>%Y)Se%(upload_date>%m%d)S99 - %(title)S.%(ext)S --print-to-file after_move:%()j /tmp/pinchflat/data/2f/74/2f74319ccbdc118b22fd3495857aeeafd5faa6b551285e4862801b0660888f01.json --sleep-requests 7 --sleep-interval 7 --sleep-subtitles 8 --windows-filenames --quiet --cache-dir /tmp/pinchflat/data/yt-dlp-cache exited: 1 with: ERROR: unable to download video data: HTTP Error 403: Forbidden

I'm running v2025.1.14

[kieraneglin]

Looks like yt-dlp just dropped another update - could y'all try updating yt-dlp and letting me know if that resolves it?

I won't be able to test for at least a few hours so any help is appreciated 🤙

BTW I'm going to be adding in a way to auto-update yt-dlp on container start (or maybe just add an update button in the UI), but I haven't gotten a chance to get to that yet

[Gliptopolis]

I'm not sure how to update just the yt-dlp in an unraid docker.

[kieraneglin]

@Gliptopolis I've included instructions linked in my last post - it's at the very bottom of that page. You open a console by clicking on Pinchflat in the Unraid Docker page and selecting "Console"

[Gliptopolis]

I tried that via the console option on the docker, but the commands aren't recognized. I know I'm doing it all wrong and I'm actively searching for the proper way to do it. I'm new to both docker and unraid.

[Gliptopolis]

@kieraneglin You're busy with updates, don't worry about me. I'll eventually get it or you'll have an update I can install. Thanks for responding.

[memaxwelll]

At least for me I am still seeing errors on age restricted videos.
Last time a download was successful was the 14th of January.
Let me know if there is anything I can help with (logs etc.)

[alexindigo]

Updating yt-dlp works.