[aquamarine] try to fix direct scanout

[ikalco]

Describe your PR, what does it fix/add?

will try to fix direct scanout and linux-dmabuf protocol for #6608

Is there anything you want to mention? (unchecked code, possible bugs, found problems, breaking compatibility, etc.)

Is it ready for merging, or does it need work?

• use monitor primary plane formats for ds (add getting monitor render formats aquamarine#13)
• support monitor connect/disconnect for format_table (handle monitor hotplugs and fix a few bugs that it revealed aquamarine#14)
• fix weird bug that disables ds randomly
• fix weird artifacts that happen randomly when ds ¹ (thanks vaxry!!!)
• only send formats than can fallback to renderer even in scanout tranches
  add renderer fallback tranche when attempting ds ²
• maybe use sets in format table to conserve memory, would be hard to keep track of indicies tho
• maybe create seperate format_tables to mitigate bad clients ³

Footnotes

1. after reading some of weston's code this might be related to plane damaging (drm.c@470)? this is a guess so far tho ↩

2. https://gitlab.freedesktop.org/wayland/wayland-protocols/-/blob/main/stable/linux-dmabuf/feedback.rst

   Compositors should not send feedback parameters if they don't have a fallback path. For instance, compositors shouldn't send a format/modifier supported for direct scan-out but not supported by the rendering API for texturing.

   after some more reading the ds tranche is allowed to have formats that the egl doesnt support as long as there is a fallback tranche that the renderer does support ↩

3. the format_table contains all the formats from egl and planes but some wl clients only use format_table to find formats instead of tranches
   only "bad" client i could find was mpv so I just bug reported there ↩

[ikalco]

Found an interesting repro case for the random artifacts during ds, xonotic
if run with SDL_VIDEODRIVER=x11 xonotic-sdl there are some arifacts
but run with SDL_VIDEODRIVER=wayland xonotic-sdl there is major artifacting, looks related damaging

edit:
for wayland the anti aliasing option almost completely removed artifacting, with vsync option on aswell its gone
for x11 just enabling vsync completely removes artifacting

[ikalco]

another repro case for artifacting
https://github.com/ascent12/compositor-killer.git

edit:
this patch.txt makes it stand out more, and it kind of looks like its something the client is rendering??? idfk

[ikalco]

gamescope has a bug report that looks a lot like the artifacting
ValveSoftware/gamescope#1368

edit:
can repro that lowering frame rate in game (xonotic) to below vsync removed all artifacts

[vaxerski]

so is that a kernel bug?

[ikalco]

so is that a kernel bug?

not entirely sure, it supposedly works on nvidia and igpu without artifacting but amd gets artifacting
but then again amd doesnt get artifacting on sway so maybe they have a mitigation related to below

edit:
can repro that lowering frame rate in game (xonotic) to below vsync removed all artifacts

not sure how to test how many fps the client is actually rendering to test if this is true though

edit:
the gamescope bug also mentions that completely disabling explicit sync fixed the issue
i tried doing it through config and nothing changed, i might try testing ripping out all the explicit sync suff just to make sure tho

[Agent00Ming]

recorded on this branch, intel iGPU

direct_scanout_tearing.mp4

[ikalco]

After a lot of testing the artifacts dont seem to be a kernel/driver/mesa bug
its definitely related to vsync, the more accurate the fps is to the refresh rate the less the artifacts
I've tested a few games and enabling vsync will make most artifacts go away

PS:
I dont understand enough about how Hyprland maintains fps and rr to be able to fix the artifacting
how is fps and rr maintained by Hyprland? is page flip event called by driver exactly right to line up with refresh rate and therefore renderMonitor?
either way I think something in attemptDirectScanout isn't the same as the rest of renderMonitor in relation to vsync/fps/rr which causes the bug

[vaxerski]

@phonetic112 in 6608 said the glitches go away on legacy drm api, perhaps something there is off?

how is fps and rr maintained by Hyprland?

render ---> submit with flip event

submit submits a frame to be presented on next vblank (monitor changes image = vblank)
then, when that vblank happens and our image is presented, we get an event (see handlePF in aq) in which (if applicable) we send a .frame event to signify its a good time to render the next frame. If there is no damage, and a frame isnt needed, we dont. This stops the cycle and will resume it the next time something schedules a frame (e.g. damage)

[vaxerski]

wild guess, do the glitches happen without hardware cursors but when the cursor isnt on the screen?

(if that doesnt allow DS then just patch it out, probably in attemptDirectScanout)

[ikalco]

no disabling hw cursors doesnt fix artifacting

btw to get artifacting to go away i used vrrtest, vsync on, fps cap 1 fps above vsync, and busy wait on (busy wait says it more accurately tracks vsync)
you'll also see that the closer you get to vsync the artifacting will slow down

edit:
legacy drm doesn't seem to fix artifacting for me :(

[ikalco]

@phonetic112 in 6608 said the glitches go away on legacy drm api, perhaps something there is off?

how is fps and rr maintained by Hyprland?

render ---> submit with flip event

submit submits a frame to be presented on next vblank (monitor changes image = vblank) then, when that vblank happens and our image is presented, we get an event (see handlePF in aq) in which (if applicable) we send a .frame event to signify its a good time to render the next frame. If there is no damage, and a frame isnt needed, we dont. This stops the cycle and will resume it the next time something schedules a frame (e.g. damage)

wait, so how do we make sure we dont schedule more than, lets say, 60 fps?

[vaxerski]

backend will err out if you commit a new buffer before handlePF

[ikalco]

last commit was able to make artifacting only appear when vrrtest frame cap is above monitor refresh rate except when starting it up when it happens for a few frames

also arch ci seems to be broken, something with updating packages

[ikalco]

the wlroots ds pr mentions holding client buffers longer

We need to hold client buffers for longer than usual. When compositing, we need to hold buffers until the buffer swap (and the kernel will take care of the rest with implicit synchronization). When scanning out, we need to hold buffers as long as they're visible on screen.

didn't have time to check this out but if we arent then that could be causing artifacts

[vaxerski]

heh I did once make a patch to do that but it didnt stop the glitches, I can do it again

[vaxerski]

committed the new infra impl here, feel free to drop it if it doesnt work

[ikalco]

committed the new infra impl here, feel free to drop it if it doesnt work

it worked!!! artifacting is now completely gone on AMD, should be gtg other than testing on nvidia and igpu

[Agent00Ming]

works on nvidia but something is going wrong with the cursor hanging the whole desktop nvm, the cursor simply does not cause any damage to happen

[ikalco]

yeah i just saw that, i think i might know that it is

edit:
try now @Agent00Ming

btw if you get a crash related to sendReleaseWithSurface i think i found the bug

[Agent00Ming]

Yeah, works on nvidia. Occasionally spams entered/left direct scanout in logs when initially starting a fullscreen window or when system is under heavy load when starting the direct scanout but that's a non-issue and stabilizes in a few seconds. Will test intel in a moment.

Unsurprisingly, it also works on intel 👍

[vaxerski]

ready to merge then? or no

[ikalco]

yes this is ready

[vaxerski]

thanks!