Steps to reproduce:

1. Start playing an Amazon Prime video.
2. Hit "f" to go full screen.

Expected behavior: During the transition to or from full screen, the video should continue to be displayed.

Actual behavior: What looks like the underlying page, minus the video, is displayed. To emphasize, the video plays perfectly fine in and out of full screen mode. It is only during the transition that it is an issue.

The transition to full screen while playing video on YouTube and Max still works. It also works during the short promotional video that plays before the actual video, so maybe this is DRM related? It also works for me on Amazon Prime in Chrome.

I think this may be a regression in the last week or two, as it is rather jarring so I think I would have noticed.

This happens to me on Nightly on two MacOS machines, one x86 and one ARM.

[Tracking Requested - why for this release]: user visible regression on Amazon Prime video

Might be related to the fullscreen work we've been doing. Brad, any thoughts?

We do have different handling of DRM video that is tied to our specialized video layer. As long as the source is DRM, and the OS supports it, we should choose to use this specialized video layer. We also force the specialized media layer when the video goes fullscreen, so you could be seeing two different video layers during the transition, explaining the black video frames during the transition.

How about this? Would you please try setting the pref that forces the specialized video layer? Here's how:

1. Navigate to "about:config".
2. In the search box, type in "gfx.core-animation.specialize-video". On the line that appears, switch the value from "false" to "true".
3. Restart the browser and try to replicate the issue.

Hmmm... I'm reading more carefully comment 0 where you say that you're seeing this issue in Nightly. In Nightly, we should have the gfx.core-animation.specialize-video pref true already. The thing that's changed recently is the flip of the full-screen-api.macos-native-full-screen pref to true in Nightly. Very likely that setting that pref back to false will restore the old behavior. Trouble is, we want that pref set to true to better match expected fullscreen app behavior in macOS. I'll try to replicate with some DRM video and see what I can do to fix this.

I see this on both my ARM Mac (where the gfx.core-animation.specialize-video pref is already true) and my x86 Mac (where the pref is false). The timing of bug 1802193 landing does line up roughly when I remember starting to notice it, so it could be the full-screen-api.macos-native-full-screen pref changing.

Flipping full-screen-api.macos-native-full-screen back to false does get rid of the jarring weird frames during the transition, but I guess I just noticed that there's no smooth transition, but it blanks the entire screen (at least on x86). That's still better.

(In reply to Brad Werth [:bradwerth] from comment #3)

explaining the black video frames during the transition.

"Black video frames" sounds better than what I am seeing. It is more like the video frames are completely translucent so I can see the web page underneath. So like if I start playing a video directly from the Amazon Prime Video front page, I see the web page itself (with the thumbnails of movies I can select from) in the background. I took a photo of it on my phone that I can upload if that would help.

Yes please do upload a video of the effect so I can be sure if I am replicating it. It's very interesting that it's only happening for you on Amazon Prime. I wonder if we have a bad interaction with a codec. We have some debug logging that will output a lot of internal detail on the video source, and it would be helpful for me to see that from your machine. It's just printf-style debug info that you'll only see if you launch Nightly from the Terminal. To try it out, set the pref gfx.core-animation.specialize-video.log to true, restart Nightly from the Terminal and trigger the Bug. Then copy the text starting with VIDEO_LOG: LogSurface... from the Terminal and attach it to this Bug.

This is what it looks like during the full screen transition. Sometimes there's a big grey rectangle mixed in there.

Set release status flags based on info from the regressing bug 1802193

Thanks for the video log. It confirms that we're showing the same type of 420v video in windowed and fullscreen modes, but we're throwing the video away during the transitions and recreating the layer during the transition animation. That's no good. I'll see if I can figure out what WebRender is doing that's causing it to consider the fullscreen video to be a different source than the windowed video.

I see the same effect on YouTube. In this case the first time was actually fine, but the subsequent fullscreen enters had an empty video.

Thanks, Markus. Now that I know what to look for, I can sometimes get this effect in a transition on YouTube. My theory remains: we're creating a new WebRender texture host for the video in the fullscreen transition, and if we can keep the old one then this problem should go away.

The bug is marked as tracked for firefox116 (nightly). We have limited time to fix this, the soft freeze is in 8 days. However, the bug still has low priority and has low severity.

:bhood, could you please increase the priority and increase the severity for this tracked bug? Given that it is a regression and we know the cause, we could also simply backout the regressor. If you disagree with the tracking decision, please talk with the release managers.

For more information, please visit BugBot documentation.

I think priority and severity are set correctly for this Bug. This is a Nightly-only issue, because the pref is set to true on Nightly only.

[Tracking Requested - why for this release]: This is due to a pref that is changed only in Nightly so maybe this doesn't need to be marked as tracking.

I've confirmed that this layer rebuild happens even with bare video files with the built-in video viewer. I can see transparent bleed-through of other windows during the transition. Something about the transition to fullscreen (and back) is causing WebRender to discard the existing texture host and create another.

I'm also seeing this on Disney+, on my ARM MacBookPro (different than what I've posted the about:support for here). Instead of seeing the video image, I see a brown box.

It seems that the problem is that the WebRenderUserDataTable is set as a property on the frame, and the frame itself is destroyed when the page is restyled for fullscreen. This is a totally sensible approach for the data lifecycle, but it has the unfortunate side effect of churning WebRender for this infrequent use case. Most of the time a continuous video display is not restyled in such a way that destroys the frame. There's probably no good-enough reason to change the way this works.

So. Possible fixes:

1. Actually "fix" the issue by hanging the WebRenderUserDataTable off of the video source itself, somehow. This would require the WebRender command builder to special case its sourcing for cached data for videos. Tricky.
2. Make WebRender somehow second-guess the ImageKey it gets from Gecko, and connect new ImageKey to an existing surface. Tricky.
3. Make NativeLayerCA second-guess WebRender and not create new layers when it is asked to do so. Less tricky, but gross.
4. Make NativeLayerCA do a cleaner swap of the new layer for the old one, possibly coordinate with nsCocoaWindow to delay the fullscreen transition until one frame has been displayed in the new layer. Gross.

No great options, but options 3 and 4 have the advantage of not messing with display lists or with WebRender. I'll think about this some more.

This is also only an issue on macOS because the "specialized video layer" AVSampleBufferDisplayLayer doesn't update at the same cadence as all our other layer updates. So when we create one, it's blank/transparent and there's not much we can do about it. This typically only happens on page load or whenever a video initially appears so nobody notices because there was nothing there before. But in this case we're replacing one method of displaying a video frame with another method of displaying the same (or next) video frame and the new layer is momentarily blank. Probably this is extended during the macOS fullscreen transition since the OS window manager must snapshot the window contents before animating the transition. Thinking through this, this implies some new ways to solve the problem:

5. Whenever we create a AVSampleBufferDisplayLayer, put a same-sized layer behind it that is displaying the first frame contents. The transparent new layer will show through to what we want it to be showing anyway.
6. Try to affect the timing or contents of the snapshot that the OS takes before the fullscreen transition. The CoreAnimation API has startCustomAnimationToEnterFullScreenWithDuration to support this.

I think option 5 is the best one since it solves the general problem. I'll see if I can do this.

From testing, the problem is not limited to the use of AVSampleBufferDisplayLayer and therefore not due to the asynchronous timing of the drawing in that type of layer. The problem will also happen if the replaced layer is a CALayer both before and after. So it's really just an issue of the timing of the call to NativeLayerRootCA::SetLayers (which happens every frame, but we sometimes are able to turn into a no-op) interfering with the fullscreen transition and the snapshot it takes.

Weird. Looks like we can fix this by adding a [NSAnimationContext beginGrouping] and [NSAnimationContext endGrouping] pair to the native fullscreen transition delegate methods. In my testing, this gives us a different but maybe no worse into-fullscreen transition. It has no fade-in, but the size stability of the video image is better (no layout popping). The out-of-fullscreen transition remains largely the same. This more-or-less matches Chrome behavior, though Chrome has a secondary layout and fade-out of the menu bar -- not necessarily better. Safari has the best implementation with a smoothly animated resize plus fade in both directions. This may be due to the use of the startCustomAnimationToEnterFullScreenWithDuration API.

Anyway, I'll post a patch and we can debate whether it's an improvement.

This ensures that incidental things like layer changes that might
happen due to frame rebuilding aren't snapshotted in a weird state that
differs from what was visible when the transition started.

(In reply to Brad Werth [:bradwerth] from comment #20)

From testing, the problem is not limited to the use of AVSampleBufferDisplayLayer and therefore not due to the asynchronous timing of the drawing in that type of layer. The problem will also happen if the replaced layer is a CALayer both before and after.

Very strangely, I confirmed that we are swapping out the old CALayer and the new CALayer in the same transaction, but still the window composition manager is snapshotting a page with an empty area where the video should be. I can't explain this, which might mean that there's a flaw in my analysis. To check yourself, see what happens with the "gfx.core-animation.specialize-video" pref off on a non-DRM, non-HDR video in the base video player during the transition to fullscreen. For me, I see my desktop background.

Hi :mstange, will this be able to land for 116? Just a reminder that soft code freeze is tomorrow.

(In reply to Cieara Meador [:cmkm] from comment #24)

Hi :mstange, will this be able to land for 116? Just a reminder that soft code freeze is tomorrow.

I just want to repeat what was said in comment 14. This is an issue that is only going to be encountered in Nightly for most users, because the pref is off-by-default for Beta and Release. I don't think that our remediation needs to be closely aligned to the trains.

(In reply to Brad Werth [:bradwerth] from comment #25)

(In reply to Cieara Meador [:cmkm] from comment #24)

Hi :mstange, will this be able to land for 116? Just a reminder that soft code freeze is tomorrow.

I just want to repeat what was said in comment 14. This is an issue that is only going to be encountered in Nightly for most users, because the pref is off-by-default for Beta and Release. I don't think that our remediation needs to be closely aligned to the trains.

Thanks!

Looking at this again. Video is not necessary to demonstrate the problem. "about:blank" pages will turn transparent on the transition to fullscreen.

(In reply to Brad Werth [:bradwerth] from comment #27)

Looking at this again. Video is not necessary to demonstrate the problem. "about:blank" pages will turn transparent on the transition to fullscreen.

Which also can be replicated in both Safari and Chrome. We're encountering some kind of default behavior of the OS compositor.

This ensures that the window is opaque throughout the fullscreen
transition. It does not attempt to do any animation of the window or its
contents. Ideally, we would like to "break out" the contents of the
window and expand them to match the screen dimensions, but I haven't
been able to figure out the API to do that. Specifically, it's difficult
to get view contents to appear outside of window frame bounds, and we
don't want to actually change the window bounds just for a visual
effect. This patch stubs out the animation delegate method and includes
a TODO comment explaining what could be done to improve the visual
effect.

Comment on attachment 9340809 [details]
Bug 1839425: Stop showing layout changes during macOS native fullscreen transitions.

Do you think this is landable as-is, or do we need to go the extra step of figuring out a resizing visual transition?

Update
The video is posted as attachment 9371601 [details].

https://justbeamit.com/tjwss is a 11 second screen recording showing the corner of a Safari window displaying a YouTube video. Between the 0:04 and 0:05 mark, you can see the transition to windowed fullscreen and see that all parts of the window are transparent. The white background and the video itself are both transparent. This is what Firefox also does.

Safari does have a superior DOM fullscreen transition which is evident in YouTube.

This is a screen recording showing the problem. During the transition to windowed fullscreen on a YouTube video, the "start" window continues to play the video, but the "end" window has a blank space where the video will appear. This gives the effect of the video becoming more transparent over a white background, which is also transparent and therefore shows desktop content. This is visible in this video between 0:02 and 0:03.

To improve user experience in Firefox we need to at least ensure that the "end" window shows a video frame during the transition. The transparency issue is macOS-standard and can be ignored for now.

This is a compressed version of the video mentioned in comment 31. It shows that Safari has better behavior where it ensures the "end" window has a video frame before the transition begins.

I've confirmed that this issue disappears if we don't change the type the video layers during fullscreen transition. We only do this because of https://searchfox.org/mozilla-central/rev/c130c69b7b863d5e28ab9524b65c27c7a9507c48/gfx/layers/NativeLayerCA.mm#964 which presumes that the only upside in using AVSampleBufferDisplayLayer is if it achieves the low power mode in fullscreen. Since that code was created, we've identified other cases where we must use that layer type (HDR, DRM) and over time we'll probably want to use that layer type for video in all cases just to simplify our code paths. So one solution is to use AVSampleBufferDisplayLayer for all video (when the pref is set). I'll create a patch that does that and we can discuss.

This shows a patched Firefox that does not change the video layer types during the transition to fullscreen. Between 0:03 and 0:04 you can see that the video is present in both "before" and "after" windows for the cross-fade. As with all other examples, its also clear that the window itself is transparent and therefore parts of the desktop are visible through the window.

Prior to this patch, windowed video was presented in
a AVSampleBufferDisplayLayer only when it was detected to be HDR or DRM
video. This patch allows the use of that specialized video layer for
windowed video as long as the gfx.core-animation.specialize-video pref
is set. This might slightly increase the power consumption of windowed
video, though likely not enough to affect user experience.

https://hg.mozilla.org/mozilla-central/rev/5da1e842ef70

This issue is still happening in at least some circumstances. The patch that landed is necessary but not sufficient. I'll keep at it.

(In reply to Brad Werth [:bradwerth] from comment #39)

This issue is still happening in at least some circumstances. The patch that landed is necessary but not sufficient. I'll keep at it.

I'll keep this closed and open a new Bug to track the fixup.

I tried to reproduce the issue on Mac 12.6 (I don't have 10.15) using build from about 116.0a1(20230620094433), but all I was able to see was a white bottom of the page after pressing 'F' on both youtube and amazon prime, while switching to full screen, same issue reproduces on latest nightly 125.0a1 and beta 124.b2.
Andrew McCreight can you please confirm issue is fixed on latest Beta for you ? (https://archive.mozilla.org/pub/firefox/candidates/124.0b2-candidates/). Thank you.

I tried the beta link, and I properly saw the video image during the transition to full screen. (Also, as a side note, it looks like I was using MacOS 13.4 when I filed the bug, as that's what Darwin 22.5.0 corresponds to. I guess the UA string is fixed to be 10.15 on MacOS?)

Mark as verified on beta based on comment #42.

This is ultimately the right change to have in-tree -- all video should be routed through AVSampleBufferDisplayLayer for performance reasons -- but the regressions here are annoying and will take time to fix. We should back this out and work on a different solution to the visual problems in the windowed-to-fullscreen transition. Later, when the problems underlying the regression Bugs have been solved, we can do the changeover to AVSampleBufferDisplayLayer in a different, dedicated Bug.

This will be backed out from Nightly and Beta.

Backed out from Beta for 125.0b9 along with bug 1877153.
https://hg.mozilla.org/releases/mozilla-beta/rev/1704ba05b8d441100a2f8dbd172eb4c2f7e039ec

Backout merged in central: https://hg.mozilla.org/mozilla-central/rev/fff626bfbf352066d8d96f072d3e3d165f243a71.