[Cocoa] Adopt VTDecompressionSessionDecodeFrameWithMultiImageCapableOutputHandler to decode stereo images #47186

jyavenard · 2025-06-25T15:24:45Z

`c0688ab`

[Cocoa] Adopt VTDecompressionSessionDecodeFrameWithMultiImageCapableOutputHandler to decode stereo images
https://bugs.webkit.org/show_bug.cgi?id=295143
rdar://154516073

Reviewed by Jer Noble.

In 296748@main we fixed a regression caused by 295444@main. The change was
a simple temporary workaround to disable MediaSourcePrefersDecompressionSession
for spatial or immersive videos as WebCoreDecompressionSession didn't
decode stereo images. We now use the VTDecompressionSessionDecodeFrameWithMultiImageCapableOutputHandler
method instead which can handle both images in CMSampleBuffer to decode.

We add support for decoding multi-images (stereo) samples and readback of
such images.

As such, we can revert 296748@main which is no longer needed.

We make WebCoreDecompression's WorkQueue a singleton. It simplifies the code
a bit and this workqueue carries very little load as all the object it uses already
use their own workqueues.

Spatial portal manually tested on visionOS.
Added test to verify decoding and readback when playing a spatial video in stereo mode.

* LayoutTests/media/media-source/content/test-spatial-manifest.json: Added.
* LayoutTests/media/media-source/content/test-spatial.mp4: Added.
* LayoutTests/media/media-source/media-source-paint-stereo-to-canvas-expected.txt: Added.
* LayoutTests/media/media-source/media-source-paint-stereo-to-canvas.html: Added.
* Source/WebCore/PAL/pal/cf/CoreMediaSoftLink.cpp: Adding dynamically linked methods
* Source/WebCore/PAL/pal/cf/CoreMediaSoftLink.h:
* Source/WebCore/platform/cocoa/VideoToolboxSoftLink.cpp: Adding dynamically linked methods
* Source/WebCore/platform/cocoa/VideoToolboxSoftLink.h:
* Source/WebCore/platform/graphics/avfoundation/FormatDescriptionUtilities.cpp:
(WebCore::spatialVideoMetadataFromFormatDescription): Remove canLoad method for symbols that will always be found.
* Source/WebCore/platform/graphics/cocoa/VideoMediaSampleRenderer.mm:
(WebCore::VideoMediaSampleRenderer::enqueueSample): Revert 296748@main as it's no longer needed.
(WebCore::VideoMediaSampleRenderer::decodeNextSampleIfNeeded):
(WebCore::VideoMediaSampleRenderer::imageForSample const): Handle multi-images sample
(WebCore::VideoMediaSampleRenderer::assignResourceOwner): Amend to assign all multi-images sample's IOSurfaces

Canonical link: https://commits.webkit.org/296854@main

96af1f0

Misc	iOS, visionOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	~~🛠 wpe~~	~~🛠 win~~
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	~~🧪 wpe-wk2~~	~~🧪 win-tests~~
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	✅ 🧪 api-mac	~~🧪 api-wpe~~
	✅ 🧪 ios-wk2-wpt	✅ 🧪 mac-wk1	✅ 🛠 wpe-cairo
	✅ 🧪 api-ios	✅ 🧪 mac-wk2	~~🛠 gtk~~
	✅ 🛠 vision	✅ 🧪 mac-AS-debug-wk2	~~🧪 gtk-wk2~~
	✅ 🛠 vision-sim	✅ 🧪 mac-wk2-stress	~~🧪 api-gtk~~
✅ 🛠 🧪 merge	✅ 🧪 vision-wk2	✅ 🧪 mac-intel-wk2	✅ 🛠 playstation
	✅ 🛠 tv	✅ 🛠 mac-safer-cpp
	✅ 🛠 tv-sim
	✅ 🛠 watch
	✅ 🛠 watch-sim

webkit-early-warning-system · 2025-06-25T15:24:51Z

EWS run on previous version of this PR (hash 5e4a146)

Misc	iOS, visionOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	✅ 🛠 wpe	✅ 🛠 win
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	✅ 🧪 wpe-wk2	⏳ 🧪 win-tests
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	✅ 🧪 api-mac	✅ 🧪 api-wpe
	✅ 🧪 ios-wk2-wpt	✅ 🧪 mac-wk1	⏳ 🛠 wpe-cairo
	✅ 🧪 api-ios	✅ 🧪 mac-wk2	✅ 🛠 gtk
	✅ 🛠 vision	✅ 🧪 mac-AS-debug-wk2	✅ 🧪 gtk-wk2
	✅ 🛠 vision-sim	✅ 🧪 mac-wk2-stress	✅ 🧪 api-gtk
	✅ 🧪 vision-wk2	✅ 🧪 mac-intel-wk2	✅ 🛠 playstation
	✅ 🛠 tv	❌ 🛠 mac-safer-cpp
	✅ 🛠 tv-sim
	✅ 🛠 watch
	✅ 🛠 watch-sim

webkit-ews-buildbot · 2025-06-25T18:19:55Z

Safer C++ Build #41549 (`5e4a146`)

❌ Found 1 failing file with 1 issue. Please address these issues before landing. See WebKit Guidelines for Safer C++ Programming.
(cc @rniwa)

webkit-early-warning-system · 2025-06-25T23:17:10Z

EWS run on previous version of this PR (hash 6eb05b2)

Misc	iOS, visionOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	✅ 🛠 wpe	✅ 🛠 win
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	✅ 🧪 wpe-wk2	⏳ 🧪 win-tests
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	✅ 🧪 api-mac	✅ 🧪 api-wpe
	✅ 🧪 ios-wk2-wpt	✅ 🧪 mac-wk1	⏳ 🛠 wpe-cairo
	✅ 🧪 api-ios	✅ 🧪 mac-wk2	✅ 🛠 gtk
	✅ 🛠 vision	~~🧪 mac-AS-debug-wk2~~	✅ 🧪 gtk-wk2
	✅ 🛠 vision-sim	✅ 🧪 mac-wk2-stress	✅ 🧪 api-gtk
	✅ 🧪 vision-wk2	✅ 🧪 mac-intel-wk2	⏳ 🛠 playstation
	✅ 🛠 tv	❌ 🛠 mac-safer-cpp
	✅ 🛠 tv-sim
	✅ 🛠 watch
	✅ 🛠 watch-sim

webkit-early-warning-system · 2025-06-26T02:15:36Z

EWS run on previous version of this PR (hash 701b299)

Misc	iOS, visionOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	✅ 🛠 wpe	⏳ 🛠 win
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	~~🧪 wpe-wk2~~	⏳ 🧪 win-tests
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	~~🧪 api-mac~~	✅ 🧪 api-wpe
	⏳ 🧪 ios-wk2-wpt	~~🧪 mac-wk1~~	⏳ 🛠 wpe-cairo
⏳ 🛠 🧪 jsc	✅ 🧪 api-ios	~~🧪 mac-wk2~~	✅ 🛠 gtk
	✅ 🛠 vision	⏳ 🧪 mac-AS-debug-wk2	~~🧪 gtk-wk2~~
	✅ 🛠 vision-sim	~~🧪 mac-wk2-stress~~	✅ 🧪 api-gtk
	✅ 🧪 vision-wk2	~~🧪 mac-intel-wk2~~	⏳ 🛠 playstation
	✅ 🛠 tv	⏳ 🛠 mac-safer-cpp
	✅ 🛠 tv-sim
	⏳ 🛠 watch
	✅ 🛠 watch-sim

webkit-early-warning-system · 2025-06-26T02:37:53Z

EWS run on previous version of this PR (hash 16bafbc)

Misc	iOS, visionOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	❌ 🛠 ios	~~🛠 mac~~	~~🛠 wpe~~	⏳ 🛠 win
✅ 🧪 bindings	❌ 🛠 ios-sim	~~🛠 mac-AS-debug~~	~~🧪 wpe-wk2~~	⏳ 🧪 win-tests
✅ 🧪 webkitperl	❌ 🧪 ios-wk2	~~🧪 api-mac~~	~~🧪 api-wpe~~
	❌ 🧪 ios-wk2-wpt	~~🧪 mac-wk1~~	⏳ 🛠 wpe-cairo
⏳ 🛠 🧪 jsc	❌ 🧪 api-ios	~~🧪 mac-wk2~~	~~🛠 gtk~~
	❌ 🛠 vision	~~🧪 mac-AS-debug-wk2~~	~~🧪 gtk-wk2~~
	~~🛠 vision-sim~~	~~🧪 mac-wk2-stress~~	~~🧪 api-gtk~~
	⏳ 🧪 vision-wk2	~~🧪 mac-intel-wk2~~	⏳ 🛠 playstation
	❌ 🛠 tv	⏳ 🛠 mac-safer-cpp
	❌ 🛠 tv-sim
	~~🛠 watch~~
	❌ 🛠 watch-sim

webkit-early-warning-system · 2025-06-26T02:40:28Z

EWS run on previous version of this PR (hash c655439)

Misc	iOS, visionOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	✅ 🛠 wpe	✅ 🛠 win
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	✅ 🧪 wpe-wk2	⏳ 🧪 win-tests
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	✅ 🧪 api-mac	✅ 🧪 api-wpe
	✅ 🧪 ios-wk2-wpt	✅ 🧪 mac-wk1	✅ 🛠 wpe-cairo
	✅ 🧪 api-ios	❌ 🧪 mac-wk2	✅ 🛠 gtk
	✅ 🛠 vision	❌ 🧪 mac-AS-debug-wk2	✅ 🧪 gtk-wk2
	✅ 🛠 vision-sim	✅ 🧪 mac-wk2-stress	❌ 🧪 api-gtk
	✅ 🧪 vision-wk2	✅ 🧪 mac-intel-wk2	✅ 🛠 playstation
	✅ 🛠 tv	✅ 🛠 mac-safer-cpp
	✅ 🛠 tv-sim
	✅ 🛠 watch
	✅ 🛠 watch-sim

webkit-ews-buildbot · 2025-06-26T03:22:03Z

Safer C++ Build #41610 (`6eb05b2`)

❌ Found 1 failing file with 1 issue. Please address these issues before landing. See WebKit Guidelines for Safer C++ Programming.
(cc @rniwa)

jernoble

r=me with nit

jernoble · 2025-06-26T16:02:00Z

Source/WebCore/platform/graphics/cocoa/WebCoreDecompressionSession.mm

+#if PLATFORM(VISION)
+            m_stereoViewMap.clear();
+            RetainPtr<CFMutableArrayRef> layersArray = nullptr;
+            // Get the tag collection array
+            CFArrayRef rawTagCollectionArray = nullptr;
+            if (auto status = PAL::CMVideoFormatDescriptionCopyTagCollectionArray(videoFormatDescription, &rawTagCollectionArray); status == noErr && rawTagCollectionArray) {


Nit: this is a pretty deep nested if/for/if/if section. Consider refactoring to use continue or early return to flatten and make the resulting code more readable.

it's a draft still.

webkit-early-warning-system · 2025-06-27T12:28:05Z

EWS run on previous version of this PR (hash ca45962)

Misc	iOS, visionOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	❌ 🛠 ios	❌ 🛠 mac	✅ 🛠 wpe	✅ 🛠 win
✅ 🧪 bindings	❌ 🛠 ios-sim	✅ 🛠 mac-AS-debug	✅ 🧪 wpe-wk2	⏳ 🧪 win-tests
✅ 🧪 webkitperl	❌ 🧪 ios-wk2	❌ 🧪 api-mac	✅ 🧪 api-wpe
	❌ 🧪 ios-wk2-wpt	❌ 🧪 mac-wk1	✅ 🛠 wpe-cairo
	❌ 🧪 api-ios	❌ 🧪 mac-wk2	✅ 🛠 gtk
	✅ 🛠 vision	❌ 🧪 mac-AS-debug-wk2	✅ 🧪 gtk-wk2
	✅ 🛠 vision-sim	❌ 🧪 mac-wk2-stress	❌ 🧪 api-gtk
	✅ 🧪 vision-wk2	❌ 🧪 mac-intel-wk2	✅ 🛠 playstation
	❌ 🛠 tv	❌ 🛠 mac-safer-cpp
	❌ 🛠 tv-sim
	❌ 🛠 watch
	❌ 🛠 watch-sim

webkit-early-warning-system · 2025-06-28T07:25:59Z

EWS run on previous version of this PR (hash 7f5d9c5)

Misc	iOS, visionOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	✅ 🛠 wpe	✅ 🛠 win
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	~~🧪 wpe-wk2~~	⏳ 🧪 win-tests
✅ 🧪 webkitperl	~~🧪 ios-wk2~~	~~🧪 api-mac~~	~~🧪 api-wpe~~
	~~🧪 ios-wk2-wpt~~	~~🧪 mac-wk1~~	✅ 🛠 wpe-cairo
	~~🧪 api-ios~~	~~🧪 mac-wk2~~	✅ 🛠 gtk
	✅ 🛠 vision	~~🧪 mac-AS-debug-wk2~~	~~🧪 gtk-wk2~~
	✅ 🛠 vision-sim	~~🧪 mac-wk2-stress~~	~~🧪 api-gtk~~
	~~🧪 vision-wk2~~	~~🧪 mac-intel-wk2~~	✅ 🛠 playstation
	✅ 🛠 tv	✅ 🛠 mac-safer-cpp
	✅ 🛠 tv-sim
	✅ 🛠 watch
	✅ 🛠 watch-sim

webkit-early-warning-system · 2025-06-28T07:42:55Z

EWS run on previous version of this PR (hash 460473b)

Misc	iOS, visionOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	✅ 🛠 wpe	✅ 🛠 win
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	~~🧪 wpe-wk2~~	⏳ 🧪 win-tests
✅ 🧪 webkitperl	~~🧪 ios-wk2~~	~~🧪 api-mac~~	~~🧪 api-wpe~~
	~~🧪 ios-wk2-wpt~~	~~🧪 mac-wk1~~	✅ 🛠 wpe-cairo
	~~🧪 api-ios~~	~~🧪 mac-wk2~~	✅ 🛠 gtk
	✅ 🛠 vision	~~🧪 mac-AS-debug-wk2~~	~~🧪 gtk-wk2~~
	✅ 🛠 vision-sim	~~🧪 mac-wk2-stress~~	~~🧪 api-gtk~~
	~~🧪 vision-wk2~~	~~🧪 mac-intel-wk2~~	✅ 🛠 playstation
	✅ 🛠 tv	✅ 🛠 mac-safer-cpp
	✅ 🛠 tv-sim
	✅ 🛠 watch
	✅ 🛠 watch-sim

webkit-early-warning-system · 2025-06-28T08:12:09Z

EWS run on previous version of this PR (hash 7bc6408)

Misc	iOS, visionOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	✅ 🛠 wpe	✅ 🛠 win
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	✅ 🧪 wpe-wk2	⏳ 🧪 win-tests
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	✅ 🧪 api-mac	✅ 🧪 api-wpe
	✅ 🧪 ios-wk2-wpt	~~🧪 mac-wk1~~	✅ 🛠 wpe-cairo
	✅ 🧪 api-ios	~~🧪 mac-wk2~~	✅ 🛠 gtk
	✅ 🛠 vision	~~🧪 mac-AS-debug-wk2~~	✅ 🧪 gtk-wk2
	✅ 🛠 vision-sim	❌ 🧪 mac-wk2-stress	✅ 🧪 api-gtk
	✅ 🧪 vision-wk2	✅ 🧪 mac-intel-wk2	✅ 🛠 playstation
	✅ 🛠 tv	✅ 🛠 mac-safer-cpp
	✅ 🛠 tv-sim
	✅ 🛠 watch
	✅ 🛠 watch-sim

webkit-early-warning-system · 2025-06-28T10:01:02Z

EWS run on previous version of this PR (hash a350070)

Misc	iOS, visionOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	~~🛠 wpe~~	~~🛠 win~~
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	~~🧪 wpe-wk2~~	~~🧪 win-tests~~
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	✅ 🧪 api-mac	~~🧪 api-wpe~~
	✅ 🧪 ios-wk2-wpt	~~🧪 mac-wk1~~	✅ 🛠 wpe-cairo
	✅ 🧪 api-ios	~~🧪 mac-wk2~~	~~🛠 gtk~~
	✅ 🛠 vision	~~🧪 mac-AS-debug-wk2~~	~~🧪 gtk-wk2~~
	✅ 🛠 vision-sim	❌ 🧪 mac-wk2-stress	~~🧪 api-gtk~~
	✅ 🧪 vision-wk2	✅ 🧪 mac-intel-wk2	✅ 🛠 playstation
	✅ 🛠 tv	✅ 🛠 mac-safer-cpp
	✅ 🛠 tv-sim
	✅ 🛠 watch
	✅ 🛠 watch-sim

webkit-early-warning-system · 2025-06-28T11:48:42Z

EWS run on previous version of this PR (hash 132092b)

Misc	iOS, visionOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	~~🛠 ios~~	~~🛠 mac~~	~~🛠 wpe~~	~~🛠 win~~
✅ 🧪 bindings	❌ 🛠 ios-sim	❌ 🛠 mac-AS-debug	~~🧪 wpe-wk2~~	~~🧪 win-tests~~
✅ 🧪 webkitperl	❌ 🧪 ios-wk2	~~🧪 api-mac~~	~~🧪 api-wpe~~
	❌ 🧪 ios-wk2-wpt	~~🧪 mac-wk1~~	⏳ 🛠 wpe-cairo
	❌ 🧪 api-ios	~~🧪 mac-wk2~~	~~🛠 gtk~~
	~~🛠 vision~~	❌ 🧪 mac-AS-debug-wk2	~~🧪 gtk-wk2~~
	❌ 🛠 vision-sim	~~🧪 mac-wk2-stress~~	~~🧪 api-gtk~~
	⏳ 🧪 vision-wk2	~~🧪 mac-intel-wk2~~	~~🛠 playstation~~
	~~🛠 tv~~	❌ 🛠 mac-safer-cpp
	~~🛠 tv-sim~~
	~~🛠 watch~~
	~~🛠 watch-sim~~

webkit-early-warning-system · 2025-06-28T11:58:41Z

EWS run on previous version of this PR (hash 4a997dd)

Misc	iOS, visionOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	~~🛠 wpe~~	🛠 win
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	~~🧪 wpe-wk2~~	🧪 win-tests
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	✅ 🧪 api-mac	~~🧪 api-wpe~~
	✅ 🧪 ios-wk2-wpt	~~🧪 mac-wk1~~	✅ 🛠 wpe-cairo
	✅ 🧪 api-ios	✅ 🧪 mac-wk2	~~🛠 gtk~~
	✅ 🛠 vision	✅ 🧪 mac-AS-debug-wk2	~~🧪 gtk-wk2~~
	✅ 🛠 vision-sim	✅ 🧪 mac-wk2-stress	~~🧪 api-gtk~~
	✅ 🧪 vision-wk2	✅ 🧪 mac-intel-wk2	✅ 🛠 playstation
	✅ 🛠 tv	✅ 🛠 mac-safer-cpp
	✅ 🛠 tv-sim
	✅ 🛠 watch
	✅ 🛠 watch-sim

webkit-early-warning-system · 2025-06-28T14:12:38Z

EWS run on previous version of this PR (hash 35e3261)

Misc	iOS, visionOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	🛠 wpe	⏳ 🛠 win
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	🧪 wpe-wk2	⏳ 🧪 win-tests
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	✅ 🧪 api-mac	🧪 api-wpe
	✅ 🧪 ios-wk2-wpt	~~🧪 mac-wk1~~	✅ 🛠 wpe-cairo
	✅ 🧪 api-ios	~~🧪 mac-wk2~~	~~🛠 gtk~~
	✅ 🛠 vision	~~🧪 mac-AS-debug-wk2~~	~~🧪 gtk-wk2~~
	✅ 🛠 vision-sim	❌ 🧪 mac-wk2-stress	~~🧪 api-gtk~~
	✅ 🧪 vision-wk2	✅ 🧪 mac-intel-wk2	✅ 🛠 playstation
	✅ 🛠 tv	✅ 🛠 mac-safer-cpp
	✅ 🛠 tv-sim
	✅ 🛠 watch
	✅ 🛠 watch-sim

webkit-early-warning-system · 2025-06-28T15:38:37Z

EWS run on current version of this PR (hash 96af1f0)

Misc	iOS, visionOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	~~🛠 wpe~~	~~🛠 win~~
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	~~🧪 wpe-wk2~~	~~🧪 win-tests~~
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	✅ 🧪 api-mac	~~🧪 api-wpe~~
	✅ 🧪 ios-wk2-wpt	✅ 🧪 mac-wk1	✅ 🛠 wpe-cairo
	✅ 🧪 api-ios	✅ 🧪 mac-wk2	~~🛠 gtk~~
	✅ 🛠 vision	✅ 🧪 mac-AS-debug-wk2	~~🧪 gtk-wk2~~
	✅ 🛠 vision-sim	✅ 🧪 mac-wk2-stress	~~🧪 api-gtk~~
✅ 🛠 🧪 merge	✅ 🧪 vision-wk2	✅ 🧪 mac-intel-wk2	✅ 🛠 playstation
	✅ 🛠 tv	✅ 🛠 mac-safer-cpp
	✅ 🛠 tv-sim
	✅ 🛠 watch
	✅ 🛠 watch-sim

…utputHandler to decode stereo images https://bugs.webkit.org/show_bug.cgi?id=295143 rdar://154516073 Reviewed by Jer Noble. In 296748@main we fixed a regression caused by 295444@main. The change was a simple temporary workaround to disable MediaSourcePrefersDecompressionSession for spatial or immersive videos as WebCoreDecompressionSession didn't decode stereo images. We now use the VTDecompressionSessionDecodeFrameWithMultiImageCapableOutputHandler method instead which can handle both images in CMSampleBuffer to decode. We add support for decoding multi-images (stereo) samples and readback of such images. As such, we can revert 296748@main which is no longer needed. We make WebCoreDecompression's WorkQueue a singleton. It simplifies the code a bit and this workqueue carries very little load as all the object it uses already use their own workqueues. Spatial portal manually tested on visionOS. Added test to verify decoding and readback when playing a spatial video in stereo mode. * LayoutTests/media/media-source/content/test-spatial-manifest.json: Added. * LayoutTests/media/media-source/content/test-spatial.mp4: Added. * LayoutTests/media/media-source/media-source-paint-stereo-to-canvas-expected.txt: Added. * LayoutTests/media/media-source/media-source-paint-stereo-to-canvas.html: Added. * Source/WebCore/PAL/pal/cf/CoreMediaSoftLink.cpp: Adding dynamically linked methods * Source/WebCore/PAL/pal/cf/CoreMediaSoftLink.h: * Source/WebCore/platform/cocoa/VideoToolboxSoftLink.cpp: Adding dynamically linked methods * Source/WebCore/platform/cocoa/VideoToolboxSoftLink.h: * Source/WebCore/platform/graphics/avfoundation/FormatDescriptionUtilities.cpp: (WebCore::spatialVideoMetadataFromFormatDescription): Remove canLoad method for symbols that will always be found. * Source/WebCore/platform/graphics/cocoa/VideoMediaSampleRenderer.mm: (WebCore::VideoMediaSampleRenderer::enqueueSample): Revert 296748@main as it's no longer needed. (WebCore::VideoMediaSampleRenderer::decodeNextSampleIfNeeded): (WebCore::VideoMediaSampleRenderer::imageForSample const): Handle multi-images sample (WebCore::VideoMediaSampleRenderer::assignResourceOwner): Amend to assign all multi-images sample's IOSurfaces Canonical link: https://commits.webkit.org/296854@main

webkit-commit-queue · 2025-07-01T06:47:36Z

Committed 296854@main (c0688ab): https://commits.webkit.org/296854@main

Reviewed commits have been landed. Closing PR #47186 and removing active labels.

jyavenard self-assigned this Jun 25, 2025

jyavenard added the Media Bugs related to the HTML 5 Media elements. label Jun 25, 2025

jyavenard force-pushed the eng/Safari-shows-black-UI-after-entering-Portal-Mode-for-MSE-spatial-Videos branch from 5e4a146 to 6eb05b2 Compare June 25, 2025 23:17

jyavenard force-pushed the eng/Safari-shows-black-UI-after-entering-Portal-Mode-for-MSE-spatial-Videos branch from 6eb05b2 to 701b299 Compare June 26, 2025 02:15

jyavenard force-pushed the eng/Safari-shows-black-UI-after-entering-Portal-Mode-for-MSE-spatial-Videos branch from 701b299 to 16bafbc Compare June 26, 2025 02:37

jyavenard force-pushed the eng/Safari-shows-black-UI-after-entering-Portal-Mode-for-MSE-spatial-Videos branch from 16bafbc to c655439 Compare June 26, 2025 02:40

webkit-ews-buildbot added the merging-blocked Applied to prevent a change from being merged label Jun 26, 2025

jernoble approved these changes Jun 26, 2025

View reviewed changes

jyavenard removed the merging-blocked Applied to prevent a change from being merged label Jun 27, 2025

jyavenard force-pushed the eng/Safari-shows-black-UI-after-entering-Portal-Mode-for-MSE-spatial-Videos branch from c655439 to ca45962 Compare June 27, 2025 12:28

webkit-ews-buildbot added the merging-blocked Applied to prevent a change from being merged label Jun 27, 2025

jyavenard removed the merging-blocked Applied to prevent a change from being merged label Jun 28, 2025

jyavenard force-pushed the eng/Safari-shows-black-UI-after-entering-Portal-Mode-for-MSE-spatial-Videos branch from ca45962 to 7f5d9c5 Compare June 28, 2025 07:25

jyavenard changed the title ~~Safari shows black UI after entering Portal Mode for MSE spatial Videos.~~ [Cocoa] Adopt VTDecompressionSessionDecodeFrameWithMultiImageCapableOutputHandler to decode stereo images Jun 28, 2025

jyavenard force-pushed the eng/Safari-shows-black-UI-after-entering-Portal-Mode-for-MSE-spatial-Videos branch from 7f5d9c5 to 460473b Compare June 28, 2025 07:42

jyavenard requested a review from jernoble June 28, 2025 07:43

jyavenard marked this pull request as ready for review June 28, 2025 07:43

jyavenard requested review from cdumez and rniwa as code owners June 28, 2025 07:43

jyavenard force-pushed the eng/Safari-shows-black-UI-after-entering-Portal-Mode-for-MSE-spatial-Videos branch from 460473b to 7bc6408 Compare June 28, 2025 08:12

jyavenard force-pushed the eng/Safari-shows-black-UI-after-entering-Portal-Mode-for-MSE-spatial-Videos branch from 7bc6408 to a350070 Compare June 28, 2025 10:00

jyavenard changed the title ~~[Cocoa] Adopt VTDecompressionSessionDecodeFrameWithMultiImageCapableOutputHandler to decode stereo images~~ test disabling HEVC stereo key Jun 28, 2025

jyavenard force-pushed the eng/Safari-shows-black-UI-after-entering-Portal-Mode-for-MSE-spatial-Videos branch from 132092b to 4a997dd Compare June 28, 2025 11:58

jyavenard changed the title ~~test disabling HEVC stereo key~~ have video in DOM and visible. Jun 28, 2025

jyavenard force-pushed the eng/Safari-shows-black-UI-after-entering-Portal-Mode-for-MSE-spatial-Videos branch from 4a997dd to 35e3261 Compare June 28, 2025 14:12

jyavenard changed the title ~~have video in DOM and visible.~~ [Cocoa] Adopt VTDecompressionSessionDecodeFrameWithMultiImageCapableOutputHandler to decode stereo images Jun 28, 2025

jyavenard force-pushed the eng/Safari-shows-black-UI-after-entering-Portal-Mode-for-MSE-spatial-Videos branch from 35e3261 to 96af1f0 Compare June 28, 2025 15:38

webkit-ews-buildbot added the merging-blocked Applied to prevent a change from being merged label Jun 28, 2025

jyavenard requested a review from a team June 28, 2025 22:56

jyavenard removed the merging-blocked Applied to prevent a change from being merged label Jun 30, 2025

jernoble approved these changes Jun 30, 2025

View reviewed changes

jyavenard added the merge-queue Applied to send a pull request to merge-queue label Jul 1, 2025

webkit-commit-queue force-pushed the eng/Safari-shows-black-UI-after-entering-Portal-Mode-for-MSE-spatial-Videos branch from 96af1f0 to c0688ab Compare July 1, 2025 06:47

webkit-commit-queue merged commit c0688ab into WebKit:main Jul 1, 2025

webkit-commit-queue removed the merge-queue Applied to send a pull request to merge-queue label Jul 1, 2025

[Cocoa] Adopt VTDecompressionSessionDecodeFrameWithMultiImageCapableOutputHandler to decode stereo images #47186

[Cocoa] Adopt VTDecompressionSessionDecodeFrameWithMultiImageCapableOutputHandler to decode stereo images #47186

Conversation

jyavenard commented Jun 25, 2025 • edited by webkit-early-warning-system Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

c0688ab

Uh oh!

webkit-early-warning-system commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

webkit-ews-buildbot commented Jun 25, 2025

Safer C++ Build #41549 (5e4a146)

Uh oh!

webkit-early-warning-system commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

webkit-early-warning-system commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

webkit-early-warning-system commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

webkit-early-warning-system commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

webkit-ews-buildbot commented Jun 26, 2025

Safer C++ Build #41610 (6eb05b2)

Uh oh!

jernoble left a comment

Choose a reason for hiding this comment

Uh oh!

jernoble Jun 26, 2025

Choose a reason for hiding this comment

Uh oh!

jyavenard Jun 27, 2025

Choose a reason for hiding this comment

Uh oh!

webkit-early-warning-system commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

webkit-early-warning-system commented Jun 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

webkit-early-warning-system commented Jun 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

webkit-early-warning-system commented Jun 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

webkit-early-warning-system commented Jun 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

webkit-early-warning-system commented Jun 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

webkit-early-warning-system commented Jun 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

webkit-early-warning-system commented Jun 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

webkit-early-warning-system commented Jun 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

webkit-commit-queue commented Jul 1, 2025

Uh oh!

Uh oh!

jyavenard commented Jun 25, 2025 •

edited by webkit-early-warning-system

Loading

`c0688ab`

webkit-early-warning-system commented Jun 25, 2025 •

edited

Loading

Safer C++ Build #41549 (`5e4a146`)

webkit-early-warning-system commented Jun 25, 2025 •

edited

Loading

webkit-early-warning-system commented Jun 26, 2025 •

edited

Loading

webkit-early-warning-system commented Jun 26, 2025 •

edited

Loading

webkit-early-warning-system commented Jun 26, 2025 •

edited

Loading

Safer C++ Build #41610 (`6eb05b2`)

webkit-early-warning-system commented Jun 27, 2025 •

edited

Loading

webkit-early-warning-system commented Jun 28, 2025 •

edited

Loading

webkit-early-warning-system commented Jun 28, 2025 •

edited

Loading

webkit-early-warning-system commented Jun 28, 2025 •

edited

Loading

webkit-early-warning-system commented Jun 28, 2025 •

edited

Loading

webkit-early-warning-system commented Jun 28, 2025 •

edited

Loading

webkit-early-warning-system commented Jun 28, 2025 •

edited

Loading

webkit-early-warning-system commented Jun 28, 2025 •

edited

Loading

webkit-early-warning-system commented Jun 28, 2025 •

edited

Loading