feat: SG-43127: SG-43594: 10-bit Vulkan presentation (Linux + Windows)

[cedrik-fuoco-adsk]

SG-43127 - 10-bit support with Vulkan (Linux and Windows)

Linked issues

none

Summarize your change.

Adds a 10-bit Vulkan presentation backend for both Linux and Windows, enabling 30-bit (RGB10 + A2) display output. This supersedes the Linux-only work in #1310 by extending the same backend to Windows in a single PR.

New rendering path (Vulkan 10-bit):

When the user requests a 10-bit display (RGB10 + A2) and the hardware/driver supports it, RV renders through a Vulkan presentation path instead of the usual OpenGL one. The renderer still draws into an offscreen 16-bit float framebuffer (GL_RGBA16F) using OpenGL as before.

The difference is the final step: instead of presenting that framebuffer through a GLView, the GL color buffer is handed to QTVulkanVideoDevice, which blits it into a texture whose memory is shared with a Vulkan image through external-memory interop, and VulkanView presents it through a true 10-bit (VK_FORMAT_A2B10G10R10_UNORM_PACK32) Vulkan swapchain. The GL-to-Vulkan handoff is zero-copy (shared memory and semaphores, no cross-API blit or CPU copy). This is what gives us real 10-bit output without banding, which the OpenGL path could not do on either platform.

Backend selection happens at view-creation time: a 10-bit preference routes to Vulkan when VulkanView::supports10BitPresentation() returns true, and otherwise everything falls back to the existing GLView path unchanged. On the Vulkan path m_glView is null, but the DiagnosticsView is still created: it is an independent QOpenGLWidget with its own GL context, so when m_glView is null it simply falls back to the global default surface format (OpenGL 2.1) instead of the main view's format.

Platform-neutral view abstraction:

RvDocument is decoupled from GLView by routing the active view through a generic QWidget* m_viewWidget / viewWidget() accessor instead of hardcoding m_glView at every shared call site (focus, geometry, stacked-layout, popup/menu coordinate mapping). RvApplication now tolerates a null view()/GL context and uses the session's control video device for the primary display group. The GL path is unchanged; this is the shared base for the Vulkan backend (and the same base can be shared with a future macOS Metal backend).

Vulkan backend (Linux + Windows):

Adds VulkanView (a Vulkan swapchain that selects the A2B10G10R10_UNORM_PACK32 format) and QTVulkanVideoDevice (offscreen 16F FBO + GL/Vulkan external-memory interop), wired into RvDocument behind #if defined(PLATFORM_LINUX) || defined(PLATFORM_WINDOWS). Most of the backend is platform-neutral and lives in shared .cpp files; only the GPU-interop handle plumbing differs per platform:

• Linux uses opaque-FD external memory and semaphores (VK_KHR_external_memory_fd, VK_KHR_external_semaphore_fd, GL_EXT_memory_object_fd, GL_EXT_semaphore_fd).
• Windows uses the Win32-HANDLE equivalents (VK_KHR_external_memory_win32, VK_KHR_external_semaphore_win32, GL_EXT_memory_object_win32, GL_EXT_semaphore_win32).

Handle ownership differs between the two: on Linux the FD is duplicated with dup() and the GL import takes ownership, while on Windows glImportMemoryWin32HandleEXT does not take ownership, so the GL and Vulkan sides each keep and close their own handle. Cleanup paths are written accordingly.

When the driver lacks the interop extensions (or the stride cannot be represented as an integer pixel width), the path degrades gracefully: getSharedImageInfo returns null and a CPU pack-and-upload fallback (VulkanView::presentPixelData) presents through the same Vulkan swapchain, so the output is still 10-bit even when the zero-copy GPU bridge is off. If Vulkan 10-bit presentation is unavailable entirely (or the real surface does not offer A2B10G10R10), it falls back to the OpenGL GLView path.

Vulkan (headers + loader) is fetched as a managed dependency (cmake/dependencies/vulkan.cmake) and linked on both Linux and Windows. CMake gates that were previously IF(RV_TARGET_LINUX) are widened to IF(RV_TARGET_LINUX OR RV_TARGET_WINDOWS), and Windows links Vulkan::Vulkan without pulling in X11. A small VulkanBuildProbe.cpp keeps a direct vkGetInstanceProcAddr reference so a missing loader fails at link time rather than at runtime.

Describe the reason for the change.

OpenRV's OpenGL presentation path is limited to 8-bit output on both Linux and Windows, which causes visible banding on 10-bit-capable displays. Vulkan provides reliable access to 30-bit swapchain formats on modern drivers, allowing true 10-bit presentation. The view abstraction was extracted first so the new backend could be plugged in cleanly without disturbing the existing GL path.

Describe what you have tested and on which operating system.

Linux: 10-bit gradient ramps render smoothly through the Vulkan swapchain (A2B10G10R10_UNORM_PACK32) with the GPU-interop path active. The 8-bit default launch is unchanged and uses GLView.

Windows (AMD Radeon 740M, dual monitor): the Vulkan path builds and runs, selects a true A2B10G10R10_UNORM_PACK32 swapchain, and activates the GPU-interop bridge. One important platform caveat surfaced during testing:

• Windows always composites windowed apps through the Desktop Window Manager (DWM), and DWM only composes at greater than 8 bpc when the target display is in HDR mode. On an SDR-only monitor, DWM truncates the 10-bit swapchain to 8 bpc before scanout, so banding remains even though the swapchain, the GL/Vulkan interop, and the AMD link depth are all correctly 10-bit.
• On an HDR-capable monitor with HDR enabled, the 10-bit swapchain visibly reaches the panel and banding is reduced (confirmed: the surface then advertises HDR colorspaces, a clear signal DWM is in deep-color composition mode).
• This is a known Windows platform limitation, not a bug in RV. Linux has no mandatory compositor gate, so the same Vulkan code, same GPU, and same monitor deliver 10-bit directly.

On the format/colorspace choice: createSwapchain() selects the first A2B10G10R10_UNORM_PACK32 format the surface offers and uses whichever colorspace is paired with it. On the tested hardware that first match is VK_COLOR_SPACE_SRGB_NONLINEAR_KHR, which is the correct 10-bit target for our sRGB-emitting renderer (the HDR10/scRGB colorspaces would require a separate color-managed re-encode of the shader output and are out of scope here).

Add a list of changes, and note any that might need special attention during the review.

• RvDocument / RvApplication view abstraction (m_viewWidget / viewWidget()), shared and platform-neutral.
• New VulkanView (10-bit swapchain) and QTVulkanVideoDevice (16F FBO + GL/Vulkan interop), shared .cpp with per-platform handle plumbing.
• New VulkanBuildProbe.cpp to force loader symbol resolution at link time on both platforms.
• CMake: Vulkan dependency and link gates widened from Linux-only to Linux + Windows; Windows links Vulkan::Vulkan without X11.
• Graceful CPU pack-and-upload fallback and Vulkan-to-GLView fallback paths.

Worth special attention during review:

• Handle-ownership asymmetry between platforms: Linux dup()s the FD and the GL import takes ownership, while the Windows glImportMemoryWin32HandleEXT import does not take ownership, so each side must close exactly the handles it owns.
• GLEW Win32 import symbols (glImportMemoryWin32HandleEXT, glImportSemaphoreWin32HandleEXT) are resolved through wglGetProcAddress on Windows; confirm they are present on the target driver.
• No 10-bit Windows surface in CI yet, so the Windows GPU path was verified manually on a 10-bit-capable monitor.

[bernie-laberge]

LGTM

[cedrik-fuoco-adsk]

Before the latest changes:

The present was single-frame-in-flight. Every frame ended with a blocking wait on the GPU fence, so the CPU sat idle until the GPU finished before starting the next frame. Resize made it worse by rebuilding the shared image each step. The result was choppy resize and playback that stalled then fast-forwarded.

After:

Up to 2 frames run in flight. Per-frame resources (fence, semaphores, shared image, staging buffer) are now per-slot rings indexed by m_currentFrame, so the CPU can prepare the next frame while the GPU finishes the current one. The GL side reads m_view->currentFrame() and uses the same slot, so its imported objects stay paired with the matching Vulkan ones. The blocking wait moved from the end of the frame to the start of the slot it reuses, so it only blocks when the CPU is a full frame ahead. Pacing still comes from FIFO vsync plus that start-of-frame wait.

[cedrik-fuoco-adsk]

Added @bernie-laberge fix for Nvidia and optimization for the CPU fallback.