Integrate with Win32 app

[venki-thiyag]

What are the steps needed to integrate FFmpegInteropX with Win32 or WinUI3 app?

[brabebhin]

Hi,

Basically, you would use this the same way as you'd use any windows runtime component in a desktop app.

See https://docs.microsoft.com/en-us/windows/apps/desktop/modernize/desktop-to-uwp-enhance

Typically this involves adding a packaging project in your solution, and adding installing the nuget or referring the ffmpeginteropx project directly in the packaging project.

You can then use it as you'd use any other winRT component.
Problem is though, we relay on some constructs that are UWP specific to get full rendering. Audio rendering is fully supported, but video may require some hacks:

1. If you use C++, you will be able to feed our media stream source to an MF EVR object and render video with it. But no subtitles there :(
2. If you use something else, you will need to use the MediaPlayer object in frame server mode and copy the frames to a DirectX surface.
3. Probably good would be to use XAML islands and a wrapped MediaPlayerElement to get full video + subtitles working. However, this won't work with winUI 3. I haven't tested it when it comes to performance though, you may drop some frames.

We are exploring ways of rendering the subtitles ourselves so that we can eventually fully implement video playback ourselves with minimal winRT support, but that is probably still some months away.

[venki-thiyag]

Thanks @brabebhin for the update. I got this working with following steps:

1. Successfully built FFmpegInteropXWinUI (branch: standard-cpp-migration repo: https://github.com/ffmpeginteropx/FFmpegInteropX.git). FFmpeg were pre-built built ones
2. Created independent Win32 project and followed steps mentioned in https://blogs.windows.com/windowsdeveloper/2019/04/30/enhancing-non-packaged-desktop-apps-using-windows-runtime-components/ to integreate FFmpegInteropXWinUI components
3. One problem I faced was VCRUNTIME140_APP.dll not found, so I copied VC runtime libraries from following location C:\program files (x86)\msbuild\15.0.net.netnative\15.0.24211\x64\ilc\lib\MSCRT to output folder, post this FFmpegInteropX component got loaded into application
4. I am using MediaFoundation source reader to generate RGBA frames of 360p format, note that input video file might be of any resolution. For this I am following the below steps:
   4.a) Used FFmpegMediaSource::CreateFromUriAsync and wait_for to get FFmpegInteropXWinUI::FFmpegMediaSource object
   4.b) Used FFmpegMediaSource::GetMediaStreamSource() and convert to IMFGetService. From IMFGetService fetch MF_MEDIASOURCE_SERVICE object
   4.c) Used MFCreateSourceReaderFromMediaSource with IMFMediaSource, also in attributes have set MF_SOURCE_READER_D3D_MANAGER, MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS = true and MF_SOURCE_READER_ENABLE_ADVANCED_VIDEO_PROCESSING = true

Only problem I am facing is video file decode seems to happen in software and not in HW, whereas in MediaPlayerCPP app in FFmpegInteropXWinUI seems to use hardware.

Any configurations that I am missing here?

[brabebhin]

Ok so you're using media foundation directly.
The various createFrom methods on the ffmpeg media source can receive a MediaSourceConfig, where you can control hardware acceleration. But by default it should work with hardware.

[lukasf]

Hardware acceleration in MF usually only works if you are using EnhancedVideoRenderer in the MF chain. The EnhancedVideoRenderer will make its HW device available to other services in the filter graph. We are using IMFDXGIDeviceManagerSource to get hold of the device. I am not sure if it is possible to manually inject a device into the MF engine without EVR.

If you want to do video playback in a Win32 app, you are probably better off using FFmpegInteropX with MediaPlayer in Frame Server Mode.

https://docs.microsoft.com/en-us/windows/uwp/audio-video-camera/play-audio-and-video-with-mediaplayer#use-mediaplayer-in-frame-server-mode

This generally works, only subtitles are a problem.

[venki-thiyag]

I got this working, problem was IMFSourceReader::ReadSample API was getting called continuously instead of at frame rate of the video file. After making it constant at 33 fps, now I see both CPU and GPU is matching with the MediaPlayerCS sample app.

There is no EVR involved here and for rendering is we use D3D11 which is not part of Media Foundation pipeline. Our use case a bit different where we need to use video file as background of camera.

Another question, steps which were taken for Windows runtime integration, will the same be applicable if the consuming type is library(dll)?

[brabebhin]

Unfortunately i never tried that. But i don't see anything that could be different. Ensure you have all the dependencies together in the same folder.

[venki-thiyag]

Was able to load WinRT component inside Win32 dll, solution is proposed in the following location:
https://docs.microsoft.com/en-us/answers/questions/924996/how-to-consume-winrt-component-inside-win32-dll-pr.html
Not a clean one, but was unable to find any other way!

[lukasf]

You can now use the latest FFmpegInteropX (1.1.0-pre12) package with latest prerelease WinUI 3 (WindowsAppSdk 1.2.220902.1-preview1). Check the WinUI sample, which is now included in our codebase. Please note that this currently only works in packaged app projects. And there are a few things that do not work 100% (some bugs on WinUI side, some issues on our side such as slight subtitle flickering). Things will probably get better once WindowsAppSdk 1.2 gets released officially.

[venki-thiyag]

Thanks @lukasf , our team were able to integrate inside win32 app using the above mentioned mechanisms and inside win32 dll project using mechanisms mentioned in #304