Embedding unity in WPF, WinUI, UWP application with control overlay

Question

I need to do a POC of Unity integration in a UI framework (WPF, WinUI or UWP) for the company I work for but I am stuck on one of the constraints. The Unity process has to be embedded in the main window and not lose the ability to overlay UI elements. For example

In WPF, WinUI and UWP I can integrate my Unity process in a control (either through a SwapChainPanel, or by creating the process and hosting it in a control with a Window Handle), the WinForm is not possible since a good part of the code base has already been done in XAML.

Unity also allows to build a solution compatible with UWP which supports very well the overlay of elements on top of the Unity rendering, unfortunately the output build is in CPP, and C# is mandatory in our context.

I have mainly used this example to try to embed Unity in an external process, adapting to the several frameworks:

https://docs.unity3d.com/2018.4/Documentation/uploads/Examples/EmbeddedWindow.zip

I've found a lot of documentation that deals with the issue, but none that allows me to properly handle the overlay of elements in C# with WPF, WinUI or UWP (in order of preference).

In summary, I manage to integrate Unity in a control, I manage to interact with my scene (Mouse input), the scene adapts correctly to the host window (resizing) but no UI elements are overlaid on the scene. And despite that, Unity displays me in the scene this error. (Invalid window handle)

My questions: Is it only possible in C# to overlay UI elements on top of a Unity render? If so, how do I go about it?

[Update] Here is the code generated by Unity (only the one related to the SwapChainPanel):

[XAML]

<Page
x:Class="POCTest.MainPage"
IsTabStop="false"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
xmlns:local="using:POCTest"
xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
mc:Ignorable="d"
Background="#FFFFFF">

<SwapChainPanel x:Name="m_DXSwapChainPanel">
    <Grid x:Name="m_ExtendedSplashGrid" Background="#FFFFFF">
        <Image x:Name="m_ExtendedSplashImage" Source="Assets/SplashScreen.png" VerticalAlignment="Center" HorizontalAlignment="Center"/>
    </Grid>
    <Button x:Name="Overlaytest" Content="Overlaytest" HorizontalAlignment="Right"/>
</SwapChainPanel>

[CPP]

//
// MainPage.xaml.cpp
// Implementation of the MainPage class.
//

#include "pch.h"
#include "MainPage.xaml.h"

using namespace POCTest;

using namespace Concurrency;
using namespace Platform;
using namespace UnityPlayer;
using namespace Windows::ApplicationModel::Activation;
using namespace Windows::Foundation;
using namespace Windows::Storage;
using namespace Windows::System::Threading;
using namespace Windows::UI;
using namespace Windows::UI::Core;
using namespace Windows::UI::Xaml;
using namespace Windows::UI::Xaml::Controls;
using namespace Windows::UI::Xaml::Media;
using namespace Windows::UI::Xaml::Navigation;

// The Blank Page item template is documented at http://go.microsoft.com/fwlink/?LinkId=402352&clcid=0x409

MainPage::MainPage()
{
    m_SplashScreenRemovalEventToken.Value = 0;
    m_OnResizeRegistrationToken.Value = 0;

    InitializeComponent();
    NavigationCacheMode = ::NavigationCacheMode::Required;

    auto appCallbacks = AppCallbacks::Instance;

    bool isWindowsHolographic = false;

#if UNITY_HOLOGRAPHIC
    // If application was exported as Holographic check if the device actually supports it,
    // otherwise we treat this as a normal XAML application
    isWindowsHolographic = AppCallbacks::IsMixedRealitySupported();
#endif

    if (isWindowsHolographic)
    {
        appCallbacks->InitializeViewManager(Window::Current->CoreWindow);
    }
    else
    {
        m_SplashScreenRemovalEventToken = appCallbacks->RenderingStarted += ref new RenderingStartedHandler(this, &MainPage::RemoveSplashScreen);

        appCallbacks->SetSwapChainPanel(m_DXSwapChainPanel);
        appCallbacks->SetCoreWindowEvents(Window::Current->CoreWindow);
        appCallbacks->InitializeD3DXAML();

        m_SplashScreen = safe_cast<App^>(App::Current)->GetSplashScreen();

        auto dispatcher = CoreWindow::GetForCurrentThread()->Dispatcher;
        ThreadPool::RunAsync(ref new WorkItemHandler([this, dispatcher](IAsyncAction^)
        {
            GetSplashBackgroundColor(dispatcher);
        }));

        OnResize();

        m_OnResizeRegistrationToken = Window::Current->SizeChanged += ref new WindowSizeChangedEventHandler([this](Object^, WindowSizeChangedEventArgs^)
        {
            OnResize();
        });
    }
}

MainPage::~MainPage()
{
    if (m_SplashScreenRemovalEventToken.Value != 0)
    {
        AppCallbacks::Instance->RenderingStarted -= m_SplashScreenRemovalEventToken;
        m_SplashScreenRemovalEventToken.Value = 0;
    }

    if (m_OnResizeRegistrationToken.Value != 0)
    {
        Window::Current->SizeChanged -= m_OnResizeRegistrationToken;
        m_OnResizeRegistrationToken.Value = 0;
    }
}

/// <summary>
/// Invoked when this page is about to be displayed in a Frame.
/// </summary>
/// <param name="e">Event data that describes how this page was reached.  The Parameter
/// property is typically used to configure the page.</param>
void MainPage::OnNavigatedTo(NavigationEventArgs^ e)
{
    m_SplashScreen = safe_cast<SplashScreen^>(e->Parameter);
    OnResize();
}

void MainPage::OnResize()
{
    if (m_SplashScreen != nullptr)
    {
        m_SplashImageRect = m_SplashScreen->ImageLocation;
        PositionImage();
    }
}

void MainPage::PositionImage()
{
    auto inverseScaleX = 1.0f;
    auto inverseScaleY = 1.0f;

    m_ExtendedSplashImage->SetValue(Canvas::LeftProperty, m_SplashImageRect.X * inverseScaleX);
    m_ExtendedSplashImage->SetValue(Canvas::TopProperty, m_SplashImageRect.Y * inverseScaleY);
    m_ExtendedSplashImage->Height = m_SplashImageRect.Height * inverseScaleY;
    m_ExtendedSplashImage->Width = m_SplashImageRect.Width * inverseScaleX;
}

void MainPage::GetSplashBackgroundColor(CoreDispatcher^ dispatcher)
{
    HandleHolder manifestHandle = CreateFile2(L"AppxManifest.xml", GENERIC_READ, FILE_SHARE_READ, OPEN_EXISTING, nullptr);
    if (manifestHandle == nullptr)
        return;

    LARGE_INTEGER fileSize;
    auto result = GetFileSizeEx(manifestHandle, &fileSize);
    if (result == FALSE)
        return;

    std::string manifest;
    manifest.resize(static_cast<size_t>(fileSize.QuadPart));

    DWORD bytesRead;
    result = ReadFile(manifestHandle, &manifest[0], static_cast<DWORD>(fileSize.QuadPart), &bytesRead, nullptr);
    if (result == FALSE)
        return;
    if (bytesRead != fileSize.QuadPart)
        return;

    auto idx = manifest.find("SplashScreen");

    if (idx == std::string::npos)
        return;

    manifest = manifest.substr(idx);
    idx = manifest.find("BackgroundColor");

    if (idx == std::string::npos)
        return;

    manifest = manifest.substr(idx);
    idx = manifest.find("\"");

    if (idx == std::string::npos || idx + 2 > manifest.length())
        return;

    manifest = manifest.substr(idx + 1); // also remove quote and # char after it
    idx = manifest.find("\"");

    if (idx == std::string::npos)
        return;

    manifest = manifest.substr(0, idx);

    int value = 0;
    bool transparent = false;
    if (manifest == "transparent")
        transparent = true;
    else if (manifest[0] == '#')
    {
        // color value has leading #
        value = std::stoi(manifest.substr(1), 0, 16);
    }
    else
        return; // we reach this point if values like 'red', 'blue' etc are used Unity does not set such, so you probably want to use hardcoded value here too

    uint8_t r = static_cast<uint8_t>(value >> 16);
    uint8_t g = static_cast<uint8_t>((value & 0x0000FF00) >> 8);
    uint8_t b = static_cast<uint8_t>(value & 0x000000FF);

    dispatcher->RunAsync(CoreDispatcherPriority::High, ref new DispatchedHandler([this, r, g, b, transparent]
    {
        Color color;
        color.R = r;
        color.G = g;
        color.B = b;
        color.A = transparent ? 0x00 : 0xFF;
        m_ExtendedSplashGrid->Background = ref new SolidColorBrush(color);
    }));
}

void MainPage::RemoveSplashScreen()
{
    uint32_t index;

    if (m_DXSwapChainPanel->Children->IndexOf(m_ExtendedSplashGrid, &index))
        m_DXSwapChainPanel->Children->RemoveAt(index);

    if (m_OnResizeRegistrationToken.Value != 0)
    {
        Window::Current->SizeChanged -= m_OnResizeRegistrationToken;
        m_OnResizeRegistrationToken.Value = 0;
    }
}

[Update 2] Here my C# code

[App.xaml.cs]

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Runtime.InteropServices.WindowsRuntime;
using Windows.ApplicationModel;
using Windows.ApplicationModel.Activation;
using Windows.Foundation;
using Windows.Foundation.Collections;
using Windows.UI.Xaml;
using Windows.UI.Xaml.Controls;
using Windows.UI.Xaml.Controls.Primitives;
using Windows.UI.Xaml.Data;
using Windows.UI.Xaml.Input;
using Windows.UI.Xaml.Media;
using Windows.UI.Xaml.Navigation;
using Windows.UI.ViewManagement;
using UnityPlayer;

namespace POCTestUWP
{
    /// <summary>
    /// Provides application-specific behavior to supplement the default Application class.
    /// </summary>
    sealed partial class App : Application
    {

        private AppCallbacks m_AppCallbacks;
        /// <summary>
        /// Initializes the singleton application object.  This is the first line of authored code
        /// executed, and as such is the logical equivalent of main() or WinMain().
        /// </summary>
        public App()
        {
            this.InitializeComponent();
            this.Suspending += OnSuspending;
            m_AppCallbacks = new AppCallbacks();
        }

        /// <summary>
        /// Invoked when the application is launched normally by the end user.  Other entry points
        /// will be used such as when the application is launched to open a specific file.
        /// </summary>
        /// <param name="e">Details about the launch request and process.</param>
        protected override void OnLaunched(LaunchActivatedEventArgs e)
        {
            Frame rootFrame = Window.Current.Content as Frame;

            // Do not repeat app initialization when the Window already has content,
            // just ensure that the window is active
            if (rootFrame == null)
            {
                // Create a Frame to act as the navigation context and navigate to the first page
                rootFrame = new Frame();

                rootFrame.NavigationFailed += OnNavigationFailed;

                if (e.PreviousExecutionState == ApplicationExecutionState.Terminated)
                {
                    //TODO: Load state from previously suspended application
                }

                // Place the frame in the current Window
                Window.Current.Content = rootFrame;
            }

            if (e.PrelaunchActivated == false)
            {
                if (rootFrame.Content == null)
                {
                    // When the navigation stack isn't restored navigate to the first page,
                    // configuring the new page by passing required information as a navigation
                    // parameter
                    rootFrame.Navigate(typeof(MainPage), e.Arguments);
                }
                // Ensure the current window is active
                Window.Current.Activate();
            }
            InitializeUnity(e.Arguments);
        }

        /// <summary>
        /// Invoked when Navigation to a certain page fails
        /// </summary>
        /// <param name="sender">The Frame which failed navigation</param>
        /// <param name="e">Details about the navigation failure</param>
        void OnNavigationFailed(object sender, NavigationFailedEventArgs e)
        {
            throw new Exception("Failed to load Page " + e.SourcePageType.FullName);
        }

        /// <summary>
        /// Invoked when application execution is being suspended.  Application state is saved
        /// without knowing whether the application will be terminated or resumed with the contents
        /// of memory still intact.
        /// </summary>
        /// <param name="sender">The source of the suspend request.</param>
        /// <param name="e">Details about the suspend request.</param>
        private void OnSuspending(object sender, SuspendingEventArgs e)
        {
            var deferral = e.SuspendingOperation.GetDeferral();
            //TODO: Save application state and stop any background activity
            deferral.Complete();
        }

        void InitializeUnity(string args)
        {
            ApplicationView.GetForCurrentView().TryEnterFullScreenMode();
            
            m_AppCallbacks.SetAppArguments(args);
            var rootFrame = (Frame)Window.Current.Content;

            // Do not repeat app initialization when the Window already has content,
            // just ensure that the window is active
            if (rootFrame == null && !m_AppCallbacks.IsInitialized())
            {
                rootFrame = new Frame();
                Window.Current.Content = rootFrame;

                rootFrame.Navigate(typeof(MainPage));
            }

            Window.Current.Activate();
        }

    }
}

[MainPage.xaml.cs]

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Runtime.InteropServices.WindowsRuntime;
using Windows.Foundation;
using Windows.Foundation.Collections;
using Windows.UI.Xaml;
using Windows.UI.Xaml.Controls;
using Windows.UI.Xaml.Controls.Primitives;
using Windows.UI.Xaml.Data;
using Windows.UI.Xaml.Input;
using Windows.UI.Xaml.Media;
using Windows.UI.Xaml.Navigation;
using UnityPlayer;

// The Blank Page item template is documented at https://go.microsoft.com/fwlink/?LinkId=402352&clcid=0x409

namespace POCTestUWP
{
    /// <summary>
    /// An empty page that can be used on its own or navigated to within a Frame.
    /// </summary>
    public sealed partial class MainPage : Page
    {
        AppCallbacks appCalbback = AppCallbacks.Instance;

        public MainPage()
        {
            this.InitializeComponent();
            Overlaytest.Click += Overlaytest_Click;
            appCalbback.SetSwapChainPanel(m_DXSwapChainPanel);
            appCalbback.SetCoreWindowEvents(Window.Current.CoreWindow);
            appCalbback.InitializeD3DXAML();
        }

        private void Overlaytest_Click(object sender, RoutedEventArgs e)
        {
            Overlaytest.Content = "Clicked";
        }
    }
}

Thanks in advance for your feedback.

It probably won't work with WPF because of the airspace issues learn.microsoft.com/en-us/dotnet/desktop/wpf/advanced/… You mention it works with UWP but with C++, where is that code? It can probably be ported to C# and maybe a similar resolution with WinUI3 which is close to UWP — Simon Mourier
– Simon Mourier, Commented Mar 30, 2022 at 8:36
@SimonMourier I edited my post to add the code generated by Unity. I just put the part related to the SwapChainPanel. — Aaron B.
– Aaron B., Commented Mar 30, 2022 at 10:57
So what prevents you from porting this code to UWP C# and possibly WinUI3 which also has a SwapChainPanel (learn.microsoft.com/en-us/windows/winui/api/…)? — Simon Mourier
– Simon Mourier, Commented Mar 30, 2022 at 11:05
Cause whenever I try to call any function from UnityPlayer.dll i get "The specified module could not be found. (Exception from HRESULT: 0x8007007E" so I thought I was wrong and abandoned the idea. — Aaron B.
– Aaron B., Commented Mar 30, 2022 at 13:39
I don't see a reason why it wouldn't work from C#, you should post your C# code somewhere. — Simon Mourier
– Simon Mourier, Commented Mar 30, 2022 at 17:45

Aaron B. · Accepted Answer · 2022-04-06 11:53:16Z

I finally managed to integrate Unity into a UWP application while keeping the possibility to overlay. This site helped me a lot, but unfortunately it's a bit too dated and it needs some modifications:

https://github.com/Myfreedom614/UWP-Samples/tree/master/RotateModelUnityUWP

So I'll make a summary if other people need it.

First, under Unity, build the project in an empty folder targeting the "Universal Windows Platform" with at least the following options checked:

Build Type : "XAML Project"
Copy reference : checked

You can either create a UWP solution that will reference the necessary dependencies, or replace the UWP project generated in CPP by Unity (as in the example provided below).

Compile the "Il2CppOutputProject" project then copy the dll "build/obj/il2cppOutputProject/x64/Master/linkresult_[Hash]/GameAssembly.dll" into "bin/x64/Debug/AppX/" folder of your project. In your UWP project add the reference to "Players/UAP/il2cpp/x64/master/UnityPlayer.winmd". The easiest way to do this is to create a post-build event to automatically copy the dll to the output folder (depending on your architecture and configuration).

Example: https://github.com/abassibe/UnityUWP-Overlay

Thanks to you for your help!

Collectives™ on Stack Overflow

Embedding unity in WPF, WinUI, UWP application with control overlay

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related