Dialog window procedure as a member function of a class

Lets start with the error message. You are trying to use a non-static class method, WinProc(), as your CreateDialog() callback. That will not work. You went to the trouble of implementing a static class method, WinProcWraper(), which calls WinProc(), but you are not actually using it. You need to use WinProcWraper() as the actual window procedure for CreateDialog().

Also, there is no need to type-cast WinProcWraper when passing it to CreateDialog(), if its signature is correct to begin with (which yours is not).

After fixing that, you still have other problems:

• the other code you are modeling after was designed for a window procedure for CreateWindow/Ex(), but you are using CreateDialog() instead, which has different requirements.

• Since you are using CreateDialog() instead of CreateWindow/Ex(), WinProcWraper() and WinProc() need to return a BOOL instead of an LRESULT. If an actual LRESULT value needs to be returned to the system, WinProc() will need to use SetWindowLong(DWL_MSGRESULT) for that purpose.

• WinProcWraper() is expecting to receive a WM_NCCREATE message to deliver it a MyClass* pointer, which it will then assign to the HWND. But, since you are using CreateDialog(), WinProcWraper() will not receive any WM_(NC)CREATE messages, it will receive a WM_INITDIALOG message instead. And even then, CreateDialog() does not allow you to pass any user-defined value to the window procedure, so you can't pass in the MyClass* pointer that WinProcWraper() requires. You need to use CreateDialogParam() instead for that.

• WM_INITDIALOG may not be the first message a window receives, just the first message that can give you access to the MyClass* pointer that you pass to CreateDialogParam(). WinProcWraper() needs to handle the possibility that it can receive messages before it has received access to the MyClass* pointer.

• You are using SetWindowLong(). Your code will not work if it is ever compiled for 64bit. Use SetWindowLongPtr() instead, even for 32bit code.

With that said, try this:

class MyClass
{
public:
    HINSTANCE hInstance;
    HWND hWnd;

    int APIENTRY WinMainProc(HINSTANCE hInstance, HINSTANCE hPrevInstance, LPSTR lpCmdLine, int nCmdShow);
    BOOL CALLBACK WinProc(UINT uMsg, WPARAM wParam, LPARAM lParam);
    ATOM MyClassRegisterClass(HINSTANCE hInstance); 

private:
    static BOOL CALLBACK WinProcWraper(HWND hwnd, UINT uMsg, WPARAM wParam, LPARAM lParam);
};

BOOL CALLBACK MyClass::WinProcWraper(HWND hWnd, UINT uMsg, WPARAM wParam, LPARAM lParam)
{
    MyClass *cls;

    if (WM_INITDIALOG == uMsg)
    {
        cls = reinterpret_cast<MyClass*>(lParam);
        SetWindowLongPtr(hWnd, GWLP_USERDATA, reinterpret_cast<LONG_PTR>(cls));
    }
    else
        cls = reinterpret_cast<MyClass*>(GetWindowLongPtr(hWnd, GWLP_USERDATA));

    if (cls)
        return cls->WinProc(uMsg, wParam, lParam);

    return FALSE;
}

int APIENTRY MyClass::WinMainProc(HINSTANCE hInstance, HINSTANCE hPrevInstance, LPSTR lpCmdLine, int nCmdShow)
{
    MSG msg;
    MyClass.hInstance = hInstance;
    MyClass.MyClassRegisterClass(hInstance);

    HWND hWnd = CreateDialogParam(hInstance, MAKEINTRESOURCE(IDD_MAIN_DIALOG), NULL, MyClass::WinProcWraper, reinterpret_cast<LPARAM>(this));
    if (!hWnd)
        return -1;

    ShowWindow(hWnd, nCmdShow); 
    UpdateWindow(hWnd);

    while (GetMessage(&msg, NULL, 0, 0))
    {
        if (!IsWindow(hWnd) || !IsDialogMessage(hWnd, &msg))
        {
            TranslateMessage(&msg);
            DispatchMessage(&msg);
        }
    }

    return static_cast<int>(msg.wParam);
}

BOOL CALLBACK MyClass::WinProc(UINT uMsg, WPARAM wParam, LPARAM lParam)
{
    ...

    if (needed)
        SetWindowLongPtr(hWnd, DWLP_MSGRESULT, ...);

    return ...; // TRUE or FALSE as needed...
}