Direct assembly code execution in VBA - only fails in Excel

Question

I have some VBA code to swap one of the function pointers in a COM vtable with a NO-OP bit of assembly code. It works in twinBASIC which is a standalone VBA environment, so I know I'm very close, however in Excel it crashes.

This is the minrepro, works in tB and in theory not in Excel VBA.

Class DummyThing
    Sub DoNothing()
    End Sub
End Class

Module VBA
    Private Const MEM_COMMIT As Long = &H1000
    Private Const MEM_RESERVE As Long = &H2000
    Private Const PAGE_EXECUTE_READWRITE  As Long = &H40

    Declare PtrSafe Function VirtualAlloc Lib "kernel32" ( _
            ByVal lpAddress As LongPtr, _
            ByVal dwSize As Long, _
            ByVal flAllocationType As Long, _
            ByVal flProtect As Long) As LongPtr

    Sub Main()
        Dim code(1 To 4) As Byte
        code(1) = CByte(&h48)    
        code(2) = CByte(&h31)
        code(3) = CByte(&hC0) 'xor rax, rax to clear it - this is like setting the hresult to 0
        code(4) = CByte(&hC3) 'ret

        Dim buffer As LongPtr
        buffer = VirtualAlloc(0&, UBound(code) - LBound(code) + 1, MEM_COMMIT Or MEM_RESERVE, PAGE_EXECUTE_READWRITE)

        If buffer = 0 Then
            Debug.Print "VirtualAlloc() failed (Err:" ; Err.LastDllError ; ")."
            Exit Sub
        End If

        CopyMemory ByVal buffer, code(1), UBound(code) - LBound(code) + 1

        Dim base As DummyThing
        Set base = New DummyThing

        Dim vtable As LongPtr
        vtable = MemLongPtr(ObjPtr(base))

        MemLongPtr(vtable + PTR_SIZE * 7) = buffer

        base.DoNothing 'Excel VBA crashes here, tB prints the message below
        Debug.Print vbNewLine ; "Done!"
    End Sub

End Module

_{The memory apis come from here https://github.com/cristianbuse/VBA-MemoryTools/blob/master/src/LibMemory.bas}

It is a super simple 64 bit assembly code that runs by swapping the vtable of Sub DoNothing() out and replacing it with a pointer to some executable opcodes. The assembly code is nothing more than

48 31 C0          xor    rax, rax                ; Set return value to 0
C3                ret                            ; Return

What might be causing the crash - maybe VBA checks the vtable integrity and that it points to memory in an expected address range? But I've never had this issue before with overloading vtables.

Do you use the code you show in your question? If not, why showing only a slice? If yes, it misses some API, Properties declarations (CopyMemory, MemLongPtr)... — FaneDuru
– FaneDuru, Commented Sep 30, 2023 at 13:27
Is your excel 64 bit? How about your twinBASIC? PS: xor eax, eax is enough in 64 bit as well as 32 so you can drop the 48 byte. That has nothing to do with the crash though. — Jester
– Jester, Commented Sep 30, 2023 at 14:39
If decoded in 32-bit mode, 48 31 C0 is dec eax / xor eax, eax, so even a 32-bit excel wouldn't explain the crash, assuming execution actually reached your machine code. (e.g. that the vtable layout is the same, etc.) — Peter Cordes
– Peter Cordes, Commented Sep 30, 2023 at 15:05
@Greedo xor eax, eax in a 64-bit segment actually does write the full 64 bits of rax due to the zero-extension when you write to a 32-bit register. — ecm
– ecm, Commented Sep 30, 2023 at 18:34
Thanks @Greedo . Yes, I was reffering to a private array member of a class. No need to share now as I wouldn'g have the time to look at it anyway but pleasse do let me know when you manage to make it work - that would be awesome. Thanks! — Cristian Buse
– Cristian Buse, Commented Oct 1, 2023 at 14:27

Greedo · Accepted Answer · 2023-10-23 07:22:06Z

Found an answer

In VBA64, for classes implemented by VBA, [the interpreter] doesn't use the vtable function pointers directly. Instead, it detects that the object is implemented by VBA (itself) and shortcuts to the direct pcode implementation of the requested member instead of using the native function pointer which has to jump back into the pcode virtual machine. It's an optimization, though in reality it's a micro-optimization, but ultimately it prevents simple vtable patching from working.

It wasn't always this way. In the early versions of VBA64 (office 2010 without service packs), they did use the native vtable function calls, and you could at that time use normal vtable patching.

https://discord.com/channels/927638153546829845/1157647794484543589/1157660431545024553

So the crash comes from VBA seeing the class is a VBA class and then using the pointer from the vtable as a signpost to shortcut into pcode as an optimisation.

Except because we overwrote the function pointer, it doesn't have corresponding pcode in the same place so the lookup fails or VBA starts trying to interpret random memory as pcode leading to the crash.

Solution is invoke the assem without vtable patching to avoid the optimisation - e.g. using dispcallfunc, or by implementing the entire COM object manually with CoTaskMemAlloc and a handcrafted vtable.

Collectives™ on Stack Overflow

Direct assembly code execution in VBA - only fails in Excel

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related