Figuring out the memory layout of objects without debugging a running program?

Question

I'm working on reverse engineering a program. I'm using IDA Pro and the Hex-Rays Decompiler. I'm coming up against chunks of code where I know there is an object and there is a method call on the object but it's shown in a way I'm not able to figure out. For example:

 if ( (*(*interfacePtr + 24))(interfacePtr, &v23) >= 0 )

I know here that interfacePtr is pointing to a ICLRRuntimeHost object. (C++, .NET CLR runtime) However.... I have no idea what is at *(*interfacePtr + 24). I can tell it's a method but how do I figure out what is sitting at +24?

Don't you think this s better answered on one of the reverse engineering forums? Or perhaps the IDA/hex-rays forum, which you should have access to after paying the hefty pricetag for this software? — snemarch
– snemarch, Commented Aug 12, 2012 at 3:03
IDA can parse C header files for structure declarations. ICLRRuntimeHost is defined in the Windows SDK mscoree.h file, with some work you should be able to either feed this file to IDA or at least extract the class and vtable declarations to another file and parse that. After IDA has a definition of this class, you can simply tell Hex-Rays that interfacePtr is of the type ICLRRuntimeHost (press Y or right-click the variable name) and it will update the display to include the function name. — user1233508
– user1233508, Commented Aug 12, 2012 at 5:38
@snemarch I think this question is perfectly valid considering SO is all about programming, which, at the end of the day.... is what my question is really about. I would use the discussion boards for a product if I had an issue with a product. However, this is simply me not having enough knowledge about the underlying mechanics of a native Windows executable... not me having an issue with IDA or the Hex-Rays decompiler. — Toby Lawrence
– Toby Lawrence, Commented Aug 12, 2012 at 16:07
@DCoder I have been simply correlating the vtable to the method offset specified in the calls for the moment... but I will definitely give your approach a shot. An automatic association would be much much nicer. :) — Toby Lawrence
– Toby Lawrence, Commented Aug 12, 2012 at 16:09

Michael · Accepted Answer · 2012-08-13 05:38:32Z

2

The vtable of a class is just a list of pointers to functions. It contains one pointer for each virtual function, in the order: very, very top base class, next base class, subclass of that, . . . most derived class.

Example:

struct A {
    virtual ~A() {}
    virtual void foo() = 0;
}
struct B : public A {
    virtual void foo() { // do something }
    virtual void bar() { // do something else }
}

The vtable for B would contain, in this order:

~A
foo
bar

(The ones for A have to come first, so that the same vtable can be used by parts of the code that have a pointer of type A to this object; that code doesn't know that the underlying object is really B.)

If you're looking at 32-bit source, pointers are 4 bytes, so 24 = 4 * 6, you're looking at the 7th virtual function (index starts at 0). If you're on 64-bit, pointers are 8 bytes, so 24 = 8 * 3, and you're looking for the 4th. Actually, I haven't used the IDA "convert to C++" functionality, so maybe 24 is actually the 24th entry in the table.

Easy way to confirm: write your own program. Declare a variable of type ICLRRuntimeHost. Call the function you suspect (based on looking at the header file and counting to 7 or 4, depending on bitness, or 24, if I misunderstood your example). Look at the generated assembly code, and confirm whether your index is right. (I'm always off by one on that kind of thing, so this will provide a check.)

edited Aug 13, 2012 at 5:38

answered Aug 12, 2012 at 4:53

Michael

6374 silver badges3 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Toby Lawrence Over a year ago

Am I right in assuming that you meant to declare B extending A?

Toby Lawrence Over a year ago

Aside from my above comment, looking at mscoree.h while reading this psuedocode is matching up nicely as far as the whole vtable ordering scheme. Looks like that was the big missing link for me. :D

Michael Over a year ago

Yes, B should extend A. Fixed.

Willem Hengeveld · Accepted Answer · 2012-09-12 08:14:35Z

Have a look at the definitions of ICLRRuntimeHostVtbl and ICLRRuntimeHostVtbl in mscoree.h

roughly translated to something hexrays would understand, they look like this:

struct ICLRRuntimeHost {
    ICLRRuntimeHostVtbl *vtbl;
};
struct ICLRRuntimeHostVtbl {
    _DWORD (*QueryInterface)(ICLRRuntimeHost*, _DWORD*, void**);
    _DWORD (*AddRef)(ICLRRuntimeHost*);
    _DWORD (*Release)(ICLRRuntimeHost*);
    _DWORD (*Start)(ICLRRuntimeHost*);
    _DWORD (*Stop)(ICLRRuntimeHost*);
    _DWORD (*SetHostControl)(ICLRRuntimeHost*, void*);
    _DWORD (*GetCLRControl)(ICLRRuntimeHost*, void**);
};

Your variable interfacePtr should have type: ICLRRuntimeHost, then your code should decompile like this:

interfacePtr->GetCLRControl(&v23);

Collectives™ on Stack Overflow

Figuring out the memory layout of objects without debugging a running program?

2 Answers 2

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related