Call Stack - What is stored in the call stack between the declaration of a caller function's last variable and a callee function's first variable?

From what I calculated, the offset between the 2 addresses of payload and buffer is 28 bytes (addresses are given in HEX in comments next to initialization). I assume the compiler hasn't allocated the full 128 bytes to payload because I haven't input anything yet, so I'll assume only 4 bytes have been allocated. So what else is there occupying the rest of the 24 bytes?

You got a new stack frame when calling vulnerable(). Try running this :

printf("%zu: %p - %p\n", sizeof(payload), payload, &payload[127]);

You will see payload has a correct size.

You might need to review how a stack frame work.

Because the stack is reused efficiently:

• It grows and shrinks with each function call.
• When a frame is popped, that memory region becomes immediately available for reuse.

That’s why the address of buffer may appear to overlap with payload -- they occupy the same stack space at different times.

From what I've learnt in my course right now, char* str will be pushed after payload taking up 4 bytes, and then comes in the return_address_to_main at 4 bytes, a saved ebp pointer for the previous frame which takes 4 more bytes and then buffer should be initialized. But this only adds up to an offset of 16 bytes between the 2 addresses mentioned, what else is in there that I am unable to observe?

strcpy() copies a null-terminated string into another, it doesn't "append" to the destination string. Instead, to make your code "work", and overflow buffer you can initialize payload this way :

char payload[128] = "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA";

I am using an Ubuntu 16.04 a 32-bit Virtual Machine OS.

I executed

sysctl -w kernel.randomize_va_space=0

to disable ASLR in root prior to gcc.

I have also compiled this using the command:

gcc -g -fno-stack-protector -z execstack -m32 -mpreferred-stack-boundary=2 testlab.c -o test

You don't need all these flags or disable ASLR for such a simple stack buffer overflow. You might also try running a shorter code like :

#include <stdlib.h>
#include <stdio.h>
#include <string.h>

int main(int argc, char **argv) {
    char payload[4];
    int toyCanary = 0x11223344;
    
    strcpy(payload, "AAAAAAA");

    if (toyCanary != 0x11223344) {
        printf("You malicious behaviour is detected!\n");
        exit(0);
    }
    return 0;
}

Expected output :

You malicious behaviour is detected!

Call Stack - What is stored in the call stack between the declaration of a caller function's last variable and a callee function's first variable?

The C language does not specify. In fact, it says nothing at all about a call stack, or any specific mechanism for argument passing, and it does not rely on there being a call stack at all.

On stack-based implementations, the caller will typically store context information such as the values of caller-saved registers; the return address; maybe arguments; and possibly other information, and it may allot space for a return value to be stored.

From what I calculated, the offset between the 2 addresses of payload and buffer is 28 bytes (addresses are given in HEX in comments next to initialization).

Yes, the difference between 0xbfffed0c and 0xbfffed28 is 0x1c, which is 28 in decimal.

I assume the compiler hasn't allotted the full 128 bytes to payload because I haven't input anything yet, so I'll assume only 4 bytes have been allocated.

Those are not reasonable assumptions. The lifetime of the payload array begins at the end of its declarator. There being no initializer, its initial value is indeterminate, but that's not the same thing as not having any value at all. It has a full 128 bytes of storage for its entire lifetime.

Moreover, on x86, the stack grows downward. You can observe this in the addresses: buffer has a lower address than payload. The difference is greater than the size of buffer, so these two objects do not overlap, regardless of the size of payload.

So what else is there occupying the rest of the 24 bytes?

The 12 bytes between the end of buffer and the beginning of payload are just enough for the 4-byte pointer argument, a 4-byte return address, and a copy of the caller's 4-byte frame pointer. And those items, in that (downward) order, would be consistent with the System V i386 ABI, which your 32-bit Linux employs. Here's a likely layout of the stack while execution is in the scope of vulnerable:

I will add this, I am sure that only 4 bytes are allotted to payload right now, because when I initialize a test standard integer variable right after, the address of the integer variable is only offset by 4 bytes,

A four-byte difference is exactly what one should expect if a 32-bit variable were immediately below payload in memory. That, in turn, is what I would expect from GCC in the case you describe, as I understand that case. The C language does not say anything about the relative addresses of unrelated objects, however.

and accordingly the offset between payload and buffer becomes 32 bytes.

Sounds like it's the same 12-byte difference between the top of buffer and the last of the caller's local variables, which is now this int. I don't see anything surprising about that.