Why does my custom global allocator crash on large allocations, and how can I handle alignment properly?

Question

I am trying to implement a custom global allocator for use in my Rust application. The goal is to track memory usage and align to 64 bytes. My allocator uses libc::malloc and libc::free for allocation and deallocation. It works fine for small allocations, but when I try to allocate a large Vec<u8> with over 1GB, my program crashes with a segmentation fault. Here is a simplified version of my code:

use core::alloc::{GlobalAlloc, Layout};
use std::ptr::null_mut;

struct TrackingAllocator;

unsafe impl GlobalAlloc for TrackingAllocator {
    unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
        let ptr = libc::malloc(layout.size());
        println!("allocating {} bytes at {:?}", layout.size(), ptr);
        ptr as *mut u8
    }

    unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
        println!("freeing {} bytes at {:?}", layout.size(), ptr);
        libc::free(ptr as *mut libc::c_void);
    }
}

#[global_allocator]
static GLOBAL: TrackingAllocator = TrackingAllocator;

fn main() {
    let mut big = Vec::<u8>::with_capacity(1_500_000_000);
    for i in 0..big.capacity() {
        big.push(i as u8);
    }
}

When I run this program on Linux x86-64 (Ubuntu 22.04, Rust 1.76.0), it prints the allocation message and then segfaults during the loop. The pointer printed by my allocator appears valid. I suspect my handling of the Layout is wrong, or maybe I need to align the pointer to layout.align() or allocate extra bytes. The documentation for alloc says the implementation must "return a pointer suitable for holding layout.size() bytes aligned to layout.align()", but I'm not sure how to do that with libc::malloc. Do I need to use posix_memalign or adjust the pointer?

How can I correctly implement a global allocator that handles large allocations without crashing? Are there other pitfalls I should be aware of when using GlobalAlloc with very large buffers?

I wrote the TrackingAllocator shown above and set it as the global allocator. My expectation was that allocating a large vector would succeed and I would see the allocation and deallocation messages printed to stdout.

Instead, the program segfaults inside big.push after the initial allocation message, and no deallocation occurs. I tried adjusting the requested capacity and running with RUSTFLAGS='-Z sanitizer=address', but the crash still occurs. I suspect misalignment or incorrect use of Layout, but I'm not sure how to fix it.

I'll repeat my question here, since it got lost during the approval from the SG: It doesn't crash on the playground, are you sure it's not the OOM killer? — Chayim Friedman
– Chayim Friedman, Commented Aug 8 at 13:01
On my computer (rust 1.89), this program gets stuck for any allocation size (small or large). Using println!() inside the global allocator will probably allocate, then lead to an infinite recursion. — prog-fh
– prog-fh, Commented Aug 8 at 13:10
Does your code work on the same PC with the standard allocator? — Jmb
– Jmb, Commented Aug 8 at 13:22
Note that when you have multiple questions, you should ask them separately. That being said, you can handle alignment with posix_memalign instead of malloc. — Jmb
– Jmb, Commented Aug 8 at 13:26

prog-fh · Accepted Answer · 2025-08-08 13:44:26Z

Internally println!() relies on some formatting, then has to allocate a buffer.
Thus, using println!() within the allocation function leads to a recursion.

    unsafe fn alloc(
        &self,
        layout: Layout,
    ) -> *mut u8 {
        fn msg(s: &str) {
            unsafe {
                libc::write(1, s.as_ptr() as *const libc::c_void, s.len());
            }
        }
        msg("one\n");
        let ptr = unsafe { libc::malloc(layout.size()) };
        msg("two\n");
        println!("allocating {} bytes at {:?}", layout.size(), ptr);
        msg("three\n");
        ptr as *mut u8
    }

Running the example (whatever the allocation size) displays

one
two
one
two
<~~~~ stuck here

Looking at the CPU activity while stuck shows that the process is blocked (probably on an internal lock around stdio).
This simple change

        // println!("allocating {} bytes at {:?}", layout.size(), ptr);
        msg(&format!(
            "allocating {} bytes at {:?}\n",
            layout.size(),
            ptr
        ));

prevents the stdio lock, but leads to an infinite recursion on the allocation function.

one
two
one
two
one
two
one
two
...
one
two
one
Segmentation fault         (core dumped) cargo run

If you want to display some messages in the allocation functions, you will have to format them yourself without relying on the standard formatting functions (because they use the allocator).

Collectives™ on Stack Overflow

Why does my custom global allocator crash on large allocations, and how can I handle alignment properly?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related