1

I need to allocate lots of memory to emulate memory consumption by .NET app. I've expected that new byte[1000*1024*1024] would allocate all memory at once, but it is not happening.

For example, consider following code:

var bytes = 1000*1024*1024;
var memory = new byte[bytes];
memory[bytes - 1] = 16;

//Step 1
Console.ReadLine();

for (int i = 0; i < memory.Length / 2; i++)
{
    memory[i] = 16;
}

//Step 2
Console.ReadLine();

for (int i = memory.Length / 2; i < memory.Length; i++)
{
    memory[i] = 16;
}

According to Process Explorer, no memory is allocated till Step 1, and before Step 2 only half of the expected memory is allocated. Only after Step 2 all 1000*1024*1024 bytes are allocated. Compiled both in Debug and Release VS configurations.

So the questions are:

  1. Why memory is not allocated in process at once?
  2. How to force process to allocate memory (except of iterating whole array)?

UPDATE:

I've inspected the memory consumption via Resource Monitor tool, and the "Commit" section shows the correct 1000 Mb, however "Working set" behaves as I've described above. As my task is to emulate real load, I suppose I need to load actual physical memory, not virtual one.

Improve this question
5
  • 2
    Are you looking at virtual memory or physical memory? Commented Jul 30, 2014 at 13:33
  • The compiler is free to reorder instructions as long as everything happens the same way from the POV of a single-threaded program. And that's before you even get to how virtual memory works (the OS doesn't necessarily give you any memory until you actually start using it - and it can take it away just as easily). How are you measuring the memory usage anyway? A tool like CLRProfiler might be a lot handier than e.g. task manager. Commented Jul 30, 2014 at 13:33
  • 1
    @Luaan: the first part is probably not the problem. The code clear accesses the array before the Console.ReadLine() so the compiler can't reorder the instructions in such a way as to violate that, at least not the .NET compiler (which I doubt would so aggressively optimize -- indeed, in this program, it could simply never allocate the array or run the loops since they have no side effects!). Commented Jul 30, 2014 at 13:38
  • @siride Yup, in that case it's rather obvious this is about virtual memory. The question is, Pavel, why do you care about whether it's virtual or physical memory? You can measure either. Do understand that even if there is data, nobody says it's in RAM - it might be in a swap file, for example. And parts of the array might be swapped out while other parts are in physical RAM. This is pretty much the whole point of virtual memory in the first place. Note that even "physical" use is still virtual memory, there isn't much of a way to bypass virtual memory. Commented Jul 30, 2014 at 15:55
  • 1
    In any case, this will not tell you anything about the performance of an application. A single huge byte array will behave significantly differently from a few thousand different objects with different allocation / deallocation patterns etc. What are you actually trying to figure out? Commented Jul 30, 2014 at 15:57

2 Answers 2

Reset to default
2

This is standard behavior on a demand-paged virtual memory operating system like Windows. Allocating memory with the new operator only allocates address space. Just numbers to the processor, one for every 4096 bytes. A page.

The commit size grows, you have a solid promise that you can get the physical storage you need once you start addressing the array elements. Windows has reserved space in the paging file for your allocation.

But the RAM allocation doesn't happen until you actually start using the array elements. That causes a paging fault, you'll get one for every 4096 elements. The operating system is now forced to make the physical storage available. RAM.

That tends to increase the working set, the amount of RAM your program uses. That is not guaranteed to happen, when the operating system is under pressure and needs to distribute the available RAM across memory-hungry processes then it may need to forcibly discard a page to make it available to your program. Typically just throwing the content away when the page contains code or is mapped to a file (like code is). Or it backs up the content to the paging file. To be retrieved again, later, when the process needs to have the page available in RAM again.

The kind of page faults you normally get for new allocations are soft page faults. They are dealt with quickly, the operating system pulls a page of RAM from a pool. You get a hard page fault if another page needs to be discarded and written to the paging file or your page was swapped out and needs to be swapped back in. That can slow a program down significantly, a problem called "thrashing". Machines that don't have enough RAM and are overloaded tend to be slow due to thrashing.

A consequence of this operating system design is that there is no real difference between a file and memory anymore. When you use memory then you also use a file. The paging file usually. And the other way around, when you use a file then you actually use memory. The file system cache. RAM is just a convenient way to avoid having to access the disk. The more RAM the machine has, the less often you'll hit the disk. Programmers tend to fret too much over this, trying too hard to keep data in memory when they also have a choice (or no given choice by an api) to use a file. And get into trouble like OutOfMemoryException or causing thrashing.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

2

I tried this program in LinqPad while having ProcessExplorer open. It indeed increases memory right at step 1, but only the private bytes (which is virtual memory). The working set stays more or less the same size. The reason for this is that private bytes here represents virtual memory. That's allocated on-demand, but won't be backed by physical memory until needed. The working set represents actual physical memory used. Once you start using the array, the parts of it that were not allocated in physical memory will be allocated and the working set size will increase. Notice that it probably won't reach the full size of the array since that would be inefficient. Instead, the parts that haven't been touched in a while will be swapped out so that the working set doesn't grow to an enormous size.

EDIT: in answer to your update, you have few options:

  1. Change the size of your working set (http://msdn.microsoft.com/en-us/library/system.diagnostics.process.maxworkingset(v=vs.110).aspx).
  2. Use threads that are accessing different parts of the array at the same time, so as much of it needs to be loaded into memory at a time as possible.
  3. A combination of the two.
Improve this answer

4 Comments

Actually I have different behavior - according to Resource Monitor tool the "Commit" section always shows 1000 MB, but "Working set" and "Private" are 10 MB and 3 MB before Step 1, and 522 MB and 515MB before Step 2.
How changing the working set size will help in my case? And about "different parts of array" - which parts should they be? Is the only guaranteed way to cover whole array?
@PavelK: yes, you have the right columns...I messed them up, but the point stands. Some columns measure actual RAM usage and other columns measure virtual memory usage. Working set measures RAM usage.
@PavelK: changing working set will increase the amount of RAM the process is allowed to use and reduce the tendency for Windows to swap out virtual memory space. Another option, which is generally a really bad idea, is to pin pages to RAM, which can be done by looking at this answer: stackoverflow.com/a/4700926/394487.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.