stack overflow error in delphi

Question

I have a procedure which calls several functions:

procedure TForm1.Button1Click(Sender: TObject);
var
  rawData: TRawData;
  rawInts: TRawInts;
  processedData: TProcessedData;
begin
  rawData := getRawData();
  rawInts := getRawInts(rawData);
  processedData := getProcessedData(rawInts);
end;

The data types are defined like this:

TRawData = array[0..131069] of Byte;
TRawInts = array[0..65534] of LongInt;
TProcessedData = array[0..65534] of Double;

running the program with just:

rawData := getRawData();
rawInts := getRawInts(rawData);

Works totally fine. However, when I try to run:

getProcessedData(rawInts)

I get a stackoverflow error. I don't see why this is. The function code for getProcessedData is very simple:

function getProcessedData(rawInts : TRawInts) : TProcessedData;
var
  i: Integer;
  tempData: TProcessedData;
  scaleFactor: Double;
begin
  scaleFactor := 0.01;

  for i := 0 to 65534 do
    tempData[i] := rawInts[i] * scaleFactor;

  Result := tempData;
end;

Why is this causing an error ?

When taking the code out of the getProcessedData function and putting it manually in the procedure this error still occurs. In addition to this, the error seem to be thrown when entering getRawData ... I have no clue why this is. — Tim M
– Tim M, Commented Mar 27, 2014 at 19:57
Local variables are allocated on the stack, and presumably, array[0..65534] of double; is too big, causing a stack overflow. — Blorgbeard
– Blorgbeard, Commented Mar 27, 2014 at 19:57
You are correct. That's a bit of a problem really, as I really need the array of doubles to be that long. Can I extend the length of the stack in some way? (That might be a really stupid question..) — Tim M
– Tim M, Commented Mar 27, 2014 at 20:02
Can I use a global variable to save space on the stack because I am not creating the large dataset inside the function? — Tim M
– Tim M, Commented Mar 27, 2014 at 20:13

Community · Accepted Answer · 2017-05-23 12:06:35Z

16

The default maximum stack size for a thread is 1 MB. The three local variables of Button1Click total 131,070 + 65,535 * 4 + 65,535 * 8 = 917,490 bytes. When you call getProcessedData, you pass the parameter by value, which means that the function makes a local copy of the parameter on the stack. That adds SizeOf(TRawInts) = 262,140 bytes to bring the stack to at least 1,179,630 bytes, or about 1.1 MB. There's your stack overflow.

You can reduce the stack use by passing the TRawInts array by reference instead. Then the function won't make its own copy. Zdravko's answer suggests using var, but since the function has no need to modify the passed-in array, you should use const instead.

function getProcessedData(const rawInts: TRawInts): TProcessedData;

Naively, we might expect the tempData and Result variables in getProcessedData to occupy additional stack space, but in reality, they probably won't. First, large return types typically result in the compiler changing the function signature, so it would act more like your function were declared with a var parameter instead of a return value:

procedure getProcessedData(rawInts: TRawInts; var Result: TProcessedData);

Then the call is transformed accordingly:

getProcessedData(rawInts, processedData);

Thus, Result doesn't take up any more stack space because it's really just an alias for the variable in the caller's frame.

Furthermore, sometimes the compiler recognizes that an assignment at the end of your function, like Result := tempData, means that tempData doesn't really need any space of its own. Instead, the compiler may treat your function as though you had been writing directly into Result all along:

begin
  scaleFactor := 0.01;

  for i := 0 to 65534 do
    Result[i] := rawInts[i] * scaleFactor;
end;

However, it's best not to count on the compiler to make those sorts of memory-saving changes. Instead, it's better not to lean so heavily on the stack in the first place. To do that, you can use dynamic arrays. Those will move the large amounts of memory out of the stack and into the heap, which is the part of memory used for dynamic allocation. Start by changing the definitions of your array types:

type
  TRawData = array of Byte;
  TRawInts = array of Integer;
  TProcessedData = array of Double;

Then, in your functions that return those types, use SetLength to assign the length of each array. For example, the function we've seen already might go like this:

function getProcessedData(const rawInts: TRawInts): TProcessedData;
var
  i: Integer;
  scaleFactor: Double;
begin
  Assert(Length(rawInts) = 65535);
  SetLength(Result, Length(rawInts));

  scaleFactor := 0.01;

  for i := 0 to High(rawInts) do
    Result[i] := rawInts[i] * scaleFactor;
end;

edited May 23, 2017 at 12:06

CommunityBot

11 silver badge

answered Mar 27, 2014 at 20:35

Rob Kennedy

164k23 gold badges288 silver badges481 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Tim M Over a year ago

Hi Rob, thanks for your answer, I have a question, you said that "since the function has no need to modify the passed-in array, you should use const instead." Do you mean in terms of length? because does the multiplication by scaleFactor not modify the array?

Ken White Over a year ago

@Tim: No, it modifies the content of the array, not the array itself. They're not the same thing. Passing a const SL: TStringList does not mean you can't add items to the stringlist; it means you can't free the stringlist itself and create a new instance in that variable.

Rob Kennedy Over a year ago

No, multiplying two values doesn't modify either of those values. See for yourself: Inspect the contents of rawInts before and after you call the function, and you'll see that the values in that array haven't changed. If you were assigning new values into the same array you were reading from, then the array couldn't be const, but in your case, you're assigning the product into an entirely separate array (tempData or Result).

Rob Kennedy Over a year ago

That's not what I was referring to, @Ken. When I wrote about using const instead of var, there were no reference types involved. Delphi doesn't allow any of the elements of a const static-array parameter to be modified, does it?

Tim M Over a year ago

I see, I was being stupid, its not rawInts which is modified at all, only result. I get this now, I'll change my code. Thanks for your time.

|

David Heffernan · Accepted Answer · 2014-03-27 20:32:51Z

2

These objects are all very large. And you appear to be allocating them as local variables. They will reside on the stack which has an fixed size, by default 1MB on Windows. You have allocated enough of these large objects in various parts of the call stack to exceed the 1MB limit. Hence a stack overflow.

Another problem in your code is the way you pass these objects as parameters. Passing large objects as value parameters results in copies being made. Copying an integer or two is nothing to worry about. Copying 65 thousand doubles is wasteful. It hurts performance. Don't do that. Pass references to large objects. Passing as const parameters achieves that.

The stack is well suited for small objects. It is not suited to these large objects. Allocate these objects on the heap. Use dynamic arrays: array of Integer, TArray<Integer> etc.

Do not increase the default stack size for your process. Especially in modern times of multi-core machines this is a recipe for out of memory errors.

Do not use magic constants. Use low() and high() to obtain array bounds.

Do pass input parameters by const. This allows the compiler to make optimisations that are significantly beneficial.

edited Mar 27, 2014 at 20:32

answered Mar 27, 2014 at 20:21

David Heffernan

616k46 gold badges1.1k silver badges1.5k bronze badges

4 Comments

Tim M Over a year ago

So I declare my arrays, within the procedure, using dynamic arrays, then how do I pass these to the functions? myFunc(const myArray : array of double) ??

David Heffernan Over a year ago

That's right. That's an open array which is good. Declare as dynamic array for the actual variables. TArray<T> if you use modern Delphi.

Tim M Over a year ago

Ok, Thank you for your patience, I come from an electronics background and this is my first real time coding using large data sets, I can see I don't have enough appreciation for the limits of a PC, I keep seeing it as an infinite resource.

David Heffernan Over a year ago

All I can say is don't do anything that Zdravko advised. And read Rob's excellent answer carefully.

TLama · Accepted Answer · 2014-03-27 20:53:58Z

The key issue here is the size of your arrays.

If you use SizeOf you will see that they are probably larger than you think:

program Project3;

{$APPTYPE CONSOLE}

uses
  SysUtils;

type
  TRawData = array[0..131069] of Byte;
  TRawInts = array[0..65534] of Longint;
  TProcessedData = array[0..65534] of Double;
begin
  try
    writeln('TProcessedData:':20, SizeOf(TProcessedData):8);
    writeln('TRawData:':20, SizeOf(TRawData):8);
    writeln('TRawInts:':20, SizeOf(TRawInts):8);
    writeln('Total:':20, SizeOf(TRawInts) + SizeOf(TProcessedData) + SizeOf(TRawData):8);

    readln;
  except
    on E:Exception do
      Writeln(E.Classname, ': ', E.Message);
  end;
end.

Output:

 TProcessedData:  524280
       TRawData:  131070
       TRawInts:  262140
          Total:  917490

So most of the 1MB stack is consumed by the arrays. As some of the stack will already be allocated, you get a stack overflow.

This can be avoid by using dynamic arrays, which have their memory allocated from the heap.

TRawData = array of Byte;
TRawInts = array of Longint;
TProcessedData = array of Double;

...
SetLength(TProcessedData, 65535); 
...

Collectives™ on Stack Overflow

stack overflow error in delphi

3 Answers 3

7 Comments

4 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

7 Comments

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related