Can't Get Powershell to assign an array over a certain size

Question

So, I'm trying to create a powershell script that "benchmarks" the speed of GPG using various size files. The part in specific that is causing me trouble is this:

1 $temppath="$($env:Temp)\"
...
4 Measure-Command {$bytes= 10000MB
5 [System.Security.Cryptography.RNGCryptoServiceProvider]$rng = New- Object.System.Security.Cryptography.RNGCryptoServiceProvider
6 $rndbytes = New-Object byte[] $bytes
7 [System.Io.File]::WriteAllBytes("$($temppath)\test.txt", $rndbytes)} | Select-Object -OutVariable gentime | Out-Null;

When $bytes >~ 1.5GB (I use 10000MB to test it), I get an error that, If I'm understanding what's going on correctly, corresponds to the fact that the byte array is indexed by a variable of type int32. The specific error is this:

New-Object : Cannot convert argument "0", with value: "10485760000", for "Byte[]" to type "System.Int32": "Cannot convert value "10485760000" to type "System.Int32". Error: "Value was either too large or too small for an Int32.""
At line:6 char:13
+ $rndbytes = New-Object byte[] $bytes
    + CategoryInfo          : InvalidOperation: (:) [New-Object], MethodException
    + FullyQualifiedErrorId : ConstructorInvokedThrowException,Microsoft.PowerShell.Commands.NewObjectCommand

Is there a way to force powershell to index $rndbytes, when the object is created, with an index of type int64 or uint64? If that's not the issue, I'd love to be shown and have what's exactly going wrong explained.

Do note, if you use, for example $bytes=1000MB (or write $bytes=1GB), the code works perfectly. I suspect I'm hitting a wall at $bytes.index > 2^31 -1 or so.

Thanks for the help~!!

No, an array cannot exceed [int]::MaxValue in length. Output in batches — Mathias R. Jessen
– Mathias R. Jessen, Commented Feb 8, 2021 at 23:56
Dang, that's unfortunate. So just create multiple arrays and process them in batches? — Shinaolord
– Shinaolord, Commented Feb 8, 2021 at 23:56
Create a small buffer array (4KB for example) and then write in chunks of that size until you've reached your desired file size — Mathias R. Jessen
– Mathias R. Jessen, Commented Feb 8, 2021 at 23:57
Time to learn how to do that in powershell then. Do you happen to know if [System.Io.File]::WriteAllBytes overwrites data, or appends? — Shinaolord
– Shinaolord, Commented Feb 9, 2021 at 0:02

Mathias R. Jessen · Accepted Answer · 2021-02-09 00:08:44Z

2

As mentioned in the comments, arrays in .NET cannot exceed 2^31-1 items in length, because the index value used for arrays is [int] - the max value of which is 2^31-1.

It's not really a problem, because buffering 2 billion items in memory is terribly inefficient anyways - if you want 10GB of random data, you generate and write it to disk in chunks instead:

# Set file size
$intendedSize = 10GB

# Create output buffer
$buffer = [byte[]]::new(4KB)

# Create RNG
$rng = [System.Security.Cryptography.RNGCryptoServiceProvider]::Create()

# Create temp file, open writable filestrem
$tempFile = [System.IO.Path]::GetTempFileName() |Get-Item
$fileStream = $tempFile.OpenWrite()

do {
    # Fill buffer, write to disk, rinse-repeat
    $rng.GetBytes($buffer)
    $fileStream.Write($buffer, 0, $buffer.Length)
} while($fileStream.Length -lt $intendedSize)

# Dispose if filestream + RNG
$fileStream.Dispose()
$rng.Dispose()

4KB is ususally a good general buffer size on Windows, since 4KB is the default cluster size on NTFS volumes and most modern CPUs have L1 cache sizes divisible by 4KB

answered Feb 9, 2021 at 0:08

Mathias R. Jessen

178k13 gold badges175 silver badges234 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Shinaolord Over a year ago

What is the difference between $rng = [System.Security.Cryptography.RNGCryptoServiceProvider]::Create() and $rng = new [System.Security.Cryptography.RNGCryptoServiceProvider]() and

[System.Security.Cryptography.RNGCryptoServiceProvider]$rng = New-Object System.Security.Cryptography.RNGCryptoServiceProvider

? In all honesty, I'm very new to Powershell, I've only really used it for very basic things before this. Also, is Powershell very similar to C#? It seems like it is, and if it is, I might just undertake learning to use C# as a way to supplement my Powershell abilities

Mathias R. Jessen Over a year ago

@Shinaolord $rng = new [TypeName]() is not valid PowerShell. New-Object TypeName is essentially the same as [TypeName]::new(), but since the RNGCryptoServiceProvider exposed a static Create() method, the author probably intended for users to use that - so that's what I used :-) PowerShell shares the same underlying base class library as C# (.NET) and some syntax element are heavily "borrowed" from C#, but they two distrinct languages.

Shinaolord Over a year ago

I just didn't see the ::Create() method when I looked at microsoft's documentation page for RNGCryptoServiceProvider, so I wasn't sure (:. I really only know Mathematica and less so matlab, and have dabbled in C++ and Python, but by no means am I knowledgeable in those languages. Thanks for the help!

Collectives™ on Stack Overflow

Can't Get Powershell to assign an array over a certain size

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related