1

I am newbie to Powershell. Need a logic for CSV automation. I have a CSV log file contains large number of API calls.

enter image description here

I need to go row by row and segregate the data, output should be like below. Sum of calls count and average of response time to be updated.

enter image description here

I have written complicated If else conditions for different types of API calls and able to take the scenario name and other values from the csv. My pain starts here, struggling to come to conclusion to move forward. Can i create an array and store all the values then do all the calculation later or write the values in another csv then do all the calculation to find the Count and average response time?

If i choose array, scenario should not be duplicated. For me its really hard to take a decision without knowing the available cmdlets for array and CSV. Please throw some light..

Thanks in advance...

4
  • I commend to your attention the Measure-Object cmdlet. Commented Dec 16, 2019 at 16:38
  • 1
    PLEASE don't post images of text/code/data/errors ... why force others to squint to read it? why force others to test things by typing in what you already have as text? Commented Dec 16, 2019 at 18:14
  • Oh am sorry, i just copied from the excel and pasted here. But it converted into image by default. Will avid in future. Commented Dec 17, 2019 at 17:13
  • Thanks @JeffZeitlin Will check and update.. Commented Dec 17, 2019 at 17:14

1 Answer 1

2

Here is an approach you can use a combination of c# available to Powershell (which can be MUCH more efficient handling larger files and data).

The first component is you need some consistent logic to isolate the API category you want each URL to be assigned. From your screenshots, sometimes it seems you use last segment of the URL but others it is some path in the middle of the resource.

Here is just a quick approach where you pass in an array of categories, and if it can be matched to URI in any way, then that category is used. Otherwise, the URI stands as its own category. Please replace with whatever logic you want here.

function Get-ApiCategory {
    param([string[]] $Categories, [string] $Text)
    foreach ($c in $Categories) {
        if ($Text.IndexOf($c) -gt 0) {
            return $c
        }
    }
    return $Text # Not found
}

Then, here is a method that (1) reads the large CSV file row-by-row and uses basic parsing logic (since your source data seems simple enough) without loading the full file into memory, and then (2) exports a CSV file with summary data.

function Write-SummaryToFile {
    param([string[]] $Categories, [string] $InputFile, [string] $Output)

    # Parse the file line-by-line (optimize for memory)
    $result = @{}
    $lineNum = 0
    Write-Host $InputFile
    foreach ($line in [System.IO.File]::ReadLines($InputFile)) {
        if ($lineNum++ -lt 1) { continue } # Skip header
        $cols = $line.Split(',')
        $category = Get-ApiCategory $Categories $cols[0]
        $new = @{
            Category = $category
            Count = [int]$cols[1]
            AvgResponse = [double]$cols[2]
        }
        if ($result.ContainsKey($category)) {
            $weighted = $result[$category].AvgResponse * $result[$category].Count
            $result[$category].Count += $new.Count
            $result[$category].AvgResponse = ($weighted + $new.AvgResponse * $new.Count) / $result[$category].Count;
        } else {
            $result[$category] = $new
        }
    }

    # Output to file
    if (Test-Path $Output) { Remove-Item $Output }
    try {
        $stream = [System.IO.StreamWriter] $Output
        $stream.WriteLine('Scenario,Count,Avg_Response_Time')
        $result.Values | ForEach-Object { $stream.WriteLine([string]::Format("{0},{1},{2}", $_.Category, $_.Count, $_.AvgResponse.ToString("0.##"))) }
    }
    finally {
        $stream.Dispose()
    }

}

Then, you are able to call these methods in an example like this:

$categories = @('MoveRequestQueue', 'DeliveryDate')
Write-SummaryToFile $categories 'c:\dev\scratch\ps1\test.csv' 'C:\dev\scratch\ps1\Output.csv'
Sign up to request clarification or add additional context in comments.

2 Comments

Thanks Jason for your efforts and time.. will try and update here
Thank you Jason.. Its working fine. I need to update my code.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.