Parsing file in Powershell

Question

I have the following raw content in a file. I am trying just print the list of all urls. I have kind of wrote some script. Getting content (reading) from the file and using ForEach line in lines - but do not know how to filter just the Url from the content. Any thoughts ?

Line 18942:         "url": "http://harvardpolitics.com/tag/brussels/",
Line 18994:         "url": "http://203.36.101.164/4f64555b4217b47b7c64b3fec19e389b/1502455203/Telstra/Foxtel-Vod/fxmultismvod5256/store2/ON307529/ON307529_hss.ism/QualityLevels(791000)/Fragments(video=9900000000)"
Line 19044:         "url": "https://www.gucci.com/int/en/ca/women/handbags/womens-shoulder-bags-c-women-handbags-shoulder-bags?filter=%3ANewest%3Acolors%3AGold%7Ccb9822",
Line 19096:         "url": "https://bagalio.cz/batohy-10l?cat=3p%3D1urceni%3D2582p%3D1kapsa_ntb_velikost%3D2179p%3D1manufacturer%3D1302p%3D1color%3D84p=1kapsa_ntb_velikost=2192",
Line 19148:         "url": "http://www.csillagjovo.gportal.hu/gindex.php?pg=31670155",
Line 19200:         "url": "http://www.copiersupplystore.com/hp/color-laserjet-4700dn/j7934a-j7934ar",

Where do the line numbers come from - are they in the file or are they your addition to it? It looks like part of a JSON file - If so, use ConvertFrom-Json. — TessellatingHeckler
– TessellatingHeckler, Commented Aug 11, 2017 at 23:54
Absolutely true, they are the response from an API as JSON blob. I have them filtered in Notepad++ with "url" and a list of around 400 urls showed up. I tried to parse them nothing was working. I will try with ConvertFrom-Json and see if it works. — user1911509
– user1911509, Commented Aug 12, 2017 at 20:14
Invoke-RestMethod will implicitly convert API responses from JSON into PowerShell objects, btw, instead of Invoke-WebRequest — TessellatingHeckler
– TessellatingHeckler, Commented Aug 12, 2017 at 22:24
Invoke-RestMethod worked and it did come in handy and a better solution than Invoke-WebRequest. Appreciate your help. — user1911509
– user1911509, Commented Aug 13, 2017 at 14:58

Olaf Reitz · Accepted Answer · 2017-08-11 23:32:57Z

2

One way could be the substring method another version could be some regex.

$Text = Get-Content D:\Test\test.txt
foreach ($Line in $Text) {
    # SubString Version
    $FirstIndex = $Line.IndexOf('http')
    $URLLength = ($Line.LastIndexOf('"') - $FirstIndex)
    $Line.Substring($FirstIndex, $URLLength)

    # Regex Version 
    $Regex = '(http[s]?|[s]?ftp[s]?)(:\/\/)([^\s,]+)'
    ([regex]::Matches($Line,$Regex)).Value.TrimEnd('"')([^\s,]+)')).Value.TrimEnd('"')
}

answered Aug 11, 2017 at 23:32

Olaf Reitz

7143 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user1911509 Over a year ago

I have tried it but doesn't output anything. I have tried to output to a file - it is empty.

KarlGdawg · Accepted Answer · 2017-08-11 23:33:38Z

2

Try this out to just get the urls:

$content = Get-Content <file-with-output> # or other way of getting the data

$urls = $content | ForEach-Object { ($_ -replace ".+?(?=http.+)","").Trim('",')}

Edit: Added $urls to catch result.

answered Aug 11, 2017 at 23:33

KarlGdawg

3511 gold badge4 silver badges9 bronze badges

4 Comments

TessellatingHeckler Over a year ago

Just throwing in $_ -replace '^.*(http[^"]+).*$', '$1' as a simpler regex approach (no lookaround, no trim)

KarlGdawg Over a year ago

My regex is a bit weak, thank you for showing me a better way.

user1911509 Over a year ago

I tried the regex but it only outputs one Urls with line 19200. Is it something with the data copied over to the file. As I have mentioned above "response from an API as JSON blob. I have them filtered in Notepad++ with "url" and a list of around 400 urls showed up. I tried to parse them nothing was working." - I will also try with 'ConvertFrom-Json'.

user1911509 Over a year ago

Thank you all, I have used the Convert-Json and all worked fine. All the above solutions worked well for parsing Urls and output to a file. I appreciate your help in resolving this.

TessellatingHeckler · Accepted Answer · 2017-08-11 23:54:34Z

2

$Urls = Get-Content file.txt | ForEach-Object { $_.Split('"')[3] }

answered Aug 11, 2017 at 23:54

TessellatingHeckler

29.3k4 gold badges55 silver badges96 bronze badges

Collectives™ on Stack Overflow

Parsing file in Powershell

3 Answers 3

1 Comment

4 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related