Powershell pull a domains from a list of urls

Question

I have a list of urls in a text file

I want to pull just the domain out.

Running the following:

(Get-Content "c:\temp\test.txt") | ForEach-Object {$url = New-Object System.uri $_;$url.Host} > d:\test.txt

It generates this error:

New-Object : Exception calling ".ctor" with "1" argument(s): "Invalid URI: The format of the URI could not be determined."

And the text file is just that domain (and just the domain, not the path following the domain) over and over again.

Any thoughts on where I'm monkeying this up?

Sample File:

lrp.mxptint.net
lrpush.apxlv.com
lsa.measure.ly
lsa.measure.ly\lsa.js?1449665894174&meta_keywords=Life%2C%20lifestyle%2C%20fashi‌on%2C%20beauty%2C%20Vancouver%2C%20British%20Columbia%2C%20family%2C%20relationsh‌ips%2C%20home%2C%20blogs%2C%20news%2C%20jobs%2C%20content%2C%20pets%2C%20food%2C%‌20beauty&autofire=true&windowlocation=www.vancouversun.com\life\diversions lsa.measure.ly\lsa.js?1449665960913&meta_keywords=horoscopes&autofire=true&windo‌wlocation=news.nationalpost.com\category\life

Sure, I had it in my original post it it wouldn't let me post it. lrp.mxptint.net lrpush.apxlv.com lsa.measure.ly lsa.measure.ly\lsa.js?1449665894174&meta_keywords=Life%2C%20lifestyle%2C%20fashion%2C%20beauty%2C%20Vancouver%2C%20British%20Columbia%2C%20family%2C%20relationships%2C%20home%2C%20blogs%2C%20news%2C%20jobs%2C%20content%2C%20pets%2C%20food%2C%20beauty&autofire=true&windowlocation=www.vancouversun.com\life\diversions lsa.measure.ly\lsa.js?1449665960913&meta_keywords=horoscopes&autofire=true&windowlocation=news.nationalpost.com\category\life — S. Ghost
– S. Ghost, Commented Dec 10, 2015 at 20:15
If it were me, I'd look up "regex domain from url" and convert an example to powershell's regex syntax — Chris N
– Chris N, Commented Dec 10, 2015 at 20:19
That last URL is not valid in that all the slashes are backwards. Is it that way in your text file? — TheMadTechnician
– TheMadTechnician, Commented Dec 10, 2015 at 21:07

djs · Accepted Answer · 2015-12-10 20:28:10Z

3

By definition, a URI starts with a scheme (usually the protocol, such as http). As such, the .Net URI constructor expects the string to start with the protocol. For example, http://www.example.com. You will get the Invalid URI exception if you pass in www.example.com, etc.

edited Dec 10, 2015 at 20:28

answered Dec 10, 2015 at 20:19

djs

1,7102 gold badges17 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

S. Ghost Over a year ago

So if I add http:// to the start of every line (strip it out first and then add it back), it should be good to go?

TheMadTechnician · Accepted Answer · 2015-12-10 21:24:03Z

1

Ok, djs gave an answer stating why what you are doing doesn't work, and Buxmaniak gave a good functional answer on how to do it via URI object and RegEx both. I just wanted to post a simplified version that works for the sample that you provided...

Get-Content "c:\temp\test.txt" | %{[URI]"http://$_"} | Select -Expand Host | Set-Content D:\Test.txt

That will add the http:// to the beginning of the string, and cast it as a URI. Then it expands the Host property from each URI object, and saves it all to the file you specified.

answered Dec 10, 2015 at 21:24

TheMadTechnician

36.5k3 gold badges48 silver badges63 bronze badges

1 Comment

djs Over a year ago

well, hey, the OP asked what was wrong, not for a working solution... :p

Buxmaniak · Accepted Answer · 2015-12-10 22:55:18Z

0

This is a solution based on [System.Uri] ...

@((Get-Content -Path "C:\temp\text.txt") | ForEach-Object { if ($_.Trim()) { (New-Object -TypeName System.Uri ("http://{0}" -f (($_.Trim()) -replace "\\", "/"))).Host } } ) | Sort-Object -Unique

... or if you like RegEx, you should try this ...

@((Get-Content -Path "C:\temp\text.txt") | ForEach-Object { if ($_ -match "^([\w\.-]*)") { $matches[1] } }) | Sort-Object -Unique

In both cases the result is ...

lrp.mxptint.net
lrpush.apxlv.com
lsa.measure.ly

edited Dec 10, 2015 at 22:55

answered Dec 10, 2015 at 21:09

Buxmaniak

4782 silver badges5 bronze badges

3 Comments

TheMadTechnician Over a year ago

Even easier for the URI code: GC $File | %{[URI]"http://$($_ -replace "^.*//")"} | Select -Expand Host (I stripped and re-added the 'http://' because his sample does not have that so this works for both your sample and the OP's). And simpler RegEx solution: GC $File -Replace "^.*?://([^/]*)(/.*)$", '$1'

TheMadTechnician Over a year ago

It worked for your original example. It doesn't work against the OP's example. This will do it: GC $File -replace "^(?:.*?:\/\/)?(.+?)(\\.*$|\/.*$|$)", '$1'

Buxmaniak Over a year ago

Get-Content doesn’t know a parameter replace. (GC $File) -replace ... would possibly working.

Collectives™ on Stack Overflow

Powershell pull a domains from a list of urls

3 Answers 3

1 Comment

1 Comment

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

1 Comment

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related