Curl function cannot parse proxy coming from a variable in bash

Question

I have a proxy txt file with the format:

102.129.249.120:3128
102.129.249.120:8080
101.4.136.34:8080
103.228.117.244:8080
etc

and I am trying to create a bash script that will do (for example): curl -x "$IP" google.com.
Unfortunately, curl is giving me an unsupported proxy syntax for all the proxies.
Any ideas?
BTW, I really doubt this question has been repeated as I have tried everything else to no avail.

My script:

Number=$(wc -l < ProxyList.txt)



for ((i=1;i<=$Number;++i))  do
ip=$(head -n ${i} ProxyList.txt | tail -n +${i})
curl -p -x "$ip" 'webpage' -H 'user-agent' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Cookie: wpml_referer_url=referer; _icl_current_language=es; PHPSESSID=tpikve1vl4ued06i082vprqdo1' -H 'If-Modified-Since: Mon, 16 May 2016 07:27:13 GMT' -H 'If-None-Match: "3d6-532f08d9d7640-gzip"' -H 'Cache-Control: max-age=0' -m 6
done

A small sample of my proxy list:

102.129.249.120:3128
102.129.249.120:8080
101.4.136.34:8080
103.228.117.244:8080
103.253.27.108:80
104.45.188.43:3128
104.250.34.179:80
105.27.238.161:80
104.154.143.77:3128
110.243.20.2:9999
111.68.26.237:8080
106.104.151.142:58198
113.252.95.19:8197
115.231.31.130:80
118.69.50.154:80
118.69.50.154:443
119.81.189.194:80
119.81.189.194:8123
119.81.199.81:8123
119.81.199.83:8123
119.81.199.80:8123
12.139.101.100:80
12.139.101.101:80
119.81.199.85:31288
119.81.199.86:8123
119.81.199.87:8123
12.139.101.102:80
124.156.98.172:443
13.228.91.252:3128
138.197.157.32:3128
138.197.157.32:8080
138.68.240.218:8080
138.68.240.218:3128
138.68.60.8:8080
138.68.60.8:3128

I'd use xargs, like xargs -I{} curl -x {} ... <ProxyList.txt. But if ProxyList.txt have DOS line endings, it'd fail either — oguz ismail
– oguz ismail, Commented Jun 16, 2020 at 16:55
of how to use xargs I tried echo $ip | xargs curl url and it still didn't work. Same error. — DAVID
– DAVID, Commented Jun 16, 2020 at 17:14
What's the error you get? Also you're missing a semicolon right before the do of your for loop. — tectux
– tectux, Commented Jun 16, 2020 at 17:52

tectux · Accepted Answer · 2020-06-16 21:31:22Z

3

Your input file has carriage return characters at the end of each line.
Each line in your input file ends with \r\n instead of just \n.

You can check with od:

$ head -1 ProxyList.txt | od -c
0000000   1   0   2   .   1   2   9   .   2   4   9   .   1   2   0   :
0000020   3   1   2   8  \r  \n
0000026

So in your script, $ip has in fact a value of 102.129.249.120:3128\r.

You can remove the \r characters with tr for example:

while read proxy; do
  curl -p -x $proxy $webpage
done < <( tr -d '\r' < ProxyList.txt )

edited Jun 16, 2020 at 21:31

answered Jun 16, 2020 at 21:20

tectux

1915 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Oliver Gaida · Accepted Answer · 2020-06-16 19:13:55Z

1

try this:

for ip in $(cat ProxyList.txt)
do
   curl -p -x "$ip" 'webpage' -H 'user-agent' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Cookie: wpml_referer_url=referer; _icl_current_language=es; PHPSESSID=tpikve1vl4ued06i082vprqdo1' -H 'If-Modified-Since: Mon, 16 May 2016 07:27:13 GMT' -H 'If-None-Match: "3d6-532f08d9d7640-gzip"' -H 'Cache-Control: max-age=0' -m 6
done

but the problem with curl might be, that should set the environment variables http_proxy and https_proxy like this:

export http_proxy=http://1.2.3.4:3128/
export https_proxy=http://1.2.3.4:3128/

answered Jun 16, 2020 at 19:13

Oliver Gaida

1,94010 silver badges14 bronze badges

11 Comments

DAVID Over a year ago

Same error ... I'm using ubuntu 20.04 could that be the problem?

DAVID Over a year ago

'url: (5) Unsupported proxy syntax in '105.29.64.222:80

Oliver Gaida Over a year ago

please give https://stackoverflow.com/questions/9445489/performing-http-requests-with-curl-using-proxy a try

Oliver Gaida Over a year ago

first you set the proxy-environment-variables and then call curl http://www.google.com etc.

DAVID Over a year ago

Yes. It only works if I input the ip without the variable (manually)

|

lab9 · Accepted Answer · 2020-06-16 20:09:19Z

0

As per the curl man page, the -x (or --proxy) switch can be prefixed with the protocol in front of the argument (if omitted, I assume that it defaults to http://):
-x, --proxy [protocol://]host[:port]

A simple bash script with xargs would look like:

#!/bin/bash
webpage=${1:-http://google.com}
cat ProxyList.txt \
| xargs -n1 -I{} curl -p -x http://{} "$webpage" -H 'user-agent' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Cookie: wpml_referer_url=referer; _icl_current_language=es; PHPSESSID=tpikve1vl4ued06i082vprqdo1' -H 'If-Modified-Since: Mon, 16 May 2016 07:27:13 GMT' -H 'If-None-Match: "3d6-532f08d9d7640-gzip"' -H 'Cache-Control: max-age=0' -m 6

edited Jun 16, 2020 at 20:09

answered Jun 16, 2020 at 20:03

lab9

5862 silver badges8 bronze badges

4 Comments

DAVID Over a year ago

Still doesn't??? Same error. What version of ubuntu do you have? Or do you have another distro?? I really don't know.

DAVID Over a year ago

'url: (5) Unsupported proxy syntax in '46.35.184.187:61003

lab9 Over a year ago

Debian 5.4.19 / curl 7.68.0

lab9 Over a year ago

I'm getting "curl: (56) Received HTTP code 400 from proxy after CONNECT" from your proxies. That seems to be related to your "-p" switch (tunneling). Seems that the proxies don't support that? On the other hand, it shows that the "-x" part of the statements work.

Collectives™ on Stack Overflow

Curl function cannot parse proxy coming from a variable in bash

3 Answers 3

Comments

11 Comments

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

11 Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related