Running curl in a loop results in unexpected output

Question

In curl, I do something like this to check if a URL is available:

$ curl -u admin:secret -s http://test.app.com/doSomething -o /dev/null -w "%{http_code} %{url_effective}\\n"
200 http://test.app.com/doSomething

I have a list of hosts and paths I want to check regularily using a GET request and the output should show me the response status and the url that was called. So, for a set of hosts tests.app.com, a1.app.com, p.app.com and the paths "/doSomething" and "doSomethingElse" I would run the script and expect the following output

200 http://test.app.com/doSomething
200 http://test.app.com/doSomethingElse
200 http://a1.app.com/doSomething
404 http://a1.app.com/doSomethingElse
200 http://p.app.com/doSomething
500 http://p.app.com/doSomethingElse

This way I can see that the path doSomethingElse does not work.

Now my script looks like this:

#!/bin/bash

HOSTS=(
        'test.app.com'
        'a1.app.com'
        'p.app.com'
)
PATHS=(
        'doSomething'
        'doSomethingElse'
)

USER=admin
PASS=secret

# fixed parameters
auth_header="-u ${USER}:${PASS}"
out_parms="-w \"%{http_code} %{url_effective}\\\n\""

for h in "${HOSTS[@]}"; do
        for p in "${PATHS[@]}"; do
                curl ${auth_header} -s http://${h}/${p} -o /dev/null ${out_parms}
        done
done

It produces several curl calls (similar to the example at the top), but when I run this script, the output looks like the following

"200"000"200"000"200"000"200"000"200"000"200"000

Note: The "200" seems to be the status code and there are 6 requests in my setup.

If I add an echo before the curl command (echo curl ${auth_header} ...), I get a list of commands that I can execute from the command line and which lead to the expected result.

What am I doing wrong?

Brace expansion (i.e. "{a,b}/{1,2}" -> "a/1 a/2 b/1 b/2") might help you simplify your code. — Micha Wiedenmann
– Micha Wiedenmann, Commented Nov 12, 2015 at 12:29
Is it possible for you to completely rewrite your question to something like: "If I run curl ... I get "200 http://...". If I put the same command in a loop, the output changes to "....". — Micha Wiedenmann
– Micha Wiedenmann, Commented Nov 12, 2015 at 14:55
@MichaWiedenmann I rewrote the question. Thank you for the suggestion. Changing the position of the -o and -w parameters does not change the output — Kariem
– Kariem, Commented Nov 16, 2015 at 11:02
I still wonder, whether one can use -o and -w in the same command, they seem to me to conflict. — Micha Wiedenmann
– Micha Wiedenmann, Commented Nov 16, 2015 at 11:18

Matt Slonetsky · Accepted Answer · 2016-11-23 23:00:24Z

Getting bash to print out what it's executing for your curl command with

(set -x; curl ${auth_header} -s http://${h}/${p} -o /dev/null ${out_parms})

the following output is generated:

++ curl -u admin:secret -s http://test.app.com/doSomething -o /dev/null -w '"%{http_code}' '%{url_effective}\\n"'
"301"000++ curl -u admin:secret -s http://test.app.com/doSomethingElse -o /dev/null -w '"%{http_code}' '%{url_effective}\\n"'
"301"000++ curl -u admin:secret -s http://a1.app.com/doSomething -o /dev/null -w '"%{http_code}' '%{url_effective}\\n"'
"301"000++ curl -u admin:secret -s http://a1.app.com/doSomethingElse -o /dev/null -w '"%{http_code}' '%{url_effective}\\n"'
"301"000++ curl -u admin:secret -s http://p.app.com/doSomething -o /dev/null -w '"%{http_code}' '%{url_effective}\\n"'
"301"000++ curl -u admin:secret -s http://p.app.com/doSomethingElse -o /dev/null -w '"%{http_code}' '%{url_effective}\\n"'
"301"000

Looking at

-w '"%{http_code}' '%{url_effective}\\n"'

We can see that the curl command isn't being formatted the way you'd like

As answered here, you should always put double quotes around variable substitutions; though, there's no reason to parameterize out_parms. Putting that definition inline like this:

curl ${auth_header} -s http://${h}/${p} -o /dev/null -w "%{http_code} %{url_effective}\n"

gave me this output in the format you'd expected.

301 http://test.app.com/doSomething
301 http://test.app.com/doSomethingElse
301 http://a1.app.com/doSomething
301 http://a1.app.com/doSomethingElse
301 http://p.app.com/doSomething
301 http://p.app.com/doSomethingElse

Alternatively, if you want to keep out_parms, the same output is generated if you change out_parms to

out_parms="-w %{http_code} %{url_effective}\n"

and put double quotes around it in the curl command

curl ${auth_header} -s http://${h}/${p} -o /dev/null "${out_parms}"

Collectives™ on Stack Overflow

Running curl in a loop results in unexpected output

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related