Bash: Looping urls using curl, giving only first url output

Question

What I want to do: I want to find all the products(URLs) which are not redirected.

To get the final URL after redirection I'm using curl command as follows:

curl -Ls -o /dev/null -w %{url_effective} "$URL"

This is working fine. Now I want to iterate over URLs to get which are the URLs that are not redirected and display them as output of program. I've the following code:

result=""
productIds=$1
for productId in $(echo $productIds | sed "s/,/ /g")
do
    echo "Checking product: $productId";
    URL="http://localhost/?go=$productId";
    updatedURL=`curl -Ls -o /dev/null -w %{url_effective} "$URL"`

    echo "URL : $URL, UpdatedUrl: $updatedURL";
    if [ "$URL" == "$updatedURL" ]
    then
            result="$result$productId,";
    fi
done

The curl command works only for the first product. But from 2nd to last product, all the URL and updatedURL are same. I can't understand the reason why? The productId is changing in every iteration, so I think it cannot be something related to caching.

Also, I've tried the following variant of curl also:

updatedURL=$(curl -Ls -o /dev/null -w %{url_effective} "$URL")

updatedURL="$(curl -Ls -o /dev/null -w %{url_effective} "$URL")"

Edit: After trying with debug mode and lot of different ways. I noticed a pattern i.e. If I manually hit the following on terminal:

curl -Ls -o /dev/null -w %{url_effective} "http://localhost/?go=32123"

Then in shell script these urls will work fine. But if I don't hit manually then curl will also not work for those products via shell script.

How are you invoking the script, i.e. what is $1/$productIds? — Tom Fenech
– Tom Fenech, Commented May 20, 2016 at 11:30
sh test.sh 123,456,789,221 is what I'm using to invoke the shell script. $1 is what I'm using to get user input inside script. — Sunil Kumar
– Sunil Kumar, Commented May 20, 2016 at 11:41
put #!/bin/bash -vx in the top line and make test.sh executable and then call test.sh without sh - should give lots of debug output so you can see if the URL assignment works — Stefan Hegny
– Stefan Hegny, Commented May 20, 2016 at 13:16
can you detail "curl will also not work via shell script", what exactly happens? can you try curl -L "$URL" echo $? updatedURL=`curl -Ls -o /dev/null -w %{url_effective} "$URL"` inside the loop to see what is so strange about these calls? — Stefan Hegny
– Stefan Hegny, Commented May 20, 2016 at 14:07

sumitya · Accepted Answer · 2016-05-28 22:30:54Z

1

Just add #!/bin/bash to be the first line of shell. It does produce required output. Now invocation should be like this bash file.sh 123,456,789,221

Invocation via Bourne shell sh file.sh 123,456,789,221 does requires code changes.Do let me know if you require that too :)

edited May 28, 2016 at 22:30

answered May 28, 2016 at 22:25

sumitya

2,7111 gold badge21 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Tom Fenech · Accepted Answer · 2016-05-22 09:21:34Z

0

I would suggest changing your loop to something like this:

IFS=, read -ra productIds <<<"$1"
for productId in "${productIds[@]}"; do
    url=http://localhost/?go=$productId
    num_redirects=$(curl -Ls -o /dev/null -w %{num_redirects} "$url")
    if [ "$num_redirects" -eq 0 ]; then
        printf 'productId %s has no redirects\n' "$productId"
    fi
done

This splits the first argument passed to the script into an array, using a comma as the delimiter. The number of redirects is stored to a variable. When the number is zero, the message is printed.

I have to admit that I can't see anything inherently broken with your original approach so it's possible that there is something extra going on that we're not aware of. If you could provide a reproducible test case then we would be able to help you more effectively.

edited May 22, 2016 at 9:21

answered May 20, 2016 at 11:45

Tom Fenech

75.1k13 gold badges119 silver badges154 bronze badges

2 Comments

Sunil Kumar Over a year ago

This loop doesn't helped. Same problem. It is doing the same thing as the previous one. Also, you might want to change product_ids in your read line to productIds.

Tom Fenech Over a year ago

Good point about the variable name, I forgot to change them both back after testing (I normally use underscores in shell scripts). I would still suggest this approach as it saves you using an external tool. Unfortunately I can't really debug without knowing what your variables are. I'd suggest using set -x.

Collectives™ on Stack Overflow

Bash: Looping urls using curl, giving only first url output

2 Answers 2

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related