Bash sending multiple curl request using GNU parallel

Question

I have a list of URLs ( 5000+ ) and I need to send 25 URLs in parallel to a internal service. I know how to send the URLs request using a single query.

curl -s http://192.168.150.113:9999/app.boxx.com

And I tried using GNU parallel,

while true;do parallel -j25 curl -s http://192.168.150.101:9999/'{}' < list;done

Is it good to use GNU parllel ? It works good but i feel the response is quite slow and the response is similar to a single API request.

Instead , Can we use ampersand ( & ) at end of each urls and send the request in parallel ?

You might want to also checkout the -P (a.k.a --max-procs) option of xargs — Jay
– Jay, Commented Jun 4, 2018 at 7:48

Ole Tange · Accepted Answer · 2018-06-04 10:55:46Z

4

Inians answer is perfectly valid and is the preferred if you need to do something more complex. But if you are only going to do a single curl you can do:

parallel -j25 curl -s http://192.168.150.101:9999/{} < list

answered Jun 4, 2018 at 10:55

Ole Tange

34.1k9 gold badges93 silver badges111 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Inian · Accepted Answer · 2018-06-04 07:43:15Z

I'm not sure if you are using the full potential of GNU parallel to the extent to what is should be used. For it to work, you need to do define a smaller job (the least smallest unit that you can breakdown) and let it run for the the number of times you want.

Define a function to read from the URL, assuming the part http://192.168.150.113:9999/ is a fixed string and rest of the URL comes from a file, define a function as

oneShot() {
    url="http://192.168.150.113:9999/"
    finalURL="$url$1"
    curl -s "$finalURL"   
}

and export this function to make it available across child-shells

export -f oneShot

and now do the magic to achieve parallelism, to run 25 jobs in parallel

parallel -j25 oneShot < list

Collectives™ on Stack Overflow

Bash sending multiple curl request using GNU parallel

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related