A bash script for multiple wget commands [duplicate]

Question

I have mutliple (over 100) similar to this:

wget -q -nH --cut-dirs=5 -r -l0 -c -N -np -R 'index*' -erobots=off --retr-symlinks http://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2009_02//00031327004/auxil/
wget -q -nH --cut-dirs=5 -r -l0 -c -N -np -R 'index*' -erobots=off --retr-symlinks http://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2009_01//00031327001/uvot/
wget -q -nH --cut-dirs=5 -r -l0 -c -N -np -R 'index*' -erobots=off --retr-symlinks http://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2010_12//00031856009/uvot/
wget -q -nH --cut-dirs=5 -r -l0 -c -N -np -R 'index*' -erobots=off --retr-symlinks http://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2008_01//00031043003/uvot/
wget -q -nH --cut-dirs=5 -r -l0 -c -N -np -R 'index*' -erobots=off --retr-symlinks http://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2012_01//00032237004/uvot/

I have been told that this can be completed very quickly and easy with a bash script, could someone give me an example if possible for multiple wget commands? What do I need to include within the script? Apologies for my n00b questions, but...I am one!

How similar? What's different between each call? Just the URL? — Mr. Llama
– Mr. Llama, Commented Dec 3, 2015 at 20:50
bash your-file-here.txt, if all you want is to run it as a script. — Charles Duffy
– Charles Duffy, Commented Dec 3, 2015 at 21:31

miken32 · Accepted Answer · 2015-12-03 21:21:53Z

2

Assuming your URL list looks like this:

http://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2009_02//00031327004/auxil/
http://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2009_01//00031327001/uvot/
http://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2010_12//00031856009/uvot/
http://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2008_01//00031043003/uvot/
http://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2012_01//00032237004/uvot/

Just read your URLs from the text file using a while loop:

#!/bin/bash
while read url; do
    wget -q -nH --cut-dirs=5 -r -l0 -c -N -np -R 'index*' -erobots=off --retr-symlinks "$url"
done < urls.txt

edited Dec 3, 2015 at 21:21

answered Dec 3, 2015 at 21:00

miken32

42.5k16 gold badges127 silver badges177 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Rozuur · Accepted Answer · 2015-12-03 20:52:38Z

0

Assuming your noob question is about speeding up the process time,

you can spawn multiple process's in background & and wait for them

wget firsturl &
wget secondurl &
...
wait

answered Dec 3, 2015 at 20:52

Rozuur

4,17527 silver badges32 bronze badges

4 Comments

GCien Over a year ago

I'd really prefer a bash script.

Rozuur Over a year ago

its a bash script, just replace firsturl with what ever you are using

GCien Over a year ago

And this can be placed in a text file to be run as a script?

Rozuur Over a year ago

yes, just replace ... with all your urls and end each line with &

Collectives™ on Stack Overflow

A bash script for multiple wget commands [duplicate]

2 Answers 2

Comments

4 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

4 Comments

Linked

Related