How to create a Python script to make multiple bash scripts?

Question

I have a file, called list.txt, which is a list of names of files:

input1.txt
input2.txt
input3.txt

I want to build a python script that will make one file for each of these filenames. More precisely, I need it to print some text, incorporate the name of the file and save it to a unique .sh file. My script is as follows:

import os
os.chdir("/Users/user/Desktop/Folder")

with open('list2.txt','r') as f:    
    lines = f.read().split(' ')

for l in lines:
    print "#!/bin/bash\n#BSUB -J",l+".sh","\n#BSUB -o /scratch/DBC/user/"+l+".sh.out\n#BSUB -e /scratch/DBC/user/"+l+".sh.err\n#BSUB -n 1\n#BSUB -q normal\n#BSUB -P DBCDOBZAK\n#BSUB -W 168:00\n"
    print "cd /scratch/DBC/user\n"
    print 'grep "input"',l+" > result."+l+".txt"

    with open('script{}.sh'.format(l), 'w') as output:
        output.write(l)

I have a few issues:

The output files just contain the name of the file - not the content I have printed.

To be clear, my output files (I should have 3) should look like this:

#!/bin/bash
#BSUB -J input3.sh 
#BSUB -o /scratch/DBC/user/input1.sh.out
#BSUB -e /scratch/DBC/user/input3.sh.err
#BSUB -n 1
#BSUB -q normal
#BSUB -P DBCDOBZAK
#BSUB -W 168:00

cd /scratch/DBC/user
grep "input" input3 > result.input3.txt

I have now made the following script, which nearly works.

import os
os.chdir("/Users/user/Desktop/Folder")

with open('list.txt','r') as f:
    lines = f.read().split('\n')

for l in lines:
    header = "#!/bin/bash \n#BSUB -J %s.sh \n#BSUB -o /scratch/DBC/user/%s.sh.out \n#BSUB -e /scratch/DBC/user/%s.sh.err \n#BSUB -n 1 \n#BSUB -q normal \n#BSUB -P DBCDOBZAK \n#BSUB -W 168:00\n"%(l,l,l)
    script = "cd /scratch/DBC/user\n"
    script2 = 'grep "input" %s > result.%s.txt\n'%(l,l)
    all= "\n".join([header,script,script2])
    
    with open('script_{}.sh'.format(l), 'w') as output:
        output.write(all)

The problem I still have is that this creates 4 scripts, not 3 as I expected: script_input1.sh, script_input2.sh, script_input3.sh and script_sh. This last one, script_sh just has the printed text but nothing where the "input" text would be.

I think this is because my list.txt file has a "\n" at the end of it? However, I looked and there really isn't. Is there a way around this? Maybe I can use some kind of length function?

There is no need to use Python really, will a Bash script also do? — Nils Werner
– Nils Werner, Commented Oct 11, 2017 at 10:37
No I would really like to use Python in this instance. I know Bash can do this. — mf94
– mf94, Commented Oct 11, 2017 at 11:21
The proper solution would probably be to pass the variable as an argument to a static script, rather than write a number of static scripts with the variable inlined. — tripleee
– tripleee, Commented Oct 11, 2017 at 11:31

FrankBr · Accepted Answer · 2017-10-11 10:44:39Z

1

So, answering in order:

1) Can you detail this issue? You count 4 txt files bu you got just 3 different scripts generated by your code?

2) Sure, you need to create a var, not just using the print statement 3) Just change permissions

So, to summurize, I'd use this approach:

import os
for i, file in enumerate(os.listdir("/Users/user/Desktop/Folder")):
  if "input" in file:
    with open(file) as f:
        lines = f.readlines()

        for l in lines:
            data = ""
            data += "#!/bin/bash\n#BSUB -J",l+".sh","\n#BSUB -o /scratch/DBC/user/"+l+".sh.out\n#BSUB -e /scratch/DBC/user/"+l+".sh.err\n#BSUB -n 1\n#BSUB -q normal\n#BSUB -P DBCDOBZAK\n#BSUB -W 168:00\n"
            data += "cd /scratch/DBC/user\n"
            data += 'grep "input"'+l+" > result."+l+".txt"

            with open('script%s.sh'%i, 'w') as output:
                output.write(data)
            os.chmod("script%s.sh'%i", 700)

By the way, my code it's just a guess. I think you should be more explicit when stating what's your issue. I didn't understood what you wanna achieve.

answered Oct 11, 2017 at 10:44

FrankBr

8962 gold badges17 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

mf94 Over a year ago

Thanks @FrankBr. To answer your questions: 1) Indeed, my list.txt file only has 3 files (input1.txt, input2.txt and input3.txt). However, my current code made 4 files: "scriptinput1.sh", "scriptinput2.sh", "scriptinput3.sh" and "script?.sh". At first, I thought my code was making a 4th file due to the fact I have a newline or spaces at the end of list.txt. But I remove any spaces and newline so I'm confused. 2) That makes sense for the var. Can you explain what "if "input" in file:" does?

FrankBr Over a year ago

Since you loop over a folder, be sure to select the file whose name contains the "input". But if you know your file names, just drop that line. I didn't understand it.

mf94 Over a year ago

Thanks. I will keep looking because its still not working. Problem 1 is solved but I using a variable to store the text is not working. Python tells me you cannot append to tuple variables.

Nils Werner · Accepted Answer · 2017-10-12 17:41:53Z

1

Another solution that, instead of creating files

script1.sh
script2.sh
script3.sh

etc. would be to create one file file script.sh

#!/bin/bash
#BSUB -J "$1".sh 
#BSUB -o /scratch/DBC/user/"$1".sh.out
#BSUB -e /scratch/DBC/user/"$1".sh.err
#BSUB -n 1
#BSUB -q normal
#BSUB -P DBCDOBZAK
#BSUB -W 168:00

cd /scratch/DBC/user
grep "input" "$1" > result."$1".txt

and run it using

script.sh input1
script.sh input2
script.sh input3

answered Oct 12, 2017 at 17:41

Nils Werner

37.2k7 gold badges85 silver badges108 bronze badges

Collectives™ on Stack Overflow

How to create a Python script to make multiple bash scripts?

2 Answers 2

3 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related