Can we launch a parallel shell command from python?

Question

I want to use a go executable: timescaledb-parallel-copy to insert data into database from a csv file. However, I plan to use Python for reading in the filename and lookup the appropriate table name for insertion. If I then launch timescaledb-parallel-copy as a Python subprocess to execute on shell, will it still be parallel? I do not need Python to make it parallel, it is parallel by default. I just do not want Python to make it single-threaded.

How could Python make it single-threaded? The GIL is an internal attribute of the Python interpreter. Subprocesses are across a process boundary; they're completely unaffected. — Charles Duffy
– Charles Duffy, Commented May 22, 2019 at 21:45
(I also wouldn't call a go executable a "shell command" at all; if you're doing it right, with shell=False, there's no shell involved anywhere, just the Python interpreter fork()ing itself and execve()ing the Go program -- and even if you're doing it wrong and using shell=True, the shell that the Python interpreter fork()s and exec()s is just fork()ing off a copy of itself and using execve() to replace that copy with the Go executable's process image). — Charles Duffy
– Charles Duffy, Commented May 22, 2019 at 21:46

tsuyoku · Accepted Answer · 2019-05-22 21:43:55Z

1

If you are using subprocess.run() then your program, timescaledb-parallel-copy will execute as if you had called it from the shell. It will still be in parallel. The python script will not be, and will wait on timescaledb-parallel-copy to return.

answered May 22, 2019 at 21:43

tsuyoku

464 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Chintan Pathak Over a year ago

I was able to test that the subprocess runs in parallel if it wants to.

azb_ · Accepted Answer · 2019-05-22 21:44:08Z

0

Yes, I believe it will be. By launching the program as a subprocess, you are running the program as it originally would be, with no interference from python.

answered May 22, 2019 at 21:44

azb_

3872 silver badges10 bronze badges

Collectives™ on Stack Overflow

Can we launch a parallel shell command from python?

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related