Force flush Python-part of Bash-script?

Question

I have a simple Bash-script that spend more than 99 % of its time on executing a Python application (the script is just a wrapper that feeds this Python script files in a for-loop and renames the output files). I have the same problem as described here and have looked at the answers but don't understand how to use them in my context.

I think I need to apply the principles in the answers to that question inside my script, on the line that executes the Python script, but how?

My script as pseudo-code:

for file in "$directory"; do
    pythonscript "$file" >> "log.txt"; then
done

I want to flush the pythonscript-line every minute, every line of output it produces or similar, it doesn't matter that much (typically execution takes several hours, I just want to be able to track the output "reasonable" frequently).

Dump questions, why are u not move whole Logic into the Python script itself, like get all files and iterate eg call your function and just adding logging Into the Python script, that way you dont need to mix Python and bash — Christoph
– Christoph, Commented Nov 3, 2023 at 9:25
@Christoph Hadn't thought about that, and it makes some sense if I had written the Python application, but it's external and is updated on a regular basis, which means that I would have to apply the modification every time it's updated. — d-b
– d-b, Commented Nov 3, 2023 at 15:55
The top-voted answer to the question you linked says that given a script /homedir/MyScript that you want to run unbuffered you should call is as stdbuf -oL /homedir/MyScript instead. Your script is pythonscript "$file" so call it as stdbuf -oL pythonscript "$file" instead. See gnu.org/software/coreutils/manual/html_node/… for more info on stdbuf. — Ed Morton
– Ed Morton, Commented Nov 3, 2023 at 20:41
By the way consider using for file in "$directory"; do whatever; done >log.txt instead of for file in "$directory"; do whatever >> log.txt; done so you only open the output file once, and for file in "$directory" is confusing terminology at best or simply a bug, depending on what it's meant to do. — Ed Morton
– Ed Morton, Commented Nov 3, 2023 at 20:51
@EdMorton I am on a Mac and don't seem to have stdbuf (despite installing GNU core utils from MacPorts). — d-b
– d-b, Commented Nov 4, 2023 at 13:10

Diego Torres Milano · Accepted Answer · 2023-11-03 15:42:03Z

1

You can use PYTHONUNBUFFERED env var

export PYTHONUNBUFFERED=true
for file in "$directory"; do
    pythonscript "$file" >> "log.txt"; then
done

answered Nov 3, 2023 at 15:42

Diego Torres Milano

69.9k9 gold badges116 silver badges145 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Force flush Python-part of Bash-script?

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related