3

I'm using a bash terminal.

I just wanted to peek at the first few names of installed python packages:

pip list | head -n5

And got an error message:

Package            Version
------------------ ----------------
attrs              21.2.0
Automat            20.2.0
Babel              2.8.0
ERROR: Pipe to stdout was broken
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
BrokenPipeError: [Errno 32] Broken pipe

Is something broken? If I'm going to capture that information programmatically, I definitely don't want an error message included.

Improve this question
3
  • 2
    it seems they didn't expect that someone will run it with head. As I know it would need to change Python code in pip to catch BrokenPipe Commented May 22 at 14:52
  • 3
    @furas, but it'd be bad practice to do that, because then you'd be ignoring completely legitimate errors when the pipe failed for a reason that wasn't user-intended (maybe you're piping to gzip to a log and the destination ran out of disk?) -- being unable to fully write one's output should be an error. Commented May 22 at 15:44
  • Some answers to Why does cURL return error "(23) Failed writing body"? describe the cause of the problem, and some provide workarounds. Commented May 22 at 19:23

2 Answers 2

Reset to default
5

This might not completely answer your question, but realize that that error message is Python accurately detecting an abrupt break in the pipe consuming the standard out from pip list, in this case just after the fifth line.

As is common with many programs, data comes out in two different streams: standard out (stdout) and standard error (stderr). Since you are at the terminal, you see both as text to the screen. The desired output (i.e., the "Package" and "Version" information) is streaming to the standard output while the error message is actually going to standard error.

If you did want to capture the standard output to a file (via the > redirect), that error message would still end up on the screen instead of in the file:

pip list | head -n5 > pip_list.txt
ERROR: Pipe to stdout was broken
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
BrokenPipeError: [Errno 32] Broken pipe

You could get a similar list via pip freeze, which doesn't seem to complain when a pipe quits pulling information from it:

pip freeze | head -n5
attrs==21.2.0
Automat==20.2.0
Babel==2.8.0
bcrypt==3.2.0
blinker==1.4

Since pip freeze can be used to make a requirements.txt file, it may be closer to what you want anyway. Also, unlike pip list you don't have the two-line header that you might want to skip.

If you really really do want to use pip list but not get that error message, you could just redirect stderr to /dev/null (a virtual nowhere, a black hole) via the 2> redirect:

pip list 2> /dev/null | head -n5
Package            Version
------------------ ----------------
attrs              21.2.0
Automat            20.2.0
Babel              2.8.0
Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

it looks strange when you write answer to own question and you use sentence This might not completely answer your question, ... and Your question ..., If you ..., You could ..., etc. :)
@furas Yes, it does sound weird when you realize that it's the same person, but I like teaching and find the Q&A method effective.
@furas This morning, I almost reported the behavior described in the question as bug on the system I was working on. But, upon reproducing the behavior on a totally separate machine, I realized that it was more general and did more research and testing. So I wrote it up, answering as best I could. I welcome additional insights, like you gave in your comment to the question. Feel free to add your own answer.
BTW, this is very similar to bash zcat head causes pipefail?, a completely non-Python-related instance of the same issue.
@ChristopherBottoms If the entire output fits into the pipe buffer, you never get an error writing to the pipe.
|
2

Is something broken?

Not in the sense you mean.

The head command terminates after it copies the the specified number of initial lines from its input to its output. This is normal and expected.

If the input is coming from another command via a pipe, and that command attempts to write additional data when there is no longer anything on the consuming end of the pipe, then the system reports an error to it. The conventional terminology for this kind of error is a "broken pipe". This also is normal and expected. It is an error from the perspective of the program receiving it because it prevents the program from completing writing its output.

It is up to the program receiving such an error what to do with it. When that program is a Python interpreter, it responds by throwing a BrokenPipeError, which the Python program has the opportunity to handle in a way appropriate to it. Oftentimes the most appropriate thing to do with that or another OSError is nothing, so that it rises all the way to the top level and causes Python to terminate. That is what pip has done.

If I'm going to capture that information programmatically, I definitely don't want an error message included.

This is why the system provides two standard streams for output:

  • the "standard output", intended for programs to use for normal output, and
  • the "standard error", intended for programs to use for diagnostic messages and similar data that are somehow different in nature from the data emitted to the standard output.

Programs are in control of what data they write to each, but you can generally expect that they will write error messages to standard error. The Python interpreter indeed does this. When you run a program interactively in the terminal, both standard output and standard error default to writing data to the terminal, but they can be captured or redirected independently. That's one of the main reasons for maintaining a separation in the first place.

Improve this answer

1 Comment

a "broken pipe"...is an error from the perspective of the program [writing data] because it prevents the program from completing writing its output. Nice explanation!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.