11

As Python 3k introduces strict distinction between strings and bytes, command line arguments in the array sys.argv are presented as strings. Sometimes it is necessary to treat the arguments as bytes, e.g. when passing a path that needn't to be in any particular character encoding in Unix.

Let's see an example. A brief Python 3k program argv.py follows:

import sys

print(sys.argv[1])
print(b'bytes')

When it is executed as python3.1 argv.py français it produces expected output:

français

b'bytes'

Note that the argument français is in my locale encoding. However, when we pass the argument in a different encoding we obtain an error: python3.1 argv.py `echo français|iconv -t latin1`

Traceback (most recent call last):
  File "argv.py", line 3, in <module>
    print(sys.argv[1])
  UnicodeEncodeError: 'utf-8' codec can't encode character '\udce7' in position 4: surrogates not allowed

How shall we pass binary data to Python 3k program via command line arguments? An example of usage is passing a path to a file of a user who uses other locale.

Improve this question
1
  • 3
    The encoding issue mislead the answers, I suggest a python3 argv.py `echo -ne "\xff\x80\x00"` which is an example of « passing binary data via command line arguments » Commented Oct 22, 2014 at 8:48

2 Answers 2

Reset to default
9

Note that the error is a UnicodeEncodeError rather than a UnicodeDecodeError. Python is preserving the exact bytes passed on the command line (via the PEP 383 surrogateescape error handler), but those bytes are not valid UTF-8 and hence can't be encoded as such for writing to the console.

The best way to deal with this is to use the application level knowledge of the correct encoding to reinterpret the command line argument inside the application, as in the following example code:

$ python3.2 -c "import os, sys; print(os.fsencode(sys.argv[1]).decode('latin-1'))" `echo français|iconv -t latin1`
français

The os.fsencode function invocation reverses the transformation Python applied automatically when processing the command line arguments. The decode('latin-1') method invocation then performs the correct conversion in order to get a properly decoded string.

Python 3.2 added os.fsencode to specifically to make this kind of problem easier to deal with.

For Python 3.1, the equivalent construct for os.fsencode(sys.argv[1]) is sys.argv[1].encode(sys.getfilesystemencoding(), 'surrogateescape')

Edit Feb 2013: updated for Python 3.2+, and to avoid assuming that Python autodetected "UTF-8" as the command line encoding

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

You can't assume that the locale is UTF-8. Therefore, you have to encode using the filesystem encoding: sys.argv[1].encode(sys.getfilesystemencoding(), 'surrogateescape').decode('latin-1')
Indeed. Although, in Python 3.3, using os.fsencode is even easier. I've also filed a bug, noting that the sys.argv docs should actually explain this point.
1

You can do:

sys.argv[1].encode() or, if you know the encoding use it as argument or call bytes(sys.argv[1], 'latin-1').

Both should give you a byte representation of the unicode string.

By default, Python3 uses UTF-8.

Improve this answer

5 Comments

No, Python doesn't necessarily use UTF8, it depends on the platform.
@Lennart Python 3 does use UTF-8 as default encoding.
No, it's not that simple. The default encoding for terminals is not necessarily UTF-8 and neither is the default encoding for file systems. I quote "There is a platform-dependent default encoding, which on Unixy platforms can be set with the LANG environment variable (and sometimes also with some other platform-specific locale-related environment variables). In many cases, but not all, the system default is UTF-8; you should never count on this default." It is in fact cp1252 that is the default on Windows. The default for .py files is UTF8 though.
Unfortunately, the solution suggested by @JBernardo does not work because Python cannot process latin1 bytes of français as UTF-8 bytes that it expects to get from my UTF-8 environment. It fails on interpreting the command line argument (bytes in C) expected to be bytes of a string encoded to locale's encoding.
I concur with Lennart that it isn't that simple - the automatic surrogateescape process needs to be reversed in order to get at the raw bytes, which can then be processed according to application specific knowledge.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.