python string unicode issue

Question

Here is my code (I am using python 2.7 )

    result = " '{0}' is unicode or something: ".format(mongdb['field'])
UnicodeEncodeError: 'ascii' codec can't encode character u'\xb0' in position 27: ordinal not in range(128)

It looks like a string I read from mongodb contains unicode. And it throws this error. How to fix it to concatenate this unicde with custom string 'is unicode or something:' ?

Thanks in advance

UPDATE

result = u" '{0}' is unicode or something: ".format(mongdb['field'])

works for me

Is this Python 2 or Python 3?

zwol
– zwol

2013-09-13 22:29:59 +00:00
Commented Sep 13, 2013 at 22:29 — zwol
– zwol, Commented Sep 13, 2013 at 22:29

Thomas Orozco · Accepted Answer · 2013-09-13 22:28:25Z

9

Use a unicode format string (recommended):

result = u" '{0}' is unicode or something: ".format(mongdb['field'])

Or encode the field:

result = " '{0}' is unicode or something: ".format(mongdb['field'].encode('utf-8'))

answered Sep 13, 2013 at 22:28

Thomas Orozco

55.6k12 gold badges120 silver badges120 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Mark Ransom Over a year ago

Second one won't work, as encoded utf-8 contains bytes outside the ASCII range.

Thomas Orozco Over a year ago

@MarkRansom Unless I'm mistaken, Python uses ASCII by default when doing implicit encoding, but that doesn't necessarily mean the string is encoded in ASCII. You're right though, in not recommending using that method anyway; using unicode everywhere internally is definitely superior.

Mark Ransom Over a year ago

x = '{0}'.format(u'\u0234') gives me an ascii codec error. If I make it a Unicode literal u'{0}' it works.

Thomas Orozco Over a year ago

@MarkRansom I know, but that's not what I'm talking about. Here, you're doing an implicit conversion, hence the error. In the answer, I was suggesting an explicit conversion: '{0}'.format(u'\u0234'.encode('utf-8')), which doesn't raise an error. Adding the u is still the recommended method though, which is why it was my first suggestion, and .encode one was the second one.

zwol · Accepted Answer · 2013-09-13 22:29:04Z

1

You have to know what encoding the text coming out of MongoDB is actually in. \xB0 suggests Windows-1252 instead of UTF-8, so try this:

 result = ("'{0}' is unicode or something"
           .format(mongdb['field'].decode('windows-1252'))

answered Sep 13, 2013 at 22:29

zwol

142k39 gold badges272 silver badges388 bronze badges

2 Comments

zwol Over a year ago

In Py3 it should work as written, and in Py2 I don't think you'd get that error in the first place.

Mark Ransom Over a year ago

Judging by the error message it's Py2, and it definitely fails there.

Collectives™ on Stack Overflow

python string unicode issue

2 Answers 2

4 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related