Encoding and Decoding of text in Python

Question

I am currently working with a python script (appengine) that takes an input from the user (text) and stores it in the database for re-distribution later.

The text that comes in is unknown, in terms of encoding and I need to have it encoded only once.

Example Texts from clients:

This%20is%20a%20test
This is a test

Now in python what I thought I could do is decode it then encode it so both samples become:

This%20is%20a%20test
This%20is%20a%20test

The code that I am using is as follows:

#
# Dencode as UTF-8
#
pl = pl.encode('UTF-8')

#
#Unquote the string, then requote to assure encoding
#
pl = urllib.quote(urllib.unquote(pl))

Where pl is from the POST parameter for payload.

The Issue

The issue is that sometimes I get special (Chinese, Arabic) type chars and I get the following error.

'ascii' codec can't encode character u'\xc3' in position 0: ordinal not in range(128)
    ..snip..
    return codecs.utf_8_decode(input, errors, True)
 UnicodeEncodeError: 'ascii' codec can't encode character u'\xc3' in position 0: ordinal not in range(128)

does anyone know the best solution to process the string given the above issue?

Thanks.

Not exactly a duplicate, but see: stackoverflow.com/questions/436220/… — mVChr
– mVChr, Commented Jul 26, 2012 at 3:31

Mechanical snail · Accepted Answer · 2012-07-26 05:56:20Z

1

Replace

pl = pl.encode('UTF-8')

with

pl = pl.decode('UTF-8')

since you're trying to decode a byte-string into a string of characters.

A design issue with Python 2 lets you .encode a bytestring (which is already encoded) by automatically decoding it as ASCII (which is why it apparently works for ASCII strings, failing only for non-ASCII bytes).

answered Jul 26, 2012 at 5:56

Mechanical snail

30.7k15 gold badges91 silver badges113 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Encoding and Decoding of text in Python

The Issue

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

The Issue

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related