How to use ast.literal_eval in a pandas dataframe and handle exceptions

Question

I have a dataframe with a column containing a tuple data as a string. Eg. '(5,6)'. I need to convert this to a tuple structure. One way of doing it is using the ast.literal_eval(). I am using it in this way.

df['Column'] = df['Column'].apply(ast.literal_eval)

Unfortunately, my data in this column contains empty strings also. The ast.literal_eval() is not able to handle this. I get this error.

SyntaxError: unexpected EOF while parsing

I am unsure if this is because it is unable to handle such a character. Based on my reading, I found that ast.literal_eval() works only in cases when a list, dict or tuple is there inside a string structure.

To overcome this I tried to create my own function and return an empty string if it raises an exception.

def literal_return(val):
    try:
        return ast.literal_eval(val)
    except ValueError:
        return (val)

df['Column2'] = df['Column'].apply(literal_return)

Even in this case, the same error pops up. How do we handle this. It would be great even if there is a way to ignore certain rows to apply the function and apply on the rest. Any help is appreciated.

Can you post a small sample of the DataFrame, just so we can verify the empty strings are the issue? — user3483203
– user3483203, Commented Sep 8, 2018 at 6:44
My apologies for the fact that I might not be able to do that. I am saying its because of that since I looked at the dataset and I could not find any other type of data in that column. Again, don't take me wrong — Harikrishna
– Harikrishna, Commented Sep 8, 2018 at 6:47
I think my workaround is correct and just that I did not handle exception for Syntax error. Once I did that, it works fine. My apologies again for not being able to post the example — Harikrishna
– Harikrishna, Commented Sep 8, 2018 at 6:56
In the except statement remove ValueError and return some empty string'' or null inside it. I think it may solve your problem for now. For the correct solution we may require more details on the dataset. — Arihant
– Arihant, Commented Sep 8, 2018 at 6:57

Sokolokki · Accepted Answer · 2020-04-21 10:44:54Z

16

I would do it simply requiring a string type from each entry:

from ast import literal_eval
df['column_2'] = df.column_1.apply(lambda x: literal_eval(str(x)))

If You need to advanced Exception handling, You could do, for example:

def f(x):
    try:
        return literal_eval(str(x))   
    except Exception as e:
        print(e)
        return []

df['column_2'] = df.column_1.apply(lambda x: f(x))

edited Apr 21, 2020 at 10:44

answered Jul 1, 2019 at 20:56

Sokolokki

8631 gold badge9 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

mirrorcoloured Over a year ago

FYI, you can use the function without wrapping it in a lambda: df.column_1.apply(f)

Harikrishna · Accepted Answer · 2018-09-08 06:58:59Z

3

This works when the function is changed to:

def literal_return(val):
    try:
        return ast.literal_eval(val)
    except (ValueError, SyntaxError) as e:
        return val

answered Sep 8, 2018 at 6:58

Harikrishna

1,1405 gold badges13 silver badges32 bronze badges

Collectives™ on Stack Overflow

How to use ast.literal_eval in a pandas dataframe and handle exceptions

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related