Replace values in dataframe Python

Question

Lon_X        Lat_Y
5,234234     6,3234234
5,234234     6,3234234
5,234234     6,3234234
5,234234     6,3234234
5,234234     6,3234234

I've GPS coordinates in a pandas/dataframe like above. These however use the comma separator. What's the best way using pandas to convert these to float GPS coordinates?

for item in frame.Lon_X:
    float(item.replace(",", ".")) # makes the conversion but does not store it back

I've tried the iteritems function, but seems very slow and gives me a warning that I don't really understand:

for index, value in frame.Lon_X.iteritems():
    frame.Lon_X[index] = float(value.replace(",", "."))

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy from ipykernel import kernelapp as app

akuiper · Accepted Answer · 2016-12-31 04:13:44Z

You can use applymap:

df[["Lon_X", "Lat_Y"]] = df[["Lon_X", "Lat_Y"]].applymap(lambda x: float(x.replace(",", ".")))
df

Here is some benchmark about these alternatives, to_float_inplace is significantly faster than all other methods:

Data:

df = pd.DataFrame({"Lon_X": ["5,234234" for i in range(1000000)], "Lat_Y": ["6,3234234" for i in range(1000000)]})

# to_float_inplace
def to_float_inplace(x):
    x[:] = x.str.replace(',', '.').astype(float)

%timeit df.apply(to_float_inplace)
# 1 loops, best of 3: 269 ms per loop

# applymap + astype
%timeit df.applymap(lambda x: x.replace(",", ".")).astype(float)
# 1 loops, best of 3: 1.26 s per loop

# to_float
def to_float(x):
    return x.str.replace(',', '.').astype(float)

%timeit df.apply(to_float)
# 1 loops, best of 3: 1.47 s per loop

# applymap + float
%timeit df.applymap(lambda x: float(x.replace(",", ".")))
# 1 loops, best of 3: 1.75 s per loop

# replace with regex
%timeit df.replace(',', '.', regex=True).astype(float)
# 1 loops, best of 3: 1.79 s per loop

Ted Petrou · Accepted Answer · 2016-12-30 22:59:47Z

1

You can skip using apply and replace directly with the replace method with regex=True

df.replace(',', '.', regex=True).astype(float)

answered Dec 30, 2016 at 22:59

Ted Petrou

62.4k19 gold badges139 silver badges139 bronze badges

Comments

Mike Müller · Accepted Answer · 2016-12-30 23:04:01Z

1

You can apply panda's vectorized methods along an axis in-place:

def to_float_inplace(x):
    x[:] = x.str.replace(',', '.').astype(float)

df.apply(to_float_inplace)

edited Dec 30, 2016 at 23:04

answered Dec 30, 2016 at 22:35

Mike Müller

86k21 gold badges174 silver badges165 bronze badges

Comments

gold_cy · Accepted Answer · 2016-12-30 22:13:52Z

0

Try this:

df.applymap(lambda x: float(x.replace(",", ".")))

Edit: forgot the map as @Psidom showed

answered Dec 30, 2016 at 22:13

gold_cy

14.2k4 gold badges27 silver badges55 bronze badges

Comments

Luke Nichols · Accepted Answer · 2018-03-17 15:55:02Z

Surprisingly, it appears to be faster to iterate over an np series rather than use pd.series.str.replace. I did the following experiment with a 2m row series

setup = '''
import pandas as pd
import numpy as np
a = pd.Series(list('aabc') * 500000)
b = a.values.astype(str)
'''

a = '''
a[:] = a.str.replace("b", "d")
'''
b = '''
b[:] = np.char.replace(b, "b", "d")
'''
c = '''
for i, x in enumerate(b):
    if "b" in x:
        b[i] = "d"
'''
a_speed = min(timeit.Timer(a, setup=setup).repeat(7, 5))
b_speed = min(timeit.Timer(b, setup=setup).repeat(7, 5))
c_speed = min(timeit.Timer(c, setup=setup).repeat(7, 5))

result:

a_speed = 2.3304627019997497

b_speed = 6.832672896000076

c_speed = 1.9407824309996613

Collectives™ on Stack Overflow

Replace values in dataframe Python

5 Answers 5

Comments

Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

Comments

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related