Adding 2 data frame in python pandas

Question

I want to combine 2 seperate data frame of the following shape in Python Pandas:

I want to have as follows:

df = 
       A    B    C    D
   1   1    2    a    b
   2   3    4    c    d
   3   5    6    e    f

I am using the following code:

dat = df1.join(df2)

But problem is that, In my actual data frame there are more than 2 Million rows and for that it takes too long time and consumes huge memory.

Is there any way to do it faster and memory efficient?

Thank you in advance for helping.

user2734178 · Accepted Answer · 2015-07-15 02:02:44Z

If I've read your question correctly, your indexes align exactly and you just need to combine columns into a single DataFrame. If that's right then it turns out that copying over a column from one DataFrame to another is the fastest way to go ([92] and [93]). f is my DataFrame in the example below:

In [85]: len(f)
Out[86]: 343720

In [87]: a = f.loc[:, ['date_val', 'price']]
In [88]: b = f.loc[:, ['red_date', 'credit_spread']]

In [89]: %timeit c = pd.concat([a, b], axis=1)
100 loops, best of 3: 7.11 ms per loop

In [90]: %timeit c = pd.concat([a, b], axis=1, ignore_index=True)
100 loops, best of 3: 10.8 ms per loop

In [91]: %timeit c = a.join(b)
100 loops, best of 3: 6.47 ms per loop

In [92]: %timeit a['red_date'] = b['red_date']
1000 loops, best of 3: 1.17 ms per loop

In [93]: %timeit a['credit_spread'] = b['credit_spread']
1000 loops, best of 3: 1.16 ms per loop

I also tried to copy both columns at once but for some strange reason it was more than two times slower than copying each column individually.

In [94]: %timeit a[['red_date', 'credit_spread']] = b[['red_date', 'credit_spread']]
100 loops, best of 3: 5.09 ms per loop

Collectives™ on Stack Overflow

Adding 2 data frame in python pandas

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related