0

I would like to fill in null values of one pandas dataframe with another dataframe (and multiple times, with multiple dataframes).

Example:

df_A
       A     B      C
index                
0      3  5.00   8.00
1      8 25.00    NaN
2      1   NaN 111.00


df_B
         A    B     C
index                
0      NaN 8.00 13.00
1     1.00  NaN   NaN
2     8.00 8.00  8.00

Resulting dataframe:

       A     B      C
index                
0      3  5.00   8.00
1      8 25.00    NaN
2      1  8.00 111.00   # <= 8.00 was filled in here

TL;DR
My actual use case is that I query meteostat for weather data, and you can query different weather stations, where you are given each weather station's distance to your target location... and different weather stations often have different pockets of empty data.

But also, I would like to know how to do this because I am sure I have wanted this in the past for other reasons.

Thank you!

I am having to loop through each row (and column) right now to accomplish this, but I am certain there is a better way!

I have seen this similar question asked, but there is always only one column... must I iterate through column-wise?

Improve this question
0

2 Answers 2

Reset to default
2

Code

use combine_first

out = df_A.combine_first(df_B)

out:

         A     B      C
index                  
0      3.0   5.0    8.0
1      8.0  25.0    NaN
2      1.0   8.0  111.0

or you can use fillna

out = df_A.fillna(df_B)

If you want to preserve the structure of df_A's indexes and columns, fillna is recommended. If you want to preserve the union of df_A and df_B as data (of course, if both exist, df_A takes precedence), use combine_first.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

1

You can concatenate all the dataframes, group by the index, then select the first value for each group (which will not include NaN):

df = pd.concat([df_A, df_B]).groupby(level=0).first()

     A     B      C
0  3.0   5.0    8.0
1  8.0  25.0    NaN
2  1.0   8.0  111.0
Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.