self.df['X'] = self.df['x'].apply(lambda x: my_map.get(x))
How can i a drop those rows where my_map.get(x) returns None.
I am looking for a solution where i do not have to iterate over the column again to drop rows.
Thanks
4 Answers
I think you need dropna, because is possible remove None in first step, by assign to new column create NaNs:
self.df['X'] = self.df['x'].apply(lambda x: my_map.get(x))
self.df = self.df.dropna('X')
Or:
self.df = self.df[self.df['X'].notnull()]
8 Comments
Raheel
I understand this logic, But i am afraid because my one df is 250k rows, and this is a chunk out of 60 million.
jezrael
@RaheelKhan - I add another solution, it is a bit faster
piRSquared
I suspect that if you ask a different question where you share what you are trying to do with the entire dataframe and what your lambda is, you’d get a much better answer
Raheel
I tried with 250k df, didn't make any such difference. Thanks
Raheel
@piRSquared
my_map is just a dictionary, where the keys will be x i am assigning the value against that key to X a new column. But since the data is huge there will be many x which will not be matched in my dictso i dont want those records in my df. |
Either loc or pd.Series.compress take a callable argument and return a subset where the callable evaluates to True
compress
self.df['x'].compress(lambda x: my_map.get(x) is not None)
loc
self.df['x'].loc[lambda x: my_map.get(x) is not None]
3 Comments
jezrael
So need
self.df = self.df.dropna('X')piRSquared
@jezrael thinking
Pyd
can you please check this one stackoverflow.com/questions/47096797/…
You can find the indices as follows
idxs = self.df.index[self.df['X'].isnull()] # find all indices with None in df.X
Full code:
self.df['X'] = self.df['x'].apply(lambda x: my_map.get(x))
idxs = self.df.index[self.df['X'].isnull()] # find all indices with None in df.X
self.df = self.df.drop(idxs)
3 Comments
Raheel
do you think
self.df = self.df.dropna('X') this will be more optimized way ?jezrael
self.df['X'] == None return False :(
Fabian Ying
@jezrael Yes, you're right. You can use .isnull(), but I think
self.df.dropna('X') is a cleaner solution.You can do this as a merge, if you convert your mymap to a dict:
mymerge = pd.DataFrame.from_dict(mymap, orient = 'index')
Then use a left join, to only join on the required columns:
mymerge.merge(df, left_index = True, right_on = 'x')
In one line:
pd.DataFrame.from_dict(mymap, orient = 'index').merge(df, left_index = True, right_on = 'x')
applyand then rundropna?