The following is a sub_set of a data frame.
I want to remove all the duplicate items in each row. For example, in the first row, the last value, dizziness, should be removed because dizziness is already exist in column WD2 of row 1.
Output should be like this:
I know how to remove duplicate in a column but I do not know how to do it in a row. Thanks in advance.
import numpy as np
import pandas as pd
df = pd.DataFrame([
['a', 'b', 'c', np.nan],
['a', 'b', 'b', 'a'],
['c', 'b', 'c', 'd']
])
duplicated = df.apply(lambda x: x.duplicated(keep='first'), axis=1) # converts each row into a pd.Series of True/False values indicating whether a cell is a duplicate
print(duplicated) # this is a pd.DataFrame of True/False indicating which cells to drop.
df[duplicated] = np.nan # assign np.nan to duplicates
