Remove duplicate values in each column in every row pandas python

Question

I have 4 columns ( Name, Area, State, Email). In column Email has duplicate email after i group by and join the value. The output turns out to be:

Area    Group   Shift         Email
KM      MFG     A1,A2,A3      [email protected],[email protected],[email protected]
RIM     TEST    A1,B1         [email protected],[email protected],[email protected]
PNG    FINISH   C1,C2,C3      [email protected],[email protected],[email protected]
KL     Manager  NORMAL,B1,B1  [email protected],[email protected],[email protected],[email protected]

I need to remove the duplicate email in column 'Email'. Anyone can help me?

Please show us your grouping code so we can tell you what to change to remove duplicates. — cs95
– cs95, Commented Jun 1, 2020 at 3:19
df = df.groupby(['Area','Group])['Email'].apply(','.join).reset_index() — Athirah Omar
– Athirah Omar, Commented Jun 1, 2020 at 9:21

Scott Boston · Accepted Answer · 2020-06-01 03:22:52Z

1

IIUC, you need this:

df['Email'] = df['Email'].str.split(',').apply(lambda x: ','.join(set(x)))

Output:

  Area    Group         Shift                                Email
0   KM      MFG      A1,A2,A3              [email protected],[email protected]
1  RIM     TEST         A1,B1              [email protected],[email protected]
2  PNG   FINISH      C1,C2,C3              [email protected],[email protected]
3   KL  Manager  NORMAL,B1,B1  [email protected],[email protected],[email protected]

answered Jun 1, 2020 at 3:22

Scott Boston

154k15 gold badges160 silver badges207 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

BENY · Accepted Answer · 2020-06-01 03:45:31Z

1

Let us try

df.Email=df.Email.str.split(',').apply(set).str.join(',')
df
  Area    Group         Shift                                Email
0   KM      MFG      A1,A2,A3              [email protected],[email protected]
1  RIM     TEST         A1,B1              [email protected],[email protected]
2  PNG   FINISH      C1,C2,C3              [email protected],[email protected]
3   KL  Manager  NORMAL,B1,B1  [email protected],[email protected],[email protected]

answered Jun 1, 2020 at 3:45

BENY

324k22 gold badges176 silver badges250 bronze badges

1 Comment

Scott Boston Over a year ago

You too.. I am rusty... you have a better answwer. :)

Collectives™ on Stack Overflow

Remove duplicate values in each column in every row pandas python

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related