Python/Pandas, .count not working on larger data frame

Question

I am trying to count the number of True/False values in a data frame like this:

df = pd.DataFrame({'a': [True, False, True],
                  'b': [True, True, True],
                  'c': [False, False, True]})
count_cols = ['a', 'b', 'c']
df['count'] = df[df[count_cols] == True].count(axis=1)

This is working fine on this example. But when I test it on my actual df (shape - (25168, 303)), I am getting the following error:

I Understood from - What does `ValueError: cannot reindex from a duplicate axis` mean? - that this usually occurs when there are duplicate values in the index and I have tried both df.reindex() and df[~df.index.duplicated()], but I am still getting the same error message.

have you tried df.sum(axis=1) ?

Andy L.
– Andy L.

2019-09-09 10:07:33 +00:00
Commented Sep 9, 2019 at 10:07 — Andy L.
– Andy L., Commented Sep 9, 2019 at 10:07
Thanks, but that throws the same error

Maverick
– Maverick

2019-09-09 10:57:44 +00:00
Commented Sep 9, 2019 at 10:57 — Maverick
– Maverick, Commented Sep 9, 2019 at 10:57

jezrael · Accepted Answer · 2019-09-09 11:00:38Z

1

Filter columns by list and count Trues values by sum - Trues are processing like 1s:

df['count'] = df[count_cols].sum(axis=1)
print (df)
       a     b      c  count
0   True  True  False      2
1  False  True  False      1
2   True  True   True      3

EDIT: For avoid error one possible solution is convert values to numpy array:

df['count'] = np.sum(df[count_cols].values, axis=1)
print (df)
       a     b      c  count
0   True  True  False      2
1  False  True  False      1
2   True  True   True      3

edited Sep 9, 2019 at 11:00

answered Sep 9, 2019 at 10:13

jezrael

868k103 gold badges1.4k silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Maverick Over a year ago

Thanks, but df[count_cols].sum(axis=1) throws the same error

Maverick Over a year ago

Perfect, thank you! Do you have any idea why mine works on the test df but not the big one?

jezrael Over a year ago

@Maverick It seems some data related issue, hard to know

Collectives™ on Stack Overflow

Python/Pandas, .count not working on larger data frame

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related