using numpy.where on multidimensional arrays

Question

I have a 2D array, each row represents an output of a classifier that classifies some input to 3 categories (array size is 1000 * 3) :

0.3 0.3 0.3
0.3 0.3 1.0
1.0 0.3 0.3
0.3 0.3 0.3
0.3 1.0 0.3
...

I want to get a list of all the inputs that the classifier is "not sure" about them. And I'm defining "not sure" as no category is above 0.8.

To solve it I use :

np.where(model1_preds.max(axis=1) < 0.8)

This works great.

But now I have 6 classifiers (that have analyzed the same inputs in the same order), and an array 6 * 1000 * 3 representing their results.

I want to find 2 things:

All the inputs that at least one classifier was "not sure" about.
All the inputs that all the classifier were "not sure" about.

I assume the general direction is something like this :

np.stack(np.where(model_preds.max(axis=1) < 0.8) for model_preds in all_preds)

But it won't work because python don't know what I mean in the for loop.

Is your matrix already a 6x1000x3? Or is it a list of six 1000x3 matrices? — willeM_ Van Onsem
– willeM_ Van Onsem, Commented Jul 21, 2017 at 21:25

hilberts_drinking_problem · Accepted Answer · 2017-07-21 21:56:37Z

3

Alternatively to np.where:

res_all_unsure = preds[:,np.amax(preds, axis=(0,2)) <= 0.8,:]
res_one_unsure = preds[:,preds.max(-1).min(0) <= 0.8,:]

answered Jul 21, 2017 at 21:56

hilberts_drinking_problem

11.6k3 gold badges25 silver badges55 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

willeM_ Van Onsem · Accepted Answer · 2017-07-21 21:41:19Z

If it is already a 6×1000×3 matrix preds, you can first np.transpose() it into a 1000×6×3 matrix.

y = preds.transpose(1,0,2)  # preds is the input matrix, 6x1000x3

Next we can turn it into a 1000×6 matrix where for each experiment and for each classifier, we know whether all the values were less than 0.8 by stating:

y = np.all(y<0.8,axis=2)

Finally we can use another np.all() to verify where all the classifiers were unsure:

all_classifiers_unsure = np.where(np.all(y,axis=1))  # all classifiers

Or where any of the classifiers was unsure:

any_classifier_unsure = np.where(np.any(y,axis=1))   # any of the classifiers

We can write it shorter like:

experiment_classifier = np.all(preds.transpose(1,0,2) < 0.8,axis=2)
all_classifiers_unsure = np.where(np.all(experiment_classifier,axis=1))
any_classifier_unsure = np.where(np.any(experiment_classifier,axis=1))

Although I am quite confident, please validate by checking a few indices (ones that are true and ones that are not true).

EDIT

You can still use your proposed method of .max() < 0.8, but with axis=2:

experiment_classifier = preds.transpose(1,0,2).max(axis=2) < 0.8
all_classifiers_unsure = np.where(np.all(experiment_classifier,axis=1))
any_classifier_unsure = np.where(np.any(experiment_classifier,axis=1))

Collectives™ on Stack Overflow

using numpy.where on multidimensional arrays

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related