How to remove duplicates from a 3D array in Python?

Question

I have a 3D array as follow, 'b', which I want to represent an array of 2-D array. I want to remove the duplicates of my 2-D arrays and get the unique ones.

>>> a = [[[1, 2], [1, 2]], [[1, 2], [4, 5]], [[1, 2], [1, 2]]]
>>> b = numpy.array(a)
>>> b
array([[[1, 2],
        [1, 2]],

       [[1, 2],
        [4, 5]],

       [[1, 2],
        [1, 2]]])

In this above example, I really want to return the following because there exist one duplicate which I want to remove.

unique = array([[[1, 2],
                 [1, 2]],

                 [[1, 2],
                  [4, 5]])

How should do this with numpy package? Thanks

Curious if any of the posted solutions work for you?

Divakar
– Divakar

2016-12-16 11:26:33 +00:00
Commented Dec 16, 2016 at 11:26 — Divakar
– Divakar, Commented Dec 16, 2016 at 11:26

Community · Accepted Answer · 2017-05-23 11:47:35Z

0

See previous answer: Remove duplicate rows of a numpy array convert to array of tuples and then apply np.unique()

edited May 23, 2017 at 11:47

CommunityBot

11 silver badge

answered Dec 10, 2016 at 1:21

Ian Conway

3771 gold badge2 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Community · Accepted Answer · 2017-05-23 11:47:35Z

0

Converting to tuple and back again is probably going to be quire expensive, instead you can do a generalized view:

def unique_by_first(a):
    tmp = a.reshape(a.shape[0], -1)
    b = np.ascontiguousarray(tmp).view(np.dtype((np.void, tmp.dtype.itemsize * tmp.shape[1])))
    _, idx = np.unique(b, return_index=True)
    return  a[idx].reshape(-1, *a.shape[1:])

Usage:

print unique_by_first(a) 
[[[1 2]
  [1 2]]

 [[1 2]
  [4 5]]]

Effectively, a generalization of previous answers.

edited May 23, 2017 at 11:47

CommunityBot

11 silver badge

answered Dec 10, 2016 at 1:29

Daniel

19.6k7 gold badges64 silver badges74 bronze badges

Comments

Divakar · Accepted Answer · 2016-12-10 06:48:56Z

You can convert each such 2D slice off the last two axes into a scalar each by considering them as indices on a multi-dimensional grid. The intention is to map each such slice to a scalar based on their uniqueness. Then, using those scalars, we could use np.unique to keep one instance only.

Thus, an implementation would be -

idx = np.ravel_multi_index(a.reshape(a.shape[0],-1).T,a.max(0).ravel()+1)
out = a[np.sort(np.unique(idx, return_index=1)[1])]

Sample run -

In [43]: a
Out[43]: 
array([[[8, 1],
        [2, 8]],

       [[3, 8],
        [3, 4]],

       [[2, 4],
        [1, 0]],

       [[3, 0],
        [4, 8]],

       [[2, 4],
        [1, 0]],

       [[8, 1],
        [2, 8]]])

In [44]: idx = np.ravel_multi_index(a.reshape(a.shape[0],-1).T,a.max(0).ravel()+1)

In [45]: a[np.sort(np.unique(idx, return_index=1)[1])]
Out[45]: 
array([[[8, 1],
        [2, 8]],

       [[3, 8],
        [3, 4]],

       [[2, 4],
        [1, 0]],

       [[3, 0],
        [4, 8]]])

If you don't mind the order of such slices being maintained, skip the np.sort() at the last step.

p-robot · Accepted Answer · 2017-12-14 17:24:47Z

0

Reshape, find the unique rows, then reshape again.

Finding unique tuples by converting to a set.

import numpy as np
a = [[[1, 2], [1, 2]], [[1, 2], [4, 5]], [[1, 2], [1, 2]]]
b = np.array(a)

new_array = [tuple(row) for row in b.reshape(3,4)]
uniques = list(set(new_array))

output = np.array(uniques).reshape(len(uniques), 2, 2)
output

Out[131]: 
array([[[1, 2],
        [1, 2]],

       [[1, 2],
        [4, 5]]])

edited Dec 14, 2017 at 17:24

answered Dec 10, 2016 at 1:21

p-robot

4,9353 gold badges33 silver badges44 bronze badges

2 Comments

richar8086 Over a year ago

This solution doesn't work:

ValueErrorTraceback (most recent call last) <ipython-input-191-754759117946> in <module>()       6 uniques = np.unique(new_array)       7  ----> 8 output = uniques.reshape(len(uniques), 2, 2)       9 output  ValueError: cannot reshape array of size 4 into shape (4,2,2)

p-robot Over a year ago

Good point. Fixed now to use a set to find uniques. Thanks.

Collectives™ on Stack Overflow

How to remove duplicates from a 3D array in Python?

4 Answers 4

Comments

Comments

Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

Comments

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related