Optimize per-element operation on sparse numpy 2D array

Question

I require M = np.log(1+M) on a large 99.9% sparse 2D matrix.

How to perform this efficiently?

x, y = M.nonzero() will retrieve coord-pairs of nonzero elements, but can I vectorize a log operation over these pairs?

numpy doesn't seem to have sparse support.

scipy has a sparse module. But I wouldn't use it just for this log — hpaulj
– hpaulj, Commented Apr 21, 2021 at 7:04

CJR · Accepted Answer · 2021-04-21 15:12:02Z

2

This is simplest:

import numpy as np
import scipy.sparse as sps

M = sps.csr_matrix(M)

M.data += 1
M.data = np.log(M.data)

If it's particularly large you could also log it in place (this prevents the full copy above):

M.data += 1
M.data=np.log(M.data,out=M.data)

Both of these options work on dense matrices as well with minor changes - if your matrix is 99.9% sparse I would start using actual sparse data structures though.

You could also use the where argument on a dense array, but I doubt it would actually be any faster:

M = np.add(M, 1, out=M, where=M!=0)
M = np.log(M, out=M, where=M!=0)

edited Apr 21, 2021 at 15:12

answered Apr 21, 2021 at 12:59

CJR

3,9872 gold badges13 silver badges24 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

max9111 Over a year ago

Maybe it is a good idea to also show the way how to perform the log in place: M.data=np.log(M.data,out=M.data) Chunking isn't necessary here.

CJR Over a year ago

@max9111 Oh, good idea - I didn't actually know you could give out the same array.

P i Over a year ago

Note that numpy has log1p (log one plus)

Collectives™ on Stack Overflow

Optimize per-element operation on sparse numpy 2D array

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related