Numpy fastest way to apply array of functions to matrix columns

Question

I have an array of functions shape (n,) and a numpy matrix of shape (m, n). Now I want to apply each function to its corresponding column in the matrix, i.e.

matrix[:, i] = funcs[i](matrix[:, i])

I could do this with a for loop (see example below), but using for loops is generally discouraged in numpy. My question is what is the quickest (and preferably most elegant) way to do this?

A working example

import numpy as np

# Example of functions to apply to each row
funcs  = np.array([np.vectorize(lambda x: x+1),
                   np.vectorize(lambda x: x-2),
                   np.vectorize(lambda x: x+3)])
# Initialise dummy matrix
matrix = np.random.rand(50, 3)

# Apply each function to each column
for i in range(funcs.shape[0]):
    matrix[:, i] = funcs[i](matrix[:, i])

Could you share the actual functions that you are working with? — Divakar
– Divakar, Commented Sep 4, 2018 at 13:11
Loops are discouraged when you're performing the same operation over and over, but as you're looping over different Python functions here, it may not be possible to make it any faster. — Denziloe
– Denziloe, Commented Sep 4, 2018 at 13:11
funcs = np.array([np.vectorize(lambda x: x+1),... looks like you've gone off course. It might be better to take a step back and show what problem you're actually trying to solve. — roganjosh
– roganjosh, Commented Sep 4, 2018 at 13:13
@Divakar, the functions I am working with are based on sklearn.cluster.dbscan, but vary slightly for a specific task. I think it would become too complex to explain that for the question I have here. — Thijs van Ede
– Thijs van Ede, Commented Sep 4, 2018 at 13:13
@roganjosh Care to explain? This is pretty close to what it would be in pure Python, i.e. a list of functions. — Denziloe
– Denziloe, Commented Sep 4, 2018 at 13:15

hpaulj · Accepted Answer · 2018-09-04 16:48:56Z

For an array that has many rows and a few columns, a simple column iteration should be time effective:

In [783]: funcs = [lambda x: x+1, lambda x: x+2, lambda x: x+3]
In [784]: arr = np.arange(12).reshape(4,3)
In [785]: for i in range(3):
     ...:     arr[:,i] = funcs[i](arr[:,i])
     ...:     
In [786]: arr
Out[786]: 
array([[ 1,  3,  5],
       [ 4,  6,  8],
       [ 7,  9, 11],
       [10, 12, 14]])

If the functions work with 1d array inputs, there's not need for np.vectorize (np.vectorize is generally slower than plain iteration anyways.) Also for iteration like this there's no need to wrap the list of functions in an array. It's faster to iterate on lists.

A variation on the indexed iteration:

In [787]: for f, col in zip(funcs, arr.T):
     ...:     col[:] = f(col)
     ...:     
In [788]: arr
Out[788]: 
array([[ 2,  5,  8],
       [ 5,  8, 11],
       [ 8, 11, 14],
       [11, 14, 17]])

I use arr.T here so the iteration is on the columns of arr, not the rows.

A general observation: a few iterations on a complex task is perfectly good numpy style. Many iterations on simple tasks is slow, and should be performed in compiled code where possible.

B. M. · Accepted Answer · 2018-09-04 19:07:37Z

1

A loop is efficient here since the job in the loop is heavy.

A readable solution is just :

np.vectorize(apply)(funcs,matrix)

answered Sep 4, 2018 at 19:07

B. M.

18.7k2 gold badges40 silver badges56 bronze badges

1 Comment

Thijs van Ede Over a year ago

Thanks for the solution, do you know if there is an equivalent to apply in python 3.x?

Collectives™ on Stack Overflow

Numpy fastest way to apply array of functions to matrix columns

A working example

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

A working example

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related