Fast mutation of ndarray (replace part of numpy ndarray)

Question

I am looking for fastest way to replace parts of pre-allocated numpy ndarray. Below you can see sim_matrix (2D ndarray). "For loop" iterates over it rowwise and overwrites values given by levels (numpy.array).

There are fast ways, which are found:

numpy.copyto()
numpy.fromiter()
= assignment to slice

NOTE: For simplicity, I assign the same values on each iteration, it is not the case in production setup. In actual setup, levels contains different values for each row. Thus, using shortcut as numpy.repeat or numpy.hstack is discouraged as an answer.

Are there any faster ways to change parts of ndarray?

numpy_overwrite.py:

import numpy
panel_size = 365 * 7
rows = 1 * (1+1+2*4) * 10000
# rows = 100
levels = [1778.24] * panel_size
levels_array = numpy.fromiter(levels,dtype='single', count = len(levels))
sim_matrix = numpy.empty([rows, panel_size],dtype='single')

def _copyto(sim_matrix_, level_):
    for i in range(sim_matrix_.shape[0]):
        numpy.copyto(sim_matrix[i], level_)


def _fromiter(sim_matrix_, level_):
    len_level = len(level_)
    for i in range(sim_matrix_.shape[0]):
        sim_matrix[i] = numpy.fromiter(level_,
                                      dtype='single', 
                                      count = len_level)

def _just_assign(sim_matrix_, level_):
    for i in range(sim_matrix_.shape[0]):
        sim_matrix[i] = level_

numpy_overwrite_test.py

import timeit
import numpy_overwrite as npo
print('_fromiter')
print(timeit.timeit('npo._fromiter(npo.sim_matrix, npo.levels)', setup="import numpy_overwrite as npo;", number=3))
print(timeit.timeit('npo._fromiter(npo.sim_matrix, npo.levels_array)', setup="import numpy_overwrite as npo;", number=3))
print('_copyto')
print(timeit.timeit('npo._copyto(npo.sim_matrix, npo.levels)', setup="import numpy_overwrite as npo;", number=3))
print(timeit.timeit('npo._copyto(npo.sim_matrix, npo.levels_array)', setup="import numpy_overwrite as npo;", number=3))
print('_just_assign')
print(timeit.timeit('npo._just_assign(npo.sim_matrix, npo.levels)', setup="import numpy_overwrite as npo;", number=3))
print(timeit.timeit('npo._just_assign(npo.sim_matrix, npo.levels_array)', setup="import numpy_overwrite as npo;", number=3))

Run test

_fromiter
=========
from python list: 21.129429172957316
from numpy.array: 61.76275303697912

_copyto
=========
from python list: 52.4522930260282
from numpy.array: 0.7605530479922891

_just_assign
=========
from python list: 52.251478374004364
from numpy.array: 0.6351132979616523

@nicoco I use them to visually indicate end of block, removed them. — Cron Merdek
– Cron Merdek, Commented Feb 19, 2018 at 15:05
I think it would be better if you then used a levels which actually does have different values. Maybe randomly distributed? — Graipher
– Graipher, Commented Feb 19, 2018 at 15:11
@Graipher then on each iteration there will be an overhead of pseudo-number generator. — Cron Merdek
– Cron Merdek, Commented Feb 19, 2018 at 15:15
Not in your function call, in your preparation for the test data. As it currently stands, I don't see why what @nicoco suggested should be wrong. You can just pass different levels to the function each call. — Graipher
– Graipher, Commented Feb 19, 2018 at 15:18

nicoco · Accepted Answer · 2018-02-19 15:45:27Z

1

Is it possible that you are simply looking for this?

sim_matrix[:] = levels

EDIT: to modify a single row, one should use sim_matrix[row_id] = levels

On a side note, beware of using mutables as default function arguments.

edited Feb 19, 2018 at 15:45

answered Feb 19, 2018 at 15:15

nicoco

1,56310 silver badges34 bronze badges

Sign up to request clarification or add additional context in comments.

11 Comments

Cron Merdek Over a year ago

Nope, values of levels will be different for each call.

nicoco Over a year ago

And where do these levels come from? A list?

nicoco Over a year ago

If you want to change a single row, just use sim_matrix[row] = levels.

Graipher Over a year ago

@CronMerdek I was trying to find out what your real use case looks like. Because depending on that there might be faster options (like, for example, overwriting multiple rows at once, if you have all the levels).

Cron Merdek Over a year ago

@Graipher I have added example of invocation for assignment

|

Collectives™ on Stack Overflow

Fast mutation of ndarray (replace part of numpy ndarray)

numpy_overwrite.py:

numpy_overwrite_test.py

Run test

1 Answer 1

11 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

numpy_overwrite.py:

numpy_overwrite_test.py

Run test

1 Answer 1

11 Comments

Your Answer

Sign up or log in

Post as a guest

Related