How to define a large list array in Python using For loop or Vectorization?

Question

I have a 2D list of 416 rows, each row having 4 columns. Rows 1-4 contain their row number 4 times (i.e., [...[1,1,1,1],[2,2,2,2]...]. Row 330 contains [41,22,13,13]. Everything else is [0,0,0,0]. Currently I am using a for loop with many explicit if statements.

myList = [[0,0,0,0]]
for i in range(1, 416):
    if i == 1 or i == 2 or i == 3 or i == 4:
        myList.append([i,i,i,i])
    elif i == 330:
        myList.append([41,22,13,13])
    else:
        myList.append([0,0,0,0])

What is a more efficient way for me to define this array?

The other questions I have seen on SO don't seem to explicitly address this question, but if someone finds one that can be considered a duplicate please mark this question as such and I will accept it.

I'd prefer list. But! The main thing I care about is the end result, so if you know of another, better method, lay it on me. — SeeDerekEngineer
– SeeDerekEngineer, Commented Jul 12, 2017 at 14:46
Have you looked at numpy arrays? Much like @Moses's answer, I'd start with a np.zeros and then adjust the necessary rows. — mauve
– mauve, Commented Jul 12, 2017 at 14:51
I do know (and love) numpy, but I have encountered more than one instance where I have a task or assignment that does not permit the use of numpy. In addition, I am a big fan of improving one's use of the simpler tools before going to more advanced ones. So I suppose their is a CS Theory aspect to this question as well. — SeeDerekEngineer
– SeeDerekEngineer, Commented Jul 12, 2017 at 14:53

Moses Koledoye · Accepted Answer · 2017-07-12 15:04:00Z

7

Since the bulk of the list is a sublist of zeros (in array term, sparse), I'll just preallocate and then use slice assignment/indexing to update:

my_list = [[0, 0, 0, 0] for _ in range(416)]
my_list[330] = [41,22,13,13]
my_list[1:5] = [[i]*4 for i in range(1, 5)]

This avoids repeated branch evaluation for a host of false cases and the repeated appends.

OTOH, having zeros all over your memory can be avoided if you actually keep the data structure as a sparse matrix. You could have a look at the SciPy 2-D sparse matrix package.

edited Jul 12, 2017 at 15:04

answered Jul 12, 2017 at 14:49

Moses Koledoye

78.8k8 gold badges139 silver badges141 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

Chris_Rands Over a year ago

Surely the right idea, but the output from their code snippet is not equal to this output (their output may be wrong)

Moses Koledoye Over a year ago

@Chris_Rands I'll say their output is wrong. They are pre-assigning myList = [[]].

javidcf Over a year ago

If my_list is not going to be modified, you can even say zero_list = [0, 0, 0, 0] and then start with my_list = [zero_list for _ in range(416)], saving the time and memory used in creating unnecessary lists.

cs95 Over a year ago

@jdehesa Not a good idea. You do a reference copy there, mutating every list just by changing 1.

Moses Koledoye Over a year ago

@CommonSense ikr, that duplicates your sublist across the list and all hell unleashes. It's a bad idea. See what happens when you update one of the sublists.

|

inspectorG4dget · Accepted Answer · 2017-07-12 14:46:33Z

1

Off the top of my head, a really simple and fast way to do it:

import itertools

answer = list(itertools.chain([[i,i,i,i] for i in range(1,5)], [[0,0,0,0] for _ in range(330-6)], [[41,22,13,13]], [[0,0,0,0] for _ in range(416-330)]))

answered Jul 12, 2017 at 14:46

inspectorG4dget

115k30 gold badges159 silver badges253 bronze badges

Collectives™ on Stack Overflow

How to define a large list array in Python using For loop or Vectorization?

2 Answers 2

8 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

8 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related