Doing Calculations On a Two-Dimensional Array Using For Loop

Question

I have a 9X51100 two-dimensional array (9 different datasets, 51,100 daily files) and I'm attempting to take 140 yearly averages of the data and append them to a new array. However, numpy.append needs two arguments and I end up appending more than I want to, I believe. Here's the code and the error I get.

import numpy as np

citydata = ['bcm2.a2.USC00101022.tmax.1960.2099.txt','bcm2.a2.USC00362682.tmax.1960.2099.txt','bcm2.a2.USC00415411.tmax.1960.2099.txt',
    'ccsm.a2.USC00101022.tmax.1960.2099.txt','ccsm.a2.USC00362682.tmax.1960.2099.txt','ccsm.a2.USC00415411.tmax.1960.2099.txt',
    'pcm.a2.USC00101022.tmax.1960.2099.txt','pcm.a2.USC00362682.tmax.1960.2099.txt','pcm.a2.USC00415411.tmax.1960.2099.txt']

year = np.asanyarray([(np.genfromtxt(item, skip_header=1)[:,0]) for item in citydata])
tmax = np.asanyarray([(np.genfromtxt(item, skip_header=1)[:,3]*(9./5.))+32 for item in citydata])

tmax_avg = np.zeros([9,140]) #initialize averaged array
for i in range(0,8,1):
    for yr in years: 
        toavg = (year == yr)
        tmax_avg[i,:] = np.append(tmax_avg,np.average(tmax[toavg]))



ValueError                                Traceback (most recent call last)
<ipython-input-23-a8a57d128124> in <module>()
  8     for yr in years:
  9         toavg = (year == yr)
 10-->      tmax_avg[i,:] = np.append(tmax_avg,np.average(tmax[toavg]))

ValueError: could not broadcast input array from shape (1261) into shape (140)

It also just seems to be only wanting to give me one 140-value array instead of a 9X140 array. Any help with either the append or looping problems? Thanks.

I know this has been mentioned before, but it really feels like you're trying to cram a pandas problem into a numpy framework. Now with enough work, anything you can do with pandas you can do with numpy -- or with plain lists, for that matter -- but you're basically reimplementing df.groupby("year").mean(). [Could also use resample.] — DSM
– DSM, Commented Mar 7, 2014 at 16:08
@DSM I hope to find the time to learn pandas sometime in the near future, but there never seem to be enough hours in the day. I know most of what's going on in numpy so that's why I continue to try and do most of my work with it. — DJV
– DJV, Commented Mar 7, 2014 at 16:13
I am not sure if I fully understand how your data is shaped, year and tmax, but would tmax_avg[i][toavg] = np.average(tmax[toavg]) work. Or maybe tmax_avg[i,j] = np.average(tmax[toavg]) with the for-loop replaced by for j, yr in enumerate(years):work? — oschoudhury
– oschoudhury, Commented Mar 7, 2014 at 16:22
@DJV it doesn't take long to learn. Think of it as just another library of convenient functions built on top of NumPy. With or without Pandas, what would be really helpful here is if you can add some example data for year and tmax or upload some short example files and link to them. As @Wicket says it's not clear how your data is shaped. — YXD
– YXD, Commented Mar 7, 2014 at 16:23
@Wicket Both year and tmax are 9X51100 arrays. Both contain daily data for 140 years worth of information (i.e. for year: 365 "1960"s "1961"s "1962"s and for tmax: a temperature value that corresponds to each day) — DJV
– DJV, Commented Mar 7, 2014 at 16:30

Jaime · Accepted Answer · 2014-03-07 16:58:38Z

1

There is a relatively simple way of vectorizing your inner for loop using np.unique and np.bincount. If I read your code correctly, year is a (9, 51100) array of year labels, and tmax is a corresponding array of the same shape. You can do the following:

tmax_avg = []

count_years = np.unique(year).size

for loc in range(year.shape[0]):
    unq_year, unq_idx = np.unique(year[loc], return_inverse=True)
    unq_sum = np.bincount(unq_idx, weights=tmax[loc], minlength=count_years)
    unq_count = np.bincount(unq_idx, minlength=count_years)
    tmax_avg.append(unq_sum / unq_count)
tmax_avg  = np.vstack(tmax_avg)

You could get rid of the loc loop, but if you onbly have 9 sites it is probably not worth it.

answered Mar 7, 2014 at 16:58

Jaime

67.7k19 gold badges128 silver badges164 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Doing Calculations On a Two-Dimensional Array Using For Loop

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related