3

I'm learning Matplotlib, and trying to implement a simple linear regression by hand. However, I've run into a problem when importing and then working with my data after using csv2rec.

data= matplotlib.mlab.csv2rec('KC_Filtered01.csv',delimiter=',')

x = data['list_price']
y = data['square_feet']

sumx = x.sum()
sumy = y.sum()

sumxSQ = sum([sq**2 for sq in x])
sumySQ = sum([sq**2 for sq in y])

I'm reading in a list of housing prices, and trying to get the sum of the squares. However, when csv2rec reads in the prices from the file, it stores the values as an int32. Since the sum of the squares of the housing prices is greater than a 32 bit integer, it overflows. However I don't see a method of changing the data type that is assigned when csv2rec reads the file. How can I change the data type when the array is read in or assigned?

Improve this question

2 Answers 2

Reset to default
2 
x = data['list_price'].astype('int64')

and the same with y.

And: csv2rec has a converterd argument: http://matplotlib.sourceforge.net/api/mlab_api.html#matplotlib.mlab.csv2rec

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Thank you! this worked like a charm. I didn't know numpy had in-place casting. This link even had the type formatting strings I was looking for in my earlier comment. Thanks again!
2

Instead of mlab.csv2rec, you can use an equivalent function of numpy, numpy.loadtxt (documentation), to read your data. This function has an argument to specify the dtype of your data.

Or if you want to work with column names (as in your example code), the function numpy.genfromtxt (documentation). This is like loadtxt, but with more options, such as to read in the column names from the first line of your file (with names = True).

An example of its usage:

In [9]:
import numpy as np
from StringIO import StringIO
data = StringIO("a, b, c\n 1, 2, 3\n 4, 5, 6")
np.genfromtxt(data, names=True, dtype = 'int64', delimiter = ',')

Out[9]: 
array([(1L, 2L, 3L), (4L, 5L, 6L)], 
      dtype=[('a', '<i8'), ('b', '<i8'), ('c', '<i8')])

Another remark on your code, when using numpy arrays you do't have to use for-loops. To calculate the square, you can just do:

xSQ = x**2
sumxSQ = xSQ.sum()

or in one line:

sumxSQ = numpy.sum(x**2)
Improve this answer

4 Comments

Thank you for the heads up regarding squares and for loops, I can really see where this will make the code simpler and cleaner. I've been avoiding using loadtxt as the dtype array doesn't seem to have a clear explanation of what the format strings are. I'll see if I can chase that down later.
What do you mean with 'the dtype array doesn't seem to have a clear explanation of what the format strings are'? You can just use dtype = 'int64'. I added an example.
The function loadtxt requires a dictionary of dtype when reading in an array that is composed of different column types, say, when mixing strings, ints, and floats. I was using csv2rec because it could read the various data types automatically, and allow me to focus on the data set and problem itself, rather than trying to understand the dtype dictionary. I know it's probably simplistic, but I'm taking a 'walk before you run' approach to learning Matplotlib and Numpy/Scipy. I really do appreciate the example, though.
Ah, I understand. That's the reason I usually use genfromtxt instead of loadtxt because it can read the names and the dtype from the data as csv2rec.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.