Python Memory Error

Question

I'm runing into Memory Error Using python 2.6, pyodbc

The code loops through several Sql statements. Singleton run Ok, but the loops get stuck. Sometimes Loop works ok.

for Sql in LoopList:
     f = csr.execute(Sql)
     LL = list(f)

The second element in loop crashes.

Sql:

Sql   =""" SELECT * FROM group WHERE 
        C>10 AND M <0 AND S<0  
        AND TC >= 200 AND OC >=1000  and Penny =.01 
        ORDER BY MSlp           
"""

How much data is returned by the SQL statement? If it's hundreds of megabytes or larger, that's your problem right there. — agf
– agf, Commented Jul 28, 2011 at 16:15
It would be helpful if you added a stack trace from the crash, told which platform you're running, what database you are using and other details that may help people identify or recognize the problem. The code you have is really so simple that I doubt it is simply a pyodbc error. — Mattias Nilsson
– Mattias Nilsson, Commented Jul 28, 2011 at 16:16
@joaguin, its hangs... or stops running error console Memory error. — Merlin
– Merlin, Commented Jul 28, 2011 at 16:16
how large is the result set? maybe too large for python to cope with? — Constantinius
– Constantinius, Commented Jul 28, 2011 at 16:17

Mark Ransom · Accepted Answer · 2011-07-28 16:16:30Z

4

Your query is returning too much data to fit into memory. The execute is probably returning an iterator which only needs to keep one item in memory at a time, but converting that into a list requires every single item returned from the query to fit into memory.

answered Jul 28, 2011 at 16:16

Mark Ransom

310k44 gold badges423 silver badges660 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Merlin · Accepted Answer · 2011-09-06 22:11:50Z

0

The solution was to do sql looplist inside MySQL as Stored procedure, call from python. This eliminated the python error.

also use numpy for testing bytesize

answered Sep 6, 2011 at 22:11

Merlin

25.9k44 gold badges141 silver badges213 bronze badges

Comments

Asif Mehmood · Accepted Answer · 2022-12-15 07:48:49Z

0

Since your query is returning too much data that it cannot be stored in RAM, for a quick solution try increasing your virtual memory.

Open My Computer
Right click and select Properties
Go to Advanced System Settings
Click on Advance Tab
Click on settings under Performance
Click on Change under Advance Tab
Increase the memory size, that will increase virtual memory size.

It will convert some of your hard disk space to virtual memory, and your issue will be resolved now if your data fits into allocated memory.

answered Dec 15, 2022 at 7:48

Asif Mehmood

1032 silver badges9 bronze badges

Collectives™ on Stack Overflow

Python Memory Error

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related