Too many open file while using numpy load

Question

Hey i guys stuck in this error, i really need help:

Error is: Too many open files.

        for i in data_files[:8]:
            try:
                data_temp = np.load(i)
                nat_queue += data_temp['nat_queue'].tolist()
                temp_radius = data_temp['temp_radius'].tolist()
                final_radius += data_temp['final_radius'].tolist()
                del data_temp.f
                data_temp.close()
                os.unlink(os.path.join('/tmp/rad_data', i))
            except:
                pass

when i run this code RAM(System Memory) increases linearly and after a while i got too many open files. what should i do..??

any help or suggestion would be appreciated?

except: pass isn't good practice, since you loose information on what is going on. Try printing the error - I would assume there is some issue with del data_temp.f - then the close statement would be skipped. — rammelmueller
– rammelmueller, Commented May 27, 2017 at 9:40
What happens to nat_queue, temp_radius, final_radius.. List take up a lot of space... — J.Down
– J.Down, Commented May 27, 2017 at 9:51

John Zwinck · Accepted Answer · 2017-05-27 12:01:00Z

1

You should be using the "Context Manager" mechanism when you call np.load() on multi-array files like this:

for path in data_files[:8]:
    with np.load(path) as arc:
        nat_queue += arc['nat_queue'].tolist()
        temp_radius = arc['temp_radius'].tolist()
        final_radius += arc['final_radius'].tolist()
    os.unlink(os.path.join('/tmp/rad_data', path))

This way, the archive of arrays arc will be closed.

By the way, if the inputs are large, you are doing yourself no favors by converting all the data to lists. You could just append each loaded array to a list of arrays, then do np.concatenate(list_of_arrays) at the end to get a single large array from all the files' contents.

answered May 27, 2017 at 12:01

John Zwinck

252k44 gold badges346 silver badges459 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

DonJon Over a year ago

thanx for reply. your answer seems very logical, i will definitely try

Collectives™ on Stack Overflow

Too many open file while using numpy load

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related