Trouble loading large excel file (Python)

Question

I am trying to load a large .xlsx file using OpenPyXL, while loading a 80 MB .xlsx file my entire 8 GB of memory is getting full

Excel file has 4 sheets with 800 000 rows.

from openpyxl import load_workbook
wb = load_workbook('Record.xlsx')

def process(ws):
    '''
    Read all rows of a worksheet
    '''
    data = []
    for a, c, f, k in zip(ws['A'], ws['C'], ws['F'], ws['K']):
        data.append([a, c, f, k])

    return data

ws1 = wb.worksheets[0] # Sheet 1
data1 = process(ws1)
ws2 = wb.worksheets[1] # Sheet 2
data2 = process(ws2)
ws3 = wb.worksheets[2] # Sheet 3
data3 = process(ws3)
ws4 = wb.worksheets[3] # Sheet 4
data4 = process(ws4)

Why while loading 80 MB of excel file 8 GB of memory is not enough?

Can you show us your code, it looks like something is causing a memory leak maybe ? — BcK
– BcK, Commented Apr 20, 2018 at 11:57
is it really that you have only these 2 rows in your code? may be parsing later does give you the issue? I never had problems reading; and for parsing use generators(yield) — Drako
– Drako, Commented Apr 20, 2018 at 11:57
Possible duplicate of Fastest Way To Run Through 50k Lines of Excel File in OpenPYXL — Charlie Clark
– Charlie Clark, Commented Apr 20, 2018 at 12:08

Colin Burke · Accepted Answer · 2018-04-20 11:58:44Z

2

Try using the read_only = True argument to use read only mode, as described here.

answered Apr 20, 2018 at 11:58

Colin Burke

17.3k13 gold badges57 silver badges84 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

arush1836 Over a year ago

While setting read_only = True, I am getting this error AttributeError: 'ReadOnlyWorksheet' object has no attribute 'iter_cols'

Colin Burke Over a year ago

@User Read-only mode doesn't support iterating through columns, only rows. There's a suggested way to do it here.

arush1836 Over a year ago

I used for row in ws.rows to access the rows and it worked. Thank you

Colin Burke Over a year ago

Glad I could help :)

Collectives™ on Stack Overflow

Trouble loading large excel file (Python)

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related