Merging files on disk in Python [closed]

Question

Closed. This question needs details or clarity. It is not currently accepting answers.

Want to improve this question? As written, this question is lacking some of the information it needs to be answered. If the author adds details in comments, consider editing them into the question. Once there's sufficient detail to answer, vote to reopen the question.

Closed last year.

Improve this question

I am trying to merge key-value pairs stored in multiple files. The files themselves are large so I want to do this in memory. The files are sorted, and contain 1 key-value pair per line, so normal k-way merge would work but I don't know how to iterate through the files together. Also in case I encounter multiple data points with the same keys, I want to combine them (add their values together) so the simple solution in Implementing an external merge sort does not work.I have tried using readlines to iterate over the files and storing indices, but readlines loads the entire file into memory so it does not acheive my need. As clarified by comments, the question boils down to reading consecutive lines from multiple files without using a for line in file loop.

readlines loads the entire file into memory so it does not acheive my need So read one line at a time. What is the difficulty? — John Gordon
– John Gordon, Commented Feb 18, 2024 at 17:27
Looks like you forgot to post the code you're struggling with MRE — jackal
– jackal, Commented Feb 18, 2024 at 17:28
@JohnGordon The problem being the way I know to iterate over IOstream is using a for loop, but I can't do that one at a time for multiple files like i need to for k-way merge — Aman Sharma
– Aman Sharma, Commented Feb 18, 2024 at 17:37
You can call file.readline() to read a single line. Is that the part you're missing? — John Gordon
– John Gordon, Commented Feb 18, 2024 at 17:40
you can have N files open at the time, put them in a list, and you can read a byte or line from any file you have open, be sure to close all the files you have opened — rioV8
– rioV8, Commented Feb 18, 2024 at 17:50

John Gordon · Accepted Answer · 2024-02-18 18:07:06Z

1

It seems your real question was "how do I arbitrarily read the next line from a file, without using a for line in file loop?"

The answer to that is file.readline().

answered Feb 18, 2024 at 18:07

John Gordon

33.8k9 gold badges48 silver badges72 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

juanpa.arrivillaga Over a year ago

Or just next(file) which I would argue is more idiomatic

Kelly Bundy Over a year ago

@juanpa.arrivillaga Or just use heapq.merge instead of trying to reimplement it, then they wouldn't even have this issue.

Collectives™ on Stack Overflow

Merging files on disk in Python [closed]

1 Answer 1

2 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Linked

Related