1

basically I have a bunch of files containing domains. I've sorted each individual file based on its TLD using .sort(key=func_that_returns_tld)

now that I've done that I want to merge all the files and end up wtih one massive sorted file. I assume I need something like this:

open all files
read one line from each file into a list
sort list with .sort(key=func_that_returns_tld)
output that list to file
loop by reading next line

am I thinking about this right? any advice on how to accomplish this would be appreciated.

3
  • What's wrong with the unix sort command? Commented Aug 24, 2010 at 18:37
  • nothing, but I'm not in linux for this project. ;/ I'm doing it for someone else and the file is just to big to migrate off their machine. Commented Aug 24, 2010 at 18:41
  • Remembering that Python exists on platforms other than Linux...Sort-Object is available in PowerShell Commented Aug 24, 2010 at 18:43

3 Answers 3

8

If your files are not very large, then simply read them all into memory (as S. Lott suggests). That would definitely be simplest.

However, you mention collation creates one "massive" file. If it's too massive to fit in memory, then perhaps use heapq.merge. It may be a little harder to set up, but it has the advantage of not requiring that all the iterables be pulled into memory at once.

import heapq
import contextlib

class Domain(object):
    def __init__(self,domain):
        self.domain=domain
    @property
    def tld(self):
        # Put your function for calculating TLD here
        return self.domain.split('.',1)[0]
    def __lt__(self,other):
        return self.tld<=other.tld
    def __str__(self):
        return self.domain

class DomFile(file):
    def next(self):
        return Domain(file.next(self).strip())

filenames=('data1.txt','data2.txt')
with contextlib.nested(*(DomFile(filename,'r') for filename in filenames)) as fhs:
    for elt in heapq.merge(*fhs):
        print(elt)

with data1.txt:

google.com
stackoverflow.com
yahoo.com

and data2.txt:

standards.freedesktop.org
www.imagemagick.org

yields:

google.com
stackoverflow.com
standards.freedesktop.org
www.imagemagick.org
yahoo.com
Sign up to request clarification or add additional context in comments.

4 Comments

I wouldnt know how to make this work in my case. I need to use the 'key' function of .sort() because i'm sorting based on TLD rather than first character in the line
I've edited my answer to show how you could sort things other than numbers.
@~unutbu: You can use sorted( lst, key=Domain) instead of explicitly mapping.
@THC4k: Thanks for the suggestion, but I'm not sure I follow. sorted will return strings. I need dom1 to be an iterable of Domain objects. (Otherwise, heapq.merge will brainlessly merge them as strings instead of according to TLD.)
0

Unless your file is incomprehensibly huge, it will fit into memory.

Your pseudo-code is hard to read. Please indent your pseudo-code correctly. The final "loop by reading next line" makes no sense.

Basically, it's this.

all_data= []
for f in list_of_files:
    with open(f,'r') as source:
        all_data.extend( source.readlines() )
all_data.sort(... whatever your keys are... )

You're done. You can write all_data to a file, or process it further or whatever you want to do with it.

1 Comment

@d-c: Nope. A few gigs is fine. You have to get to more gigs than you have virtual memory configured in your OS swap space before it begins to matter.
0

Another option (again, only if all your data won't fit into memory) is to create a SQLite3 database and do the sorting there and write it to file after.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.